Skip to content

armgit / uzu

Public Imported from GitHub
0

A high-performance inference engine for AI models

Merge dev into main (#154)

uuuvn548d14416283
diff --git a/.claude/skills b/.claude/skills
new file mode 120000
index 000000000..a769d4bd9
--- /dev/null
+++ b/.claude/skills
@@ -0,0 +1 @@
+../agents/skills
\ No newline at end of file
diff --git a/.codex/skills b/.codex/skills
new file mode 120000
index 000000000..a769d4bd9
--- /dev/null
+++ b/.codex/skills
@@ -0,0 +1 @@
+../agents/skills
\ No newline at end of file
diff --git a/.cursorrules b/.cursorrules
deleted file mode 100644
index a185a8d3d..000000000
--- a/.cursorrules
+++ /dev/null
@@ -1,145 +0,0 @@
-# Rules
-
-The rules below are CRITICAL. Each rule should be respected. Rules can only be overriden by the user.
-
-## Workflow rules
-
-- When given a coding task, AI Assistant carefully and deeply inspects the codebase to understand the context.
-- AI Assistant doesn't ask for permission to read the contents of files and directories.
-- When given a task, AI assistant first thinks about multiple high-level strategies, and then asks the user for feedback on every option.
-- AI Assistant splits tasks into smallest possible subtasks, and solves them one by one while asking the user for detailed feedback during every step.
-- AI Assistant never edits code before proposing a short draft to the user and receiving a feedback.
-- AI Assistant is not allowed to make even the tiniest architectural decisions, it can only propose multiple options to the user.
-- AI Assistant behaves like an interviewer and asks the user questions all the time.
-- Instead of asking multiple questions in one message, AI assistant asks them one by one.
-- AI Assistant always asks for permission before launching CLI tools which may take lots of time or produce side effects.
-- Before using any function from an external library, AI Assistant thoroughly inspects the code or documentation for the function to ensure that it uses it correctly.
-- When encountering an error, AI Assistant never rushes to quickly fix it. Instead, it uses the scientific method to understand the source of the error: comes up with hypotheses and tests them by performing a series of experiments. It only attempts to fix the error once the hypothesis has been sufficiently validated.
-- AI Assistant avoids creating .md files with explanations unless explicitly requested by the user.
-
-## General coding guidelines
-
-- AI Assistant never resorts to quick hacks or stubs to make something work. Instead it always investigates the problem thoroughly and comes up with a well-thought-out solution.
-- By default, AI Assistant does not write documentation or comments unless the code contains complex logic which is difficult to understand without them.
-- AI Assistant strives for simplicity and maintainability. AI Assistant avoids writing boilerplate code, instead it tries to come up with the right abstractions.
-- AI Assistant prefers functional programming style, namely:
- - Avoids mutable state.
- - Avoids modifying objects in-place.
- - Avoids implicit side effects.
- - Prefers immutable data structures.
- - Strives to make invalid state unrepresentable.
-- AI Assistant tries to write code in the way that makes bugs hard to make and easy to spot.
-- AI Assistant writes the code so that assumptions about invariants are as explicit as possible.
-- AI Assistant leverages strong typing as much as possible, for example, it always uses enums instead of literals.
-- AI Assistant uses good descriptive variable names that make it easy to understand the purpose of the variable without looking at the code that uses it.
-- AI Assistant does not manually parse things such as JSON configs. Instead it first models a schema using types, and then uses a serialisation library such as Cattrs, Pydantic, or Serde to handle serialization and deserialization and validation.
-
-## Testing guidelines
-
-- AI Assistant strives for simplicity and minimalism when writing tests.
-- AI Assistant avoids extensive mocking and stubbing.
-- AI Assistant focuses on testing core functionality, and does not test minor details such as property accessors.
-- AI Assistant avoids boilerplate in the testing code, and tries to come up with the right generic abstractions.
-
-## Swift coding rules
-
-- In SwiftUI views, organize code by sections using `// MARK: - `: Type Definitions -> Environment -> State -> Properties -> Body -> UI -> Logic
-- In SwiftUI views use nested enum State to define the view state
-- Prefer Swfit Modern Structured Concurrency over outdated APIs
-- Prefer @Observable macro for classes instead of outdated @ObservableObject
-- Use camelCase
-- Prefer self-explainatory functions and variables namings instead of leaving code comments
-
-## Python coding rules
-
-- Never use pip to manage dependencies. Instead use uv commands, such as `uv add`.
-- Assume Python 3.12. Use `T | None` instead of `Optional[T]` and `dict[K, V]` instead of `typing.Dict[K, V]`.
-- Every function should have full type annotations.
-- Prefer comprehensions to map and filter expressions.
-- Prefer dataclasses over vanilla python classes.
-- All path-like arguments should have type `Path | str`.
-- Prefer frozen dataclasses or named tuples as primary data structures.
-- Never use dicts in place of dataclasses.
-- __init__.py files should only contain reexports.
-
-## Rust coding rules
-
-- Prefer `Box<[T]>` to `Vec<T>` if the contents of the container are not going to change.
-- Avoid mutable variables as much as possible. Use iterator expressions to collect data into containers.
-- For calling API that was never used before, read the sources of the installed dependencies, otherwise, search in docs.rs.
-
-## Code best practices in examples
-
-General coding:
-
-```python
-# Bad: This expression does not communicate implicit assumptions about the list's shape, and variable names are not descriptive enough.
-x, y = my_list[0], my_list[-1]
-
-# Good: Destructuring clearly communicates the assumptions about the length of the list, and variable names are clear.
-first, *_, last = my_list
-```
-
-```python
-# Bad: it is not clear what values the dish can hold.
-def cook(dish: str) -> ...:
- ...
-
-# Good: Enum makes it clear.
-from enum import Enum
-
-class Dish(Enum):
- STEAK = "steak"
- PIZZA = "pizza"
- SALAD = "salad"
-
-def cook(dish: Dish) -> ...:
- ...
-```
-
-```python
-# Bad: This expression makes the shape of the `x` and the semantics of the operation unclear.
-permuted_x = np.permute(x, (0, 2, 1))
-
-# Good: This expression makes the shape of the `x` and the semantics of the operation clear.
-permuted_x = einops.rearrange(x, "batch in_channels out_channels -> batch out_channels in_channels")
-```
-
-```python
-# Bad: Dict is used instad of a struct.
-def model_registry():
- """Registry of all available models"""
- return {
- "llama": {
- "hf_name": "meta-llama/Llama-3.2-1B-Instruct",
- "test_class": LlamaModelTest,
- "max_layers": 16,
- },
- "qwen": {
- "hf_name": "Qwen/Qwen2.5-1.5B-Instruct",
- "test_class": QwenModelTest,
- "max_layers": 24,
- },
- }
-
-# Good: Frozen dataclasses.
-@dataclass(frozen=True)
-class ModelTestSpec:
- hf_name: str
- test_class: type[BaseModelTest]
- max_layers: int
-
-def model_registry() -> list[ModelTestSpec]:
- return [
- ModelTestSpec(
- hf_name="meta-llama/Llama-3.2-1B-Instruct",
- test_class=LlamaModelTest,
- max_layers=16,
- ),
- ModelTestSpec(
- hf_name="Qwen/Qwen2.5-1.5B-Instruct",
- test_class=QwenModelTest,
- max_layers=24,
- ),
- ]
-```
diff --git a/.cursorrules b/.cursorrules
new file mode 120000
index 000000000..95e38a6a9
--- /dev/null
+++ b/.cursorrules
@@ -0,0 +1 @@
+agents/AGENTS.md
\ No newline at end of file
diff --git a/.gemini/skills b/.gemini/skills
new file mode 120000
index 000000000..a769d4bd9
--- /dev/null
+++ b/.gemini/skills
@@ -0,0 +1 @@
+../agents/skills
\ No newline at end of file
diff --git a/.github/skills b/.github/skills
new file mode 120000
index 000000000..a769d4bd9
--- /dev/null
+++ b/.github/skills
@@ -0,0 +1 @@
+../agents/skills
\ No newline at end of file
diff --git a/.gitignore b/.gitignore
index 43da88df2..93142ca42 100644
--- a/.gitignore
+++ b/.gitignore
@@ -1,6 +1,7 @@
*.pdb
**/mutants.out*/
*.DS_Store
+.idea/
# Metal traces
*.gputrace
@@ -15,9 +16,8 @@
*__pycache__
# AI tools
-.github/*.md
.cursor
-.claude
+.claude/settings.local.json
# Models
/models/
diff --git a/AGENTS.md b/AGENTS.md
new file mode 120000
index 000000000..95e38a6a9
--- /dev/null
+++ b/AGENTS.md
@@ -0,0 +1 @@
+agents/AGENTS.md
\ No newline at end of file
diff --git a/Cargo.lock b/Cargo.lock
index 483db6a11..55f571af4 100644
--- a/Cargo.lock
+++ b/Cargo.lock
@@ -363,7 +363,7 @@ dependencies = [
"cc",
"cfg-if",
"constant_time_eq",
- "cpufeatures",
+ "cpufeatures 0.2.17",
]
[[package]]
@@ -437,6 +437,25 @@ dependencies = [
"rustversion",
]
+[[package]]
+name = "cbindgen"
+version = "0.29.2"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "befbfd072a8e81c02f8c507aefce431fe5e7d051f83d48a23ffc9b9fe5a11799"
+dependencies = [
+ "clap",
+ "heck",
+ "indexmap 2.13.0",
+ "log",
+ "proc-macro2",
+ "quote",
+ "serde",
+ "serde_json",
+ "syn 2.0.114",
+ "tempfile",
+ "toml 0.9.6",
+]
+
[[package]]
name = "cc"
version = "1.2.55"
@@ -468,13 +487,24 @@ version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "613afe47fcd5fac7ccf1db93babcb082c5994d996f20b8b159f2ad1658eb5724"
+[[package]]
+name = "chacha20"
+version = "0.10.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "6f8d983286843e49675a4b7a2d174efe136dc93a18d69130dd18198a6c167601"
+dependencies = [
+ "cfg-if",
+ "cpufeatures 0.3.0",
+ "rand_core 0.10.0",
+]
+
[[package]]
name = "clang-ast"
version = "0.1.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2d76fac0a007075fa8d38a6ec08df340bea2035cc0a6e9b0f6594bff1627692b"
dependencies = [
- "foldhash",
+ "foldhash 0.2.0",
"serde_core",
]
@@ -711,6 +741,15 @@ dependencies = [
"libc",
]
+[[package]]
+name = "cpufeatures"
+version = "0.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "8b2a41393f66f16b0823bb79094d54ac5fbd34ab292ddafb9a0456ac9f87d201"
+dependencies = [
+ "libc",
+]
+
[[package]]
name = "crc32fast"
version = "1.5.0"
@@ -810,7 +849,7 @@ dependencies = [
"cxxbridge-cmd",
"cxxbridge-flags",
"cxxbridge-macro",
- "foldhash",
+ "foldhash 0.2.0",
"link-cplusplus",
]
@@ -1165,7 +1204,7 @@ dependencies = [
"atomic 0.6.1",
"pear",
"serde",
- "toml",
+ "toml 0.8.23",
"uncased",
"version_check",
]
@@ -1192,6 +1231,12 @@ version = "1.0.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3f9eec918d3f24069decb9af1554cad7c880e2da24a9afd88aca000531ab82c1"
+[[package]]
+name = "foldhash"
+version = "0.1.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d9c4f5dac5e15c24eb999c26181a6ca40b39fe946cbe4c263c7209467bc83af2"
+
[[package]]
name = "foldhash"
version = "0.2.0"
@@ -1393,6 +1438,20 @@ dependencies = [
"wasip2",
]
+[[package]]
+name = "getrandom"
+version = "0.4.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "139ef39800118c7683f2fd3c98c1b23c09ae076556b435f8e9064ae108aaeeec"
+dependencies = [
+ "cfg-if",
+ "libc",
+ "r-efi",
+ "rand_core 0.10.0",
+ "wasip2",
+ "wasip3",
+]
+
[[package]]
name = "glob"
version = "0.3.3"
@@ -1456,6 +1515,15 @@ version = "0.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a9ee70c43aaf417c914396645a0fa852624801b24ebb7ae78fe8272889ac888"
+[[package]]
+name = "hashbrown"
+version = "0.15.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9229cfe53dfd69f0609a49f65461bd93001ea1ef889cd5529dd176593f5338a1"
+dependencies = [
+ "foldhash 0.1.5",
+]
+
[[package]]
name = "hashbrown"
version = "0.16.1"
@@ -1764,6 +1832,12 @@ dependencies = [
"zerovec",
]
+[[package]]
+name = "id-arena"
+version = "2.3.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "3d3067d79b975e8844ca9eb072e16b31c3c1c36928edf9c6789548c524d0d954"
+
[[package]]
name = "ident_case"
version = "1.0.1"
@@ -1966,6 +2040,12 @@ version = "1.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bbd2bcb4c963f2ddae06a2efc7e9f3591312473c50c6685e1f298068316e66fe"
+[[package]]
+name = "leb128fmt"
+version = "0.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "09edd9e8b54e49e587e4f6295a7d29c3ea94d469cb40ab8ca70b288248a81db2"
+
[[package]]
name = "libc"
version = "0.2.180"
@@ -2848,6 +2928,17 @@ dependencies = [
"rand_core 0.9.5",
]
+[[package]]
+name = "rand"
+version = "0.10.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bc266eb313df6c5c09c1c7b1fbe2510961e5bcd3add930c1e31f7ed9da0feff8"
+dependencies = [
+ "chacha20",
+ "getrandom 0.4.1",
+ "rand_core 0.10.0",
+]
+
[[package]]
name = "rand_chacha"
version = "0.3.1"
@@ -2886,6 +2977,12 @@ dependencies = [
"getrandom 0.3.4",
]
+[[package]]
+name = "rand_core"
+version = "0.10.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0c8d0fd677905edcbeedbf2edb6494d676f0e98d54d5cf9bda0b061cb8fb8aba"
+
[[package]]
name = "rawpointer"
version = "0.2.1"
@@ -3410,6 +3507,15 @@ dependencies = [
"serde",
]
+[[package]]
+name = "serde_spanned"
+version = "1.0.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "f8bbf91e5a4d6315eee45e704372590b30e260ee83af6639d64557f51b067776"
+dependencies = [
+ "serde_core",
+]
+
[[package]]
name = "serde_urlencoded"
version = "0.7.1"
@@ -3429,7 +3535,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a7507d819769d01a365ab707794a4084392c824f54a7a6a7862f8c3d0892b283"
dependencies = [
"cfg-if",
- "cpufeatures",
+ "cpufeatures 0.2.17",
"digest",
]
@@ -3890,11 +3996,26 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dc1beb996b9d83529a9e75c17a1686767d148d70663143c7854d8b4a09ced362"
dependencies = [
"serde",
- "serde_spanned",
- "toml_datetime",
+ "serde_spanned 0.6.9",
+ "toml_datetime 0.6.11",
"toml_edit",
]
+[[package]]
+name = "toml"
+version = "0.9.6"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ae2a4cf385da23d1d53bc15cdfa5c2109e93d8d362393c801e87da2f72f0e201"
+dependencies = [
+ "indexmap 2.13.0",
+ "serde_core",
+ "serde_spanned 1.0.4",
+ "toml_datetime 0.7.5+spec-1.1.0",
+ "toml_parser",
+ "toml_writer",
+ "winnow",
+]
+
[[package]]
name = "toml_datetime"
version = "0.6.11"
@@ -3904,6 +4025,15 @@ dependencies = [
"serde",
]
+[[package]]
+name = "toml_datetime"
+version = "0.7.5+spec-1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "92e1cfed4a3038bc5a127e35a2d360f145e1f4b971b551a2ba5fd7aedf7e1347"
+dependencies = [
+ "serde_core",
+]
+
[[package]]
name = "toml_edit"
version = "0.22.27"
@@ -3912,18 +4042,33 @@ checksum = "41fe8c660ae4257887cf66394862d21dbca4a6ddd26f04a3560410406a2f819a"
dependencies = [
"indexmap 2.13.0",
"serde",
- "serde_spanned",
- "toml_datetime",
+ "serde_spanned 0.6.9",
+ "toml_datetime 0.6.11",
"toml_write",
"winnow",
]
+[[package]]
+name = "toml_parser"
+version = "1.0.6+spec-1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a3198b4b0a8e11f09dd03e133c0280504d0801269e9afa46362ffde1cbeebf44"
+dependencies = [
+ "winnow",
+]
+
[[package]]
name = "toml_write"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d99f8c9a7727884afe522e9bd5edbfc91a3312b36a77b5fb8926e4c31a41801"
+[[package]]
+name = "toml_writer"
+version = "1.0.6+spec-1.1.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ab16f14aed21ee8bfd8ec22513f7287cd4a91aa92e44edfe2c17ddd004e92607"
+
[[package]]
name = "tower"
version = "0.5.3"
@@ -4190,6 +4335,7 @@ dependencies = [
"blake3",
"block2",
"bytemuck",
+ "cbindgen",
"clang-ast",
"clap",
"depfile",
@@ -4209,7 +4355,7 @@ dependencies = [
"prettyplease",
"proc-macro2",
"quote",
- "rand 0.9.2",
+ "rand 0.10.0",
"regex",
"schemars",
"serde",
@@ -4277,6 +4423,15 @@ dependencies = [
"wit-bindgen",
]
+[[package]]
+name = "wasip3"
+version = "0.4.0+wasi-0.3.0-rc-2026-01-06"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5428f8bf88ea5ddc08faddef2ac4a67e390b88186c703ce6dbd955e1c145aca5"
+dependencies = [
+ "wit-bindgen",
+]
+
[[package]]
name = "wasm-bindgen"
version = "0.2.108"
@@ -4336,6 +4491,28 @@ dependencies = [
"unicode-ident",
]
+[[package]]
+name = "wasm-encoder"
+version = "0.244.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "990065f2fe63003fe337b932cfb5e3b80e0b4d0f5ff650e6985b1048f62c8319"
+dependencies = [
+ "leb128fmt",
+ "wasmparser",
+]
+
+[[package]]
+name = "wasm-metadata"
+version = "0.244.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "bb0e353e6a2fbdc176932bbaab493762eb1255a7900fe0fea1a2f96c296cc909"
+dependencies = [
+ "anyhow",
+ "indexmap 2.13.0",
+ "wasm-encoder",
+ "wasmparser",
+]
+
[[package]]
name = "wasm-streams"
version = "0.4.2"
@@ -4349,6 +4526,18 @@ dependencies = [
"web-sys",
]
+[[package]]
+name = "wasmparser"
+version = "0.244.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "47b807c72e1bac69382b3a6fb3dbe8ea4c0ed87ff5629b8685ae6b9a611028fe"
+dependencies = [
+ "bitflags",
+ "hashbrown 0.15.5",
+ "indexmap 2.13.0",
+ "semver",
+]
+
[[package]]
name = "web-sys"
version = "0.3.85"
@@ -4784,6 +4973,88 @@ name = "wit-bindgen"
version = "0.51.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d7249219f66ced02969388cf2bb044a09756a083d0fab1e566056b04d9fbcaa5"
+dependencies = [
+ "wit-bindgen-rust-macro",
+]
+
+[[package]]
+name = "wit-bindgen-core"
+version = "0.51.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ea61de684c3ea68cb082b7a88508a8b27fcc8b797d738bfc99a82facf1d752dc"
+dependencies = [
+ "anyhow",
+ "heck",
+ "wit-parser",
+]
+
+[[package]]
+name = "wit-bindgen-rust"
+version = "0.51.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b7c566e0f4b284dd6561c786d9cb0142da491f46a9fbed79ea69cdad5db17f21"
+dependencies = [
+ "anyhow",
+ "heck",
+ "indexmap 2.13.0",
+ "prettyplease",
+ "syn 2.0.114",
+ "wasm-metadata",
+ "wit-bindgen-core",
+ "wit-component",
+]
+
+[[package]]
+name = "wit-bindgen-rust-macro"
+version = "0.51.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "0c0f9bfd77e6a48eccf51359e3ae77140a7f50b1e2ebfe62422d8afdaffab17a"
+dependencies = [
+ "anyhow",
+ "prettyplease",
+ "proc-macro2",
+ "quote",
+ "syn 2.0.114",
+ "wit-bindgen-core",
+ "wit-bindgen-rust",
+]
+
+[[package]]
+name = "wit-component"
+version = "0.244.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "9d66ea20e9553b30172b5e831994e35fbde2d165325bec84fc43dbf6f4eb9cb2"
+dependencies = [
+ "anyhow",
+ "bitflags",
+ "indexmap 2.13.0",
+ "log",
+ "serde",
+ "serde_derive",
+ "serde_json",
+ "wasm-encoder",
+ "wasm-metadata",
+ "wasmparser",
+ "wit-parser",
+]
+
+[[package]]
+name = "wit-parser"
+version = "0.244.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "ecc8ac4bc1dc3381b7f59c34f00b67e18f910c2c0f50015669dde7def656a736"
+dependencies = [
+ "anyhow",
+ "id-arena",
+ "indexmap 2.13.0",
+ "log",
+ "semver",
+ "serde",
+ "serde_derive",
+ "serde_json",
+ "unicode-xid",
+ "wasmparser",
+]
[[package]]
name = "writeable"
diff --git a/Cargo.toml b/Cargo.toml
index 6ca8ad1db..b6c8e1160 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -33,7 +33,7 @@ bitflags = "2.10"
minijinja = "2.15"
minijinja-contrib = { version = "2.15", features = ["pycompat"] }
clap = { version = "4.5", features = ["derive"] }
-rand = "0.9"
+rand = "0.10"
sysinfo = "0.38"
itertools = "0.14"
schemars = { version = "1.2", features = ["derive"] }
@@ -54,6 +54,7 @@ proc-macro2 = "1.0"
syn = "2.0"
prettyplease = "0.2"
mach2 = "0.6"
+cbindgen = "0.29"
ctrlc = "3.5"
indicatif = "0.18"
inquire = "0.9"
diff --git a/agents/AGENTS.md b/agents/AGENTS.md
new file mode 100644
index 000000000..de13df6ac
--- /dev/null
+++ b/agents/AGENTS.md
@@ -0,0 +1,61 @@
+# Rules
+
+The rules below are CRITICAL. Each rule should be respected. Rules can only be overridden by the user.
+
+## Workflow rules
+
+- Agent should use provided skills, especially the architecture description.
+- When given a coding task, agent carefully and deeply inspects the codebase to understand the context.
+- Agent doesn't ask for permission to read the contents of files and directories.
+- When given a task, agent first thinks about multiple high-level strategies, and then asks the user for feedback on every option.
+- Agent splits tasks into smallest possible subtasks, and solves them one by one while asking the user for detailed feedback during every step.
+- Agent never edits code before proposing a short draft to the user and receiving a feedback.
+- Agent is not allowed to make even the tiniest architectural decisions, it can only propose multiple options to the user.
+- Agent behaves like an interviewer and asks the user questions all the time.
+- Instead of asking multiple questions in one message, agent asks them one by one.
+- Agent always asks for permission before launching CLI tools which may take lots of time or produce side effects.
+- Before using any function from an external library, agent thoroughly inspects the code or documentation for the function to ensure that it uses it correctly.
+- When encountering an error, agent never rushes to quickly fix it. Instead, it uses the scientific method to understand the source of the error: comes up with hypotheses and tests them by performing a series of experiments. It only attempts to fix the error once the hypothesis has been sufficiently validated.
+- Agent avoids creating .md files with explanations unless explicitly requested by the user.
+
+## General coding guidelines
+
+- Agent never resorts to quick hacks or stubs to make something work. Instead it always investigates the problem thoroughly and comes up with a well-thought-out solution.
+- By default, agent does not write documentation or comments unless the code contains complex logic which is difficult to understand without them.
+- Agent strives for simplicity and maintainability. Agent avoids writing boilerplate code, instead it tries to come up with the right abstractions.
+- Agent prefers functional programming style, namely:
+ - Avoids mutable state.
+ - Avoids modifying objects in-place.
+ - Avoids implicit side effects.
+ - Prefers immutable data structures.
+ - Strives to make invalid state unrepresentable.
+- Agent tries to write code in the way that makes bugs hard to make and easy to spot.
+- Agent writes the code so that assumptions about invariants are as explicit as possible.
+- Agent leverages strong typing as much as possible, for example, it always uses enums instead of literals.
+- Agent uses good descriptive variable names that make it easy to understand the purpose of the variable without looking at the code that uses it.
+- Agent does not manually parse things such as JSON configs. Instead it first models a schema using types, and then uses a serialisation library.
+
+## Testing guidelines
+
+- Agent strives for simplicity and minimalism when writing tests.
+- Agent avoids extensive mocking and stubbing.
+- Agent focuses on testing core functionality, and does not test minor details such as property accessors.
+- Agent avoids boilerplate in the testing code, and tries to come up with the right generic abstractions.
+
+## Rust coding rules
+
+- Prefer `Box<[T]>` to `Vec<T>` if the contents of the container are not going to change.
+- Avoid mutable variables as much as possible. Use iterator expressions to collect data into containers.
+- For calling API that was never used before, read the sources of the installed dependencies, otherwise, search in docs.rs.
+
+## Python coding rules
+
+- Never use pip to manage dependencies. Instead use uv commands, such as `uv add`.
+- Assume Python 3.12. Use `T | None` instead of `Optional[T]` and `dict[K, V]` instead of `typing.Dict[K, V]`.
+- Every function should have full type annotations.
+- Prefer comprehensions to map and filter expressions.
+- Prefer dataclasses over vanilla python classes.
+- All path-like arguments should have type `Path | str`.
+- Prefer frozen dataclasses or named tuples as primary data structures.
+- Never use dicts in place of dataclasses.
+- __init__.py files should only contain reexports.
\ No newline at end of file
diff --git a/agents/skills/architecture/SKILL.md b/agents/skills/architecture/SKILL.md
new file mode 100644
index 000000000..c3623d9b3
--- /dev/null
+++ b/agents/skills/architecture/SKILL.md
@@ -0,0 +1,26 @@
+---
+name: architecture
+description: Description of the main concepts of the inference engine
+---
+
+Read this carefully before doing any changes:
+
+## Architecture
+
+1. uzu runs models converted via https://github.com/trymirai/lalamo toolkit in unified format.
+
+2. Session. High-level object used to perform requests to the model. It supports multiple modes (ignore previous context, use static context to perform requests, and dynamically aggregate context). It creates a generator object which implements the decoding loop, manually calls prefill and generate steps, and collects performance stats. Decoding can be a fully GPU-driven pipeline or can use CPU sync after each token when needed (speculative and constrained decoding).
+
+3. Generator. Object that implements forward pass execution for prefill and decode stages. It also implements async decoding loop logic. It creates all necessary buffers, loads weights into memory, prepares masks, speculative suffixes, and grammar masks if needed. For sync loop it also implements preencode logic.
+
+4. Encodables. High-level objects that implement kernel selection. Each encodable has an encode method and depends on the corresponding config object from the model's config.json.
+
+5. Kernels. Backend-specific shaders that use DSL from ./crates/uzu/build to automatically generate Rust interfaces. Each kernel should have a corresponding unit test that validates correctness.
+
+## What is important?
+
+1. Performance is everything. All changes should be as efficient as possible in terms of prefill/decode speed and memory usage.
+
+2. Keep in mind that this is an SDK that will be used by a wide range of developers, so excellent developer experience is crucial.
+
+3. You can explore various docs here: https://docs.trymirai.com
\ No newline at end of file
diff --git a/agents/skills/benchmark-model/SKILL.md b/agents/skills/benchmark-model/SKILL.md
new file mode 100644
index 000000000..aef2f13bd
--- /dev/null
+++ b/agents/skills/benchmark-model/SKILL.md
@@ -0,0 +1,19 @@
+---
+name: benchmark-model
+description: Benchmark inference performance for a specific model
+---
+
+Using CLI to perform benchmark.
+
+## Instructions
+
+1. Choose model you want to use from ./models/{ENGINE_VERSION}/{MODEL_NAME} or download it
+
+2. Ensure benchmark_task.json exists in the model folder (it is automatically generated after model download)
+
+3. Run
+ ```
+ cargo run --release -p cli -- bench ./models/{ENGINE_VERSION}/{MODEL_NAME} ./models/{ENGINE_VERSION}/{MODEL_NAME}/benchmark_task.json {RESULT_PATH}
+ ```
+
+4. Check result json to check memory_used, time_to_first_token, prompt_tokens_per_second and generate_tokens_per_second. Optionally check text is readable.
diff --git a/agents/skills/download-model/SKILL.md b/agents/skills/download-model/SKILL.md
new file mode 100644
index 000000000..83ee3d91a
--- /dev/null
+++ b/agents/skills/download-model/SKILL.md
@@ -0,0 +1,20 @@
+---
+name: download-model
+description: Download model to test inference engine
+---
+
+Using ./tools/helpers Python package to download model
+
+## Instructions
+
+1. Go to helpers folder:
+ ```
+ cd ./tools/helpers
+ ```
+
+2. Run:
+ ```
+ uv run main.py download-model {REPO_ID}
+ ```
+
+3. Downloaded model will be in ./models/{ENGINE_VERSION}/{MODEL_NAME} folder
diff --git a/agents/skills/list-downloaded-models/SKILL.md b/agents/skills/list-downloaded-models/SKILL.md
new file mode 100644
index 000000000..81d0c8ba7
--- /dev/null
+++ b/agents/skills/list-downloaded-models/SKILL.md
@@ -0,0 +1,8 @@
+---
+name: list-downloaded-models
+description: Get list of currently downloaded models
+---
+
+## Instructions
+
+1. List folders inside ./models/{ENGINE_VERSION}, where each folder name is a model name
\ No newline at end of file
diff --git a/agents/skills/list-models/SKILL.md b/agents/skills/list-models/SKILL.md
new file mode 100644
index 000000000..163fc9740
--- /dev/null
+++ b/agents/skills/list-models/SKILL.md
@@ -0,0 +1,20 @@
+---
+name: list-models
+description: Get list of models supported by inference engine
+---
+
+Using ./tools/helpers Python package to list supported models
+
+## Instructions
+
+1. Go to helpers folder:
+ ```
+ cd ./tools/helpers
+ ```
+
+2. Run:
+ ```
+ uv run main.py list-models
+ ```
+
+3. Choose repo you want to use
diff --git a/agents/skills/trace-model/SKILL.md b/agents/skills/trace-model/SKILL.md
new file mode 100644
index 000000000..fa94e63e0
--- /dev/null
+++ b/agents/skills/trace-model/SKILL.md
@@ -0,0 +1,20 @@
+---
+name: trace-model
+description: Validate all intermediate calculations during inference are correct using source of truth values
+---
+
+Using traces.safetensors to validate all intermediate calculations after each layer are correct.
+
+## Instructions
+
+1. Setup model you want to test in ./crates/uzu/tests/tracer_test.rs or ./crates/uzu/tests/mod.rs
+
+2. Check the model folder to ensure traces.safetensors exists (it should be automatically downloaded alongside the model)
+
+3. Run
+
+ ```
+ cargo test --release --package uzu --test tracer_test -- test_tracer --exact --nocapture
+ ```
+
+4. Check results to ensure number of violations are less than the limit
\ No newline at end of file
diff --git a/agents/skills/validate-inference/SKILL.md b/agents/skills/validate-inference/SKILL.md
new file mode 100644
index 000000000..322deaa14
--- /dev/null
+++ b/agents/skills/validate-inference/SKILL.md
@@ -0,0 +1,20 @@
+---
+name: validate-inference
+description: Validate inference generates correct text
+---
+
+Using tests to validate inference is correct and performance has not degraded.
+
+## Instructions
+
+1. Setup model you want to test in ./crates/uzu/tests/text_session_test.rs or ./crates/uzu/tests/mod.rs
+
+2. Remember that some models can be too large to fit in device RAM. Do not use these models and do not attempt to download them.
+
+3. Run
+
+ ```
+ cargo test --release --package uzu --test text_session_test -- test_text_session_base --exact --nocapture
+ ```
+
+4. Check generated text is correct and readable and stats (processed_tokens_per_second for prefill and tokens_per_second for generate)
\ No newline at end of file
diff --git a/crates/cli/src/handlers/run.rs b/crates/cli/src/handlers/run.rs
index 9c6d409c4..651a8d4c5 100644
--- a/crates/cli/src/handlers/run.rs
+++ b/crates/cli/src/handlers/run.rs
@@ -37,8 +37,10 @@ pub fn handle_run(
model_path: String,
tokens_limit: usize,
prefill_step_size: Option<usize>,
+ seed: Option<u64>,
+ mut message: Option<String>,
) {
- let mut session = load_session(model_path, prefill_step_size);
+ let mut session = load_session(model_path, prefill_step_size, seed);
let is_model_running = Arc::new(AtomicBool::new(false));
let is_model_running_for_ctrlc = is_model_running.clone();
@@ -49,15 +51,23 @@ pub fn handle_run(
})
.unwrap();
+ let non_interactive = message.is_some();
+
loop {
- let input =
+ let input = if let Some(msg) = message.take() {
+ msg
+ } else {
match Text::new("").with_placeholder("Send a message").prompt() {
Ok(input) => input,
Err(_) => {
break;
},
- };
+ }
+ };
if input.is_empty() {
+ if non_interactive {
+ break;
+ }
continue;
}
@@ -117,5 +127,9 @@ pub fn handle_run(
progress_bar.finish_and_clear();
println!("{}", result);
is_model_running.store(false, Ordering::SeqCst);
+
+ if non_interactive {
+ break;
+ }
}
}
diff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rs
index 63a44015d..a995223d7 100644
--- a/crates/cli/src/main.rs
+++ b/crates/cli/src/main.rs
@@ -15,6 +15,12 @@ enum Commands {
model_path: String,
/// Prefill step size
prefill_step_size: Option<usize>,
+ // Seed
+ #[arg(long)]
+ seed: Option<u64>,
+ /// Non-interactive mode: run a single message and exit
+ #[arg(long, short)]
+ message: Option<String>,
},
/// Start a server with the specified model path
Serve {
@@ -41,8 +47,10 @@ fn main() {
Some(Commands::Run {
model_path,
prefill_step_size,
+ seed,
+ message,
}) => {
- handle_run(model_path, 2048, prefill_step_size);
+ handle_run(model_path, 2048, prefill_step_size, seed, message);
},
Some(Commands::Serve {
model_path,
diff --git a/crates/cli/src/server/main.rs b/crates/cli/src/server/main.rs
index f0d46c414..3a2bc8ba0 100644
--- a/crates/cli/src/server/main.rs
+++ b/crates/cli/src/server/main.rs
@@ -56,7 +56,7 @@ pub async fn run_server(
"📝 Endpoints:\n POST /chat/completions - Chat completions API\n"
);
- let session = load_session(model_path, prefill_step_size);
+ let session = load_session(model_path, prefill_step_size, None);
let state = SessionState {
model_name,
session_wrapper: SessionWrapper::new(session),
diff --git a/crates/cli/src/server/state.rs b/crates/cli/src/server/state.rs
index c31980ef7..53717e3bc 100644
--- a/crates/cli/src/server/state.rs
+++ b/crates/cli/src/server/state.rs
@@ -2,8 +2,9 @@ use std::{path::PathBuf, sync::Mutex};
use console::Style;
use indicatif::{ProgressBar, ProgressStyle};
-use uzu::session::{
- Session, config::DecodingConfig, parameter::PrefillStepSize,
+use uzu::{
+ prelude::SamplingSeed,
+ session::{Session, config::DecodingConfig, parameter::PrefillStepSize},
};
pub struct SessionWrapper(Mutex<Session>);
@@ -30,6 +31,7 @@ unsafe impl Sync for SessionState {}
pub fn load_session(
model_path: String,
prefill_step_size: Option<usize>,
+ seed: Option<u64>,
) -> Session {
let style_bold = Style::new().bold();
@@ -57,7 +59,11 @@ pub fn load_session(
}
let decoding_config = DecodingConfig::default()
- .with_prefill_step_size(prefill_step_size_config);
+ .with_prefill_step_size(prefill_step_size_config)
+ .with_sampling_seed(match seed {
+ Some(seed) => SamplingSeed::Custom(seed),
+ None => SamplingSeed::Default,
+ });
let session = Session::new(model_path_buf, decoding_config)
.expect("Failed to create session");
diff --git a/crates/uzu/Cargo.toml b/crates/uzu/Cargo.toml
index be05be695..89cfb7d9a 100644
--- a/crates/uzu/Cargo.toml
+++ b/crates/uzu/Cargo.toml
@@ -60,6 +60,7 @@ proc-macro2.workspace = true
syn.workspace = true
prettyplease.workspace = true
itertools.workspace = true
+cbindgen.workspace = true
# metal feature requires clang-ast
clang-ast = { workspace = true, optional = true }
diff --git a/crates/uzu/build/common/codegen.rs b/crates/uzu/build/common/codegen.rs
new file mode 100644
index 000000000..da55239a6
--- /dev/null
+++ b/crates/uzu/build/common/codegen.rs
@@ -0,0 +1,24 @@
+use std::{ffi::OsStr, fs};
+
+use anyhow::Context;
+use proc_macro2::TokenStream;
+
+pub fn write_tokens(
+ tokens: impl Into<TokenStream>,
+ file: impl AsRef<OsStr>,
+) -> anyhow::Result<()> {
+ let tokens = tokens.into();
+ let file = file.as_ref();
+
+ let parsed = syn::parse2(tokens.into())
+ .context("cannot parse generated bindings")?;
+ fs::write(&file, prettyplease::unparse(&parsed))
+ .with_context(|| format!("cannot write file {}", file.display()))?;
+
+ std::process::Command::new("rustfmt")
+ .arg(&file)
+ .status()
+ .context("rustfmt failed")?;
+
+ Ok(())
+}
diff --git a/crates/uzu/build/common/compiler.rs b/crates/uzu/build/common/compiler.rs
index 5f402a2ed..109c81b17 100644
--- a/crates/uzu/build/common/compiler.rs
+++ b/crates/uzu/build/common/compiler.rs
@@ -1,6 +1,12 @@
+use std::collections::HashMap;
+
use async_trait::async_trait;
+use super::kernel::Kernel;
+
#[async_trait]
pub trait Compiler {
- async fn build(&self) -> anyhow::Result<()>;
+ async fn build(
+ &self
+ ) -> anyhow::Result<HashMap<Box<[Box<str>]>, Box<[Kernel]>>>;
}
diff --git a/crates/uzu/build/common/kernel.rs b/crates/uzu/build/common/kernel.rs
new file mode 100644
index 000000000..8c08c3f71
--- /dev/null
+++ b/crates/uzu/build/common/kernel.rs
@@ -0,0 +1,31 @@
+#[derive(PartialEq, Debug)]
+pub enum KernelArgumentType {
+ Buffer,
+ Constant(Box<str>),
+ Scalar(Box<str>),
+}
+
+#[derive(PartialEq, Debug)]
+pub struct KernelArgument {
+ pub name: Box<str>,
+ pub ty: KernelArgumentType,
+}
+
+#[derive(PartialEq, Debug)]
+pub enum KernelParameterType {
+ DType,
+ Specialization(Box<str>),
+}
+
+#[derive(PartialEq, Debug)]
+pub struct KernelParameter {
+ pub name: Box<str>,
+ pub ty: KernelParameterType,
+}
+
+#[derive(PartialEq, Debug)]
+pub struct Kernel {
+ pub name: Box<str>,
+ pub parameters: Box<[KernelParameter]>,
+ pub arguments: Box<[KernelArgument]>,
+}
diff --git a/crates/uzu/build/common/mod.rs b/crates/uzu/build/common/mod.rs
index 3b41dfd7f..a6c6718d7 100644
--- a/crates/uzu/build/common/mod.rs
+++ b/crates/uzu/build/common/mod.rs
@@ -1,4 +1,7 @@
pub mod caching;
+pub mod codegen;
pub mod compiler;
pub mod envs;
+pub mod kernel;
pub mod logging;
+pub mod traitgen;
diff --git a/crates/uzu/build/common/traitgen.rs b/crates/uzu/build/common/traitgen.rs
new file mode 100644
index 000000000..850be99b7
--- /dev/null
+++ b/crates/uzu/build/common/traitgen.rs
@@ -0,0 +1,121 @@
+use std::{collections::HashMap, env, path::PathBuf};
+
+use anyhow::{Context, bail};
+use itertools::Itertools;
+use proc_macro2::TokenStream;
+use quote::{format_ident, quote};
+
+use crate::common::{
+ codegen::write_tokens,
+ kernel::{KernelArgumentType, KernelParameterType},
+};
+
+use super::kernel::Kernel;
+
+pub fn traitgen(kernel: &Kernel) -> (TokenStream, TokenStream) {
+ let kernel_name = kernel.name.as_ref();
+ let trait_name = format_ident!("{kernel_name}Kernel");
+
+ let params = kernel.parameters.iter().map(|p| {
+ let name = format_ident!("{}", p.name.as_ref());
+ let ty = match &p.ty {
+ KernelParameterType::DType => quote! { DataType },
+ KernelParameterType::Specialization(ty) => {
+ let ty = format_ident!("{}", ty.as_ref());
+ quote! { #ty }
+ },
+ };
+
+ quote! { #name: #ty }
+ });
+
+ let args = kernel
+ .arguments
+ .iter()
+ .map(|a| {
+ let name = format_ident!("{}", a.name.as_ref());
+ let ty = match &a.ty {
+ KernelArgumentType::Buffer => {
+ quote! { &<Self::Backend as Backend>::NativeBuffer }
+ },
+ KernelArgumentType::Constant(ty) => {
+ let ty = format_ident!("{}", ty.as_ref());
+ quote! { &[#ty] }
+ },
+ KernelArgumentType::Scalar(ty) => {
+ let ty = format_ident!("{}", ty.as_ref());
+ quote! { #ty }
+ },
+ };
+
+ quote! { #name: #ty }
+ })
+ .collect::<Vec<_>>();
+
+ let kernel_trait = quote! {
+ pub trait #trait_name: Sized {
+ type Backend: Backend<Kernels: Kernels<#trait_name = Self>>;
+
+ fn new(context: &<Self::Backend as Backend>::Context #(, #params)*) -> Result<Self, <Self::Backend as Backend>::Error>;
+
+ fn encode(&self, #(#args ,)* encoder: &<Self::Backend as Backend>::EncoderRef);
+ fn encode_if(&self, #(#args ,)* encoder: &<Self::Backend as Backend>::EncoderRef, predicate: Option<&<Self::Backend as Backend>::NativeBuffer>);
+ }
+ };
+
+ let kernel_type =
+ quote! { type #trait_name: #trait_name<Backend = Self::Backend>; };
+
+ (kernel_trait, kernel_type)
+}
+
+pub fn traitgen_all(
+ backends_kernels: Vec<HashMap<Box<[Box<str>]>, Box<[Kernel]>>>
+) -> anyhow::Result<()> {
+ let out_dir =
+ PathBuf::from(env::var("OUT_DIR").context("missing OUT_DIR")?);
+
+ let mut kernels: HashMap<Box<[Box<str>]>, Box<[Kernel]>> = HashMap::new();
+
+ for backend_kernels in backends_kernels {
+ for (file_path, file_kernels) in backend_kernels {
+ if let Some(cached_kernels) = kernels.get(&file_path) {
+ if cached_kernels != &file_kernels {
+ bail!("{cached_kernels:?} != {file_kernels:?}");
+ }
+ } else {
+ kernels.insert(file_path, file_kernels);
+ }
+ }
+ }
+
+ let mut kernel_traits = Vec::new();
+ let mut kernel_types = Vec::new();
+
+ for (_file_path, file_kernels) in
+ kernels.into_iter().sorted_by_key(|(p, _k)| p.join("::"))
+ {
+ for (tr, ty) in file_kernels.iter().map(traitgen) {
+ kernel_traits.push(tr);
+ kernel_types.push(ty);
+ }
+ }
+
+ let kernel_traits = quote! {
+ use crate::backends::common::Backend;
+ use crate::DataType;
+
+ #(#kernel_traits)*
+
+ pub trait Kernels: Sized {
+ type Backend: Backend<Kernels = Self>;
+
+ #(#kernel_types)*
+ }
+ };
+
+ write_tokens(kernel_traits, out_dir.join("traits.rs"))
+ .context("cannot write kernel traits")?;
+
+ Ok(())
+}
diff --git a/crates/uzu/build/gpu_types/mod.rs b/crates/uzu/build/gpu_types/mod.rs
new file mode 100644
index 000000000..19baafcdc
--- /dev/null
+++ b/crates/uzu/build/gpu_types/mod.rs
@@ -0,0 +1,325 @@
+use std::{collections::HashMap, env, fs, path::PathBuf};
+
+use anyhow::Context;
+use async_trait::async_trait;
+use syn::{
+ Expr, ExprLit, Fields, Item, ItemStruct, Lit, Meta, Type, TypeArray,
+ TypePath,
+};
+use tokio::task::spawn_blocking;
+
+use crate::common::{compiler::Compiler, kernel::Kernel};
+use crate::debug_log;
+
+#[derive(Debug)]
+pub struct GpuTypesCompiler {
+ crate_dir: PathBuf,
+ generated_dir: PathBuf,
+ build_dir: PathBuf,
+}
+
+impl GpuTypesCompiler {
+ pub fn new() -> anyhow::Result<Self> {
+ let crate_dir = PathBuf::from(
+ env::var("CARGO_MANIFEST_DIR")
+ .context("missing CARGO_MANIFEST_DIR")?,
+ );
+
+ let generated_dir = crate_dir.join("src/backends/metal/generated");
+ fs::create_dir_all(&generated_dir).with_context(|| {
+ format!("cannot create {}", generated_dir.display())
+ })?;
+
+ let build_dir =
+ PathBuf::from(env::var("OUT_DIR").context("missing OUT_DIR")?)
+ .join("gpu_types");
+ fs::create_dir_all(&build_dir).with_context(|| {
+ format!("cannot create {}", build_dir.display())
+ })?;
+
+ Ok(Self {
+ crate_dir,
+ generated_dir,
+ build_dir,
+ })
+ }
+
+ pub fn generated_header_dir(&self) -> &PathBuf {
+ &self.generated_dir
+ }
+
+ fn generate(&self) -> anyhow::Result<()> {
+ let gpu_types_dir =
+ self.crate_dir.join("src/backends/common/gpu_types");
+
+ let source_files: Vec<PathBuf> = fs::read_dir(&gpu_types_dir)
+ .with_context(|| {
+ format!("cannot read {}", gpu_types_dir.display())
+ })?
+ .filter_map(|entry| entry.ok())
+ .map(|entry| entry.path())
+ .filter(|path| {
+ path.extension().map(|ext| ext == "rs").unwrap_or(false)
+ && path.file_stem() != Some("mod".as_ref())
+ })
+ .collect();
+
+ let mut hasher = blake3::Hasher::new();
+ for path in &source_files {
+ let content = fs::read(path)
+ .with_context(|| format!("cannot read {}", path.display()))?;
+ hasher.update(&content);
+ }
+
+ let source_hash: [u8; blake3::OUT_LEN] = hasher.finalize().into();
+ let cached_hash_path = self.build_dir.join("gpu_types.hash");
+
+ if let Ok(cached_hash) = fs::read(&cached_hash_path) {
+ if cached_hash == source_hash {
+ debug_log!("gpu_types cached");
+ return Ok(());
+ }
+ }
+
+ debug_log!("gpu_types generation started");
+
+ for src_path in &source_files {
+ self.generate_header_for_file(src_path)?;
+ }
+
+ fs::write(&cached_hash_path, &source_hash).with_context(|| {
+ format!("cannot write {}", cached_hash_path.display())
+ })?;
+
+ debug_log!("gpu_types generation done");
+
+ Ok(())
+ }
+
+ fn generate_header_for_file(
+ &self,
+ src_path: &PathBuf,
+ ) -> anyhow::Result<()> {
+ let module_name = src_path
+ .file_stem()
+ .and_then(|s| s.to_str())
+ .context("invalid source file name")?;
+
+ let source = fs::read_to_string(src_path)
+ .with_context(|| format!("cannot read {}", src_path.display()))?;
+
+ let syntax = syn::parse_file(&source)
+ .with_context(|| format!("cannot parse {}", src_path.display()))?;
+
+ let mut repr_c_structs: Vec<ItemStruct> = Vec::new();
+ for item in syntax.items {
+ if let Item::Struct(item_struct) = item {
+ let has_repr_c = item_struct.attrs.iter().any(|attribute| {
+ if attribute.path().is_ident("repr") {
+ if let Ok(ident) = attribute.parse_args::<syn::Ident>()
+ {
+ return ident == "C";
+ }
+ }
+ false
+ });
+
+ if has_repr_c {
+ repr_c_structs.push(item_struct);
+ }
+ }
+ }
+
+ let mut c_struct_definitions = String::new();
+ for rust_struct in &repr_c_structs {
+ for attribute in &rust_struct.attrs {
+ if attribute.path().is_ident("doc") {
+ if let Meta::NameValue(meta_name_value) = &attribute.meta {
+ if let Expr::Lit(ExprLit {
+ lit: Lit::Str(literal_string),
+ ..
+ }) = &meta_name_value.value
+ {
+ let doc_comment = literal_string.value();
+ let doc_comment = doc_comment.trim();
+ if !doc_comment.is_empty() {
+ c_struct_definitions.push_str("/**");
+ c_struct_definitions.push_str(doc_comment);
+ c_struct_definitions.push_str(" */\n");
+ }
+ }
+ }
+ }
+ }
+
+ c_struct_definitions.push_str("typedef struct {\n");
+
+ if let Fields::Named(named_fields) = &rust_struct.fields {
+ for field in &named_fields.named {
+ let field_name = field
+ .ident
+ .as_ref()
+ .map(|ident| ident.to_string())
+ .unwrap_or_default();
+ let c_type = rust_type_to_c(&field.ty);
+
+ for attribute in &field.attrs {
+ if attribute.path().is_ident("doc") {
+ if let Meta::NameValue(meta_name_value) =
+ &attribute.meta
+ {
+ if let Expr::Lit(ExprLit {
+ lit: Lit::Str(literal_string),
+ ..
+ }) = &meta_name_value.value
+ {
+ let doc_comment = literal_string.value();
+ let doc_comment = doc_comment.trim();
+ if !doc_comment.is_empty() {
+ c_struct_definitions.push_str(" /**");
+ c_struct_definitions
+ .push_str(doc_comment);
+ c_struct_definitions.push_str(" */\n");
+ }
+ }
+ }
+ }
+ }
+
+ if let Some((base_type, array_size)) =
+ c_type.split_once('[')
+ {
+ c_struct_definitions.push_str(&format!(
+ " {base_type} {field_name}[{array_size};\n"
+ ));
+ } else {
+ c_struct_definitions
+ .push_str(&format!(" {c_type} {field_name};\n"));
+ }
+ }
+ }
+
+ let struct_name = rust_struct.ident.to_string();
+ c_struct_definitions.push_str(&format!("}} {struct_name};\n\n"));
+ }
+
+ self.write_header(module_name, c_struct_definitions.trim())?;
+
+ Ok(())
+ }
+
+ fn write_header(
+ &self,
+ module_name: &str,
+ structs: &str,
+ ) -> anyhow::Result<()> {
+ let include_guard = format!("UZU_{}_H", module_name.to_uppercase());
+
+ let header_content = format!(
+ r#"// Auto-generated from gpu_types/{module_name}.rs - do not edit manually
+#pragma once
+
+#ifndef {include_guard}
+#define {include_guard}
+
+#ifdef __METAL_VERSION__
+#include <metal_stdlib>
+using namespace metal;
+
+namespace uzu {{
+namespace {module_name} {{
+#else
+#include <stdint.h>
+#endif
+
+{structs}
+
+#ifdef __METAL_VERSION__
+}} // namespace {module_name}
+}} // namespace uzu
+#endif
+
+#endif // {include_guard}
+"#
+ );
+
+ let output_path = self.generated_dir.join(format!("{module_name}.h"));
+ fs::write(&output_path, &header_content).with_context(|| {
+ format!("cannot write {}", output_path.display())
+ })?;
+
+ debug_log!(" -> {}", output_path.display());
+
+ Ok(())
+ }
+}
+
+fn rust_type_to_c(rust_type: &Type) -> String {
+ match rust_type {
+ Type::Path(TypePath {
+ path,
+ ..
+ }) => {
+ let segment = path.segments.last().unwrap();
+ let type_name = segment.ident.to_string();
+
+ match type_name.as_str() {
+ "i8" => "int8_t".to_string(),
+ "i16" => "int16_t".to_string(),
+ "i32" => "int32_t".to_string(),
+ "i64" => "int64_t".to_string(),
+ "u8" => "uint8_t".to_string(),
+ "u16" => "uint16_t".to_string(),
+ "u32" => "uint32_t".to_string(),
+ "u64" => "uint64_t".to_string(),
+ "f32" => "float".to_string(),
+ "f64" => "double".to_string(),
+ "bool" => "bool".to_string(),
+ "usize" => "size_t".to_string(),
+ "isize" => "ptrdiff_t".to_string(),
+ _ => type_name,
+ }
+ },
+ Type::Array(TypeArray {
+ elem,
+ len,
+ ..
+ }) => {
+ let element_type = rust_type_to_c(elem);
+ if let Expr::Lit(ExprLit {
+ lit: Lit::Int(literal_int),
+ ..
+ }) = len
+ {
+ let array_length = literal_int.base10_digits();
+ format!("{element_type}[{array_length}]")
+ } else {
+ format!("{element_type}[]")
+ }
+ },
+ _ => "/* unknown */".to_string(),
+ }
+}
+
+#[async_trait]
+impl Compiler for GpuTypesCompiler {
+ async fn build(
+ &self
+ ) -> anyhow::Result<HashMap<Box<[Box<str>]>, Box<[Kernel]>>> {
+ let crate_dir = self.crate_dir.clone();
+ let generated_dir = self.generated_dir.clone();
+ let build_dir = self.build_dir.clone();
+
+ spawn_blocking(move || {
+ GpuTypesCompiler {
+ crate_dir,
+ generated_dir,
+ build_dir,
+ }
+ .generate()
+ })
+ .await??;
+
+ Ok(HashMap::new())
+ }
+}
diff --git a/crates/uzu/build/main.rs b/crates/uzu/build/main.rs
index b6fec2775..8eab62ce7 100644
--- a/crates/uzu/build/main.rs
+++ b/crates/uzu/build/main.rs
@@ -4,13 +4,12 @@ use anyhow::Context;
use futures::future::try_join_all;
mod common;
-mod shared_types;
+mod gpu_types;
#[cfg(feature = "metal")]
mod metal;
-use common::compiler::Compiler;
-use common::envs;
+use common::{compiler::Compiler, envs, traitgen::traitgen_all};
#[tokio::main(flavor = "current_thread")]
async fn main() -> anyhow::Result<()> {
@@ -38,14 +37,25 @@ async fn main() -> anyhow::Result<()> {
debug_log!("cleaned caches");
}
- let mut compilers: Vec<Box<dyn Compiler>> = Vec::new();
+ let gpu_types_compiler = gpu_types::GpuTypesCompiler::new()?;
+ let generated_header_dir =
+ gpu_types_compiler.generated_header_dir().clone();
+ gpu_types_compiler.build().await?;
+ debug_log!("gpu_types build done");
- compilers.push(Box::new(shared_types::SharedTypesCompiler::new()?));
+ let mut compilers: Vec<Box<dyn Compiler>> = Vec::new();
#[cfg(feature = "metal")]
- compilers.push(Box::new(metal::MetalCompiler::new()?));
+ compilers.push(Box::new(metal::MetalCompiler::new_with_include_dir(
+ generated_header_dir,
+ )?));
+
+ let backends_kernels =
+ try_join_all(compilers.iter().map(|c| c.build())).await?;
+
+ debug_log!("backend build end");
- try_join_all(compilers.iter().map(|c| c.build())).await?;
+ traitgen_all(backends_kernels)?;
debug_log!("build script ended");
diff --git a/crates/uzu/build/metal/ast.rs b/crates/uzu/build/metal/ast.rs
index 933d16881..df079d2eb 100644
--- a/crates/uzu/build/metal/ast.rs
+++ b/crates/uzu/build/metal/ast.rs
@@ -1,8 +1,11 @@
-use std::collections::HashMap;
-
use anyhow::{Context, bail};
use serde::{Deserialize, Serialize};
+use crate::common::kernel::{
+ Kernel, KernelArgument, KernelArgumentType, KernelParameter,
+ KernelParameterType,
+};
+
pub type MetalAstNode = clang_ast::Node<MetalAstKind>;
#[derive(Debug, Clone, Deserialize)]
@@ -17,7 +20,9 @@ pub struct ParmVarDeclType {
pub enum MetalAstKind {
TranslationUnitDecl,
FunctionTemplateDecl,
- TemplateTypeParmDecl,
+ TemplateTypeParmDecl {
+ name: Option<Box<str>>,
+ },
FunctionDecl {
name: Box<str>,
},
@@ -98,13 +103,26 @@ fn annotation_from_ast_node(
.collect()
}
+#[derive(Debug, Serialize, Deserialize)]
+pub enum MetalConstantType {
+ Scalar,
+ Array,
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub enum MetalGroupsType {
+ Direct(Box<str>),
+ Indirect,
+}
+
#[derive(Debug, Serialize, Deserialize)]
pub enum MetalArgumentType {
Buffer,
- Constant(Box<str>),
- Shared(Box<str>),
+ Constant((Box<str>, MetalConstantType)),
+ Shared(Option<Box<str>>),
+ Specialize(Box<str>),
Axis(Box<str>, Box<str>),
- Groups(Box<str>),
+ Groups(MetalGroupsType),
Threads(Box<str>),
}
@@ -117,6 +135,20 @@ pub struct MetalArgument {
}
impl MetalArgument {
+ fn scalar_type_to_rust(c_type: &str) -> anyhow::Result<&'static str> {
+ let mut tokens: Vec<_> = c_type.split_whitespace().collect();
+ if tokens.first() == Some(&"const") {
+ tokens.remove(0);
+ }
+ match tokens.as_slice() {
+ ["bool"] => Ok("bool"),
+ ["uint"] | ["uint32_t"] | ["unsigned", "int"] => Ok("u32"),
+ ["int"] | ["int32_t"] => Ok("i32"),
+ ["float"] => Ok("f32"),
+ _ => bail!("unknown scalar type: {c_type}"),
+ }
+ }
+
fn from_ast_node_and_source(
argument_node: MetalAstNode,
source: &str,
@@ -182,6 +214,16 @@ impl MetalArgument {
let annotation_key = annotation.remove(0);
match &*annotation_key {
+ "dsl.specialize" => {
+ if !annotation.is_empty() {
+ bail!(
+ "dsl.specialize takes no arguments, got {}",
+ annotation.len()
+ );
+ }
+ let rust_type = Self::scalar_type_to_rust(&self.c_type)?;
+ Ok(MetalArgumentType::Specialize(rust_type.into()))
+ },
"dsl.axis" => {
if annotation.len() != 2 {
bail!(
@@ -201,7 +243,15 @@ impl MetalArgument {
annotation.len()
);
}
- Ok(MetalArgumentType::Groups(annotation.remove(0)))
+ let dim = annotation.remove(0);
+ match dim.as_ref() {
+ "INDIRECT" => Ok(MetalArgumentType::Groups(
+ MetalGroupsType::Indirect,
+ )),
+ _ => Ok(MetalArgumentType::Groups(
+ MetalGroupsType::Direct(dim),
+ )),
+ }
},
"dsl.threads" => {
if annotation.len() != 1 {
@@ -214,8 +264,7 @@ impl MetalArgument {
},
_ => bail!("unknown annotation: {annotation_key}"),
}
- } else if (self.c_type.contains("device")
- || self.c_type.contains("constant"))
+ } else if self.c_type.contains("device")
&& self.c_type.contains('*')
&& !self.c_type.contains('&')
{
@@ -223,15 +272,17 @@ impl MetalArgument {
} else if let ["const", "constant", c_type_scalar, "&"] =
self.c_type.split_whitespace().collect::<Vec<_>>().as_slice()
{
- let rust_type = match *c_type_scalar {
- "uint" | "uint32_t" => "u32",
- "int" | "int32_t" => "i32",
- "float" => "f32",
- _ => {
- bail!("unknown scalar type: {c_type_scalar}")
- },
- };
- Ok(MetalArgumentType::Constant(rust_type.into()))
+ Ok(MetalArgumentType::Constant((
+ Self::scalar_type_to_rust(c_type_scalar)?.into(),
+ MetalConstantType::Scalar,
+ )))
+ } else if let ["const", "constant", c_type_scalar, "*"] =
+ self.c_type.split_whitespace().collect::<Vec<_>>().as_slice()
+ {
+ Ok(MetalArgumentType::Constant((
+ Self::scalar_type_to_rust(c_type_scalar)?.into(),
+ MetalConstantType::Array,
+ )))
} else if self.c_type.contains("threadgroup")
&& self.c_type.contains('*')
{
@@ -245,18 +296,47 @@ impl MetalArgument {
.rfind(']')
.context("threadgroup missing size bracket")?;
let size_expr = &self.source[lbracket..rbracket];
- Ok(MetalArgumentType::Shared(size_expr.into()))
+ Ok(MetalArgumentType::Shared(Some(size_expr.into())))
+ } else if self.c_type.contains("threadgroup")
+ && self.c_type.contains('&')
+ {
+ Ok(MetalArgumentType::Shared(None))
} else {
bail!("cannot parse c type: {}", self.c_type);
}
}
+
+ fn to_parameter(&self) -> Option<KernelParameter> {
+ match self.argument_type() {
+ Ok(MetalArgumentType::Specialize(ty)) => Some(KernelParameter {
+ name: self.name.clone(),
+ ty: KernelParameterType::Specialization(ty),
+ }),
+ _ => None,
+ }
+ }
+}
+
+#[derive(Debug, Serialize, Deserialize)]
+pub struct MetalTypeParameter {
+ pub name: Box<str>,
+ pub variants: Box<[Box<str>]>,
+}
+
+impl MetalTypeParameter {
+ fn to_parameter(&self) -> KernelParameter {
+ KernelParameter {
+ name: self.name.clone(),
+ ty: KernelParameterType::DType,
+ }
+ }
}
#[derive(Debug, Serialize, Deserialize)]
pub struct MetalKernelInfo {
pub name: Box<str>,
pub arguments: Box<[MetalArgument]>,
- pub specializations: Option<(Box<str>, Box<[Box<str>]>)>,
+ pub variants: Option<Box<[MetalTypeParameter]>>,
}
impl MetalKernelInfo {
@@ -272,11 +352,79 @@ impl MetalKernelInfo {
})
}
+ pub fn has_groups_direct(&self) -> bool {
+ self.arguments.iter().any(|a| {
+ matches!(
+ a.argument_type(),
+ Ok(MetalArgumentType::Groups(MetalGroupsType::Direct(_)))
+ )
+ })
+ }
+
+ pub fn has_groups_indirect(&self) -> bool {
+ self.arguments.iter().any(|a| {
+ matches!(
+ a.argument_type(),
+ Ok(MetalArgumentType::Groups(MetalGroupsType::Indirect))
+ )
+ })
+ }
+
pub fn has_threads(&self) -> bool {
self.arguments.iter().any(|a| {
matches!(a.argument_type(), Ok(MetalArgumentType::Threads(_)))
})
}
+
+ pub fn to_kernel(&self) -> Kernel {
+ let mut indirect_flag = false;
+
+ Kernel {
+ name: self.name.clone(),
+ parameters: self
+ .variants
+ .as_ref()
+ .map(|v| v.iter().map(|p| p.to_parameter()).collect::<Vec<_>>())
+ .unwrap_or_default()
+ .into_iter()
+ .chain(self.arguments.iter().filter_map(|a| a.to_parameter()))
+ .collect(),
+ arguments: self
+ .arguments
+ .iter()
+ .filter_map(|a| match a.argument_type() {
+ Ok(MetalArgumentType::Buffer) => Some(KernelArgument {
+ name: a.name.clone(),
+ ty: KernelArgumentType::Buffer,
+ }),
+ Ok(MetalArgumentType::Groups(
+ MetalGroupsType::Indirect,
+ )) if !indirect_flag => {
+ indirect_flag = true;
+ Some(KernelArgument {
+ name: "__dsl_indirect_dispatch_buffer".into(),
+ ty: KernelArgumentType::Buffer,
+ })
+ },
+ Ok(MetalArgumentType::Constant((
+ ty,
+ MetalConstantType::Scalar,
+ ))) => Some(KernelArgument {
+ name: a.name.clone(),
+ ty: KernelArgumentType::Scalar(ty),
+ }),
+ Ok(MetalArgumentType::Constant((
+ ty,
+ MetalConstantType::Array,
+ ))) => Some(KernelArgument {
+ name: a.name.clone(),
+ ty: KernelArgumentType::Constant(ty),
+ }),
+ _ => None,
+ })
+ .collect(),
+ }
+ }
}
impl MetalKernelInfo {
@@ -284,26 +432,36 @@ impl MetalKernelInfo {
node: MetalAstNode,
source: &str,
) -> anyhow::Result<Option<Self>> {
- let (is_template, node) =
+ let (is_template, template_parameters, node) =
if matches!(node.kind, MetalAstKind::FunctionTemplateDecl) {
- let node = node
- .inner
- .into_iter()
- .find(|c| {
- matches!(
- c.kind,
+ let mut template_parameters = Vec::new();
+ let mut function_node = None;
+
+ for child in node.inner {
+ match child.kind {
+ MetalAstKind::TemplateTypeParmDecl {
+ name,
+ } => {
+ let name = name
+ .context("template parameter missing name")?;
+ template_parameters.push(name);
+ },
MetalAstKind::FunctionDecl {
- name: _
- }
- )
- })
- .context(
+ name: _,
+ } => {
+ function_node = Some(child);
+ },
+ _ => (),
+ }
+ }
+
+ let node = function_node.context(
"unexpected kind of root node: template without function",
)?;
- (true, node)
+ (true, template_parameters, node)
} else if matches!(node.kind, MetalAstKind::FunctionDecl { .. }) {
- (false, node)
+ (false, Vec::new(), node)
} else {
return Ok(None);
};
@@ -345,34 +503,58 @@ impl MetalKernelInfo {
bail!("zero length annotation");
}
})
- .collect::<anyhow::Result<HashMap<_, _>>>()?;
+ .collect::<anyhow::Result<Vec<_>>>()?;
- if !annotations.contains_key("dsl.kernel") {
+ if !annotations.iter().any(|(k, _)| k.as_ref() == "dsl.kernel") {
return Ok(None);
}
- if annotations.contains_key("dsl.specialize") != is_template {
- bail!("mismatch between AST nodes and specialization annotation");
- }
+ let variants: Box<[MetalTypeParameter]> = annotations
+ .iter()
+ .filter(|(k, _)| k.as_ref() == "dsl.variants")
+ .map(|(_, v)| {
+ let [typename, variant_values] = v.as_ref() else {
+ bail!("malformed dsl.variants annotation");
+ };
+
+ let variants = variant_values
+ .split(',')
+ .map(|v| v.trim().into())
+ .collect::<Box<[Box<str>]>>();
+
+ Ok(MetalTypeParameter {
+ name: typename.clone(),
+ variants,
+ })
+ })
+ .collect::<anyhow::Result<_>>()?;
- let specializations = if is_template {
- let specialize = annotations
- .get("dsl.specialize")
- .context("missing dsl.specialize annotation")?;
- let [specialization_typename, specialization_variants] =
- specialize.as_ref()
- else {
- bail!("malformed dsl.specialize annotation");
- };
+ if variants.is_empty() != !is_template {
+ bail!("mismatch between AST nodes and variants annotation");
+ }
- let specialization_variants = specialization_variants
- .split(',')
- .map(|v| v.trim().into())
- .collect::<Box<[Box<str>]>>();
+ if is_template {
+ let template_names = template_parameters
+ .iter()
+ .map(|name| name.as_ref())
+ .collect::<Vec<_>>();
+ let variant_names = variants
+ .iter()
+ .map(|variant| variant.name.as_ref())
+ .collect::<Vec<_>>();
+ if template_names != variant_names {
+ bail!(
+ "template parameters {:?} do not match dsl.variants order {:?}",
+ template_names,
+ variant_names
+ );
+ }
+ }
- Some((specialization_typename.clone(), specialization_variants))
- } else {
+ let variants = if variants.is_empty() {
None
+ } else {
+ Some(variants)
};
let arguments = arg_nodes
@@ -383,7 +565,7 @@ impl MetalKernelInfo {
Ok(Some(MetalKernelInfo {
name,
arguments,
- specializations,
+ variants,
}))
}
}
diff --git a/crates/uzu/build/metal/bindgen.rs b/crates/uzu/build/metal/bindgen.rs
index 843eee09f..b272f900d 100644
--- a/crates/uzu/build/metal/bindgen.rs
+++ b/crates/uzu/build/metal/bindgen.rs
@@ -4,12 +4,22 @@ use anyhow::Context;
use itertools::Itertools;
use proc_macro2::TokenStream;
use quote::{format_ident, quote};
+use syn::LitInt;
-use super::ast::{MetalArgumentType, MetalKernelInfo};
+use crate::metal::ast::MetalGroupsType;
-pub fn bindgen(kernel: &MetalKernelInfo) -> anyhow::Result<TokenStream> {
+use super::{
+ ast::{MetalArgumentType, MetalConstantType, MetalKernelInfo},
+ wrapper::SpecializeBaseIndices,
+};
+
+pub fn bindgen(
+ kernel: &MetalKernelInfo,
+ specialize_indices: &SpecializeBaseIndices,
+) -> anyhow::Result<(TokenStream, TokenStream)> {
let kernel_name = kernel.name.as_ref();
- let struct_name = format_ident!("{kernel_name}Kernel");
+ let trait_name = format_ident!("{kernel_name}Kernel");
+ let struct_name = format_ident!("{kernel_name}MetalKernel");
let parse_expr = |expr: &Box<str>| -> anyhow::Result<TokenStream> {
syn::parse_str(expr.as_ref()).with_context(|| {
@@ -20,60 +30,139 @@ pub fn bindgen(kernel: &MetalKernelInfo) -> anyhow::Result<TokenStream> {
})
};
- let (specialize_extra_argument, specialize_kernel_format) = if kernel
- .specializations
- .is_some()
+ let (variants_extra_arguments, variants_kernel_format) = if let Some(
+ variants,
+ ) =
+ &kernel.variants
{
+ let variant_names = variants
+ .iter()
+ .map(|type_parameter| {
+ format_ident!("{}", type_parameter.name.as_ref())
+ })
+ .collect::<Vec<_>>();
+
+ let kernel_format = repeat_n("{}", variant_names.len() + 1).join("_");
+
(
- quote! { , data_type: crate::backends::metal::KernelDataType },
- quote! { &format!("{}_{}", #kernel_name, data_type.function_name_suffix()) },
+ variant_names
+ .iter()
+ .map(|name| quote! { #[allow(non_snake_case)] #name: crate::DataType })
+ .collect(),
+ quote! { &format!(#kernel_format, #kernel_name #(, KernelDataType::from(#variant_names).function_name_suffix())*) },
)
} else {
- (quote! {}, quote! { #kernel_name })
+ (Vec::new(), quote! { #kernel_name })
};
- let (encode_args_defs, encode_args_sets, encode_args_names): (
- Vec<TokenStream>,
+ let base_index = specialize_indices.get(&kernel.name).copied();
+ let (specialize_args, specialize_setup): (
Vec<TokenStream>,
Vec<TokenStream>,
) = kernel
.arguments
.iter()
- .filter(|k| {
- matches!(
- k.argument_type(),
- Ok(MetalArgumentType::Buffer | MetalArgumentType::Constant(_))
- )
+ .filter(|a| {
+ matches!(a.argument_type(), Ok(MetalArgumentType::Specialize(_)))
})
.enumerate()
- .map(|(i, ka)| {
- let arg_index = i;
+ .map(|(i, a)| {
+ let arg_name = format_ident!("{}", a.name.as_ref());
+ let rust_type = match a.argument_type().unwrap() {
+ MetalArgumentType::Specialize(t) => format_ident!("{t}"),
+ _ => unreachable!(),
+ };
+ let mtl_type = match a.argument_type().unwrap() {
+ MetalArgumentType::Specialize(ref t) => match t.as_ref() {
+ "bool" => quote! { Bool },
+ "u32" => quote! { UInt },
+ "i32" => quote! { Int },
+ "f32" => quote! { Float },
+ _ => unreachable!(),
+ },
+ _ => unreachable!(),
+ };
+ let idx = base_index.unwrap_or(0) + i;
+ let arg_def = quote! { #arg_name: #rust_type };
+ let setup = quote! {
+ function_constants.set_constant_value_type_at_index(
+ std::ptr::NonNull::from(&#arg_name).cast(),
+ MTLDataType::#mtl_type,
+ #idx,
+ );
+ };
+ (arg_def, setup)
+ })
+ .unzip();
+
+ let has_specialize = !specialize_args.is_empty();
+ let function_constants_init = if has_specialize {
+ quote! {
+ let function_constants = MTLFunctionConstantValues::new();
+ #(#specialize_setup)*
+ }
+ } else {
+ quote! {}
+ };
+ let function_constants_arg = if has_specialize {
+ quote! { Some(&function_constants) }
+ } else {
+ quote! { None }
+ };
+
+ let mut arg_count: usize = 0;
+ let mut indirect_flag = false;
+
+ let (encode_args_defs, encode_args_sets, encode_args_names): (
+ Vec<TokenStream>,
+ Vec<TokenStream>,
+ Vec<TokenStream>,
+ ) = kernel
+ .arguments
+ .iter()
+ .filter_map(|ka| {
let arg_name = format_ident!("{}", ka.name.as_ref());
match ka.argument_type().unwrap() {
MetalArgumentType::Buffer => {
- let def = quote! { #arg_name: &crate::backends::metal::ProtocolObject<dyn crate::backends::metal::MTLBuffer> };
+ let def = quote! { #arg_name: &Retained<ProtocolObject<dyn MTLBuffer>> };
let set = quote! {
- compute_encoder.set_buffer(Some(#arg_name), 0, #arg_index);
+ compute_encoder.set_buffer(Some(#arg_name), 0, #arg_count);
};
- (def, set, quote! { #arg_name })
- },
- MetalArgumentType::Constant(r_type) => {
+ arg_count += 1;
+
+ Some((def, set, quote! { #arg_name }))
+ }
+ MetalArgumentType::Constant((r_type, constant_type)) => {
let arg_dtype = format_ident!("{r_type}");
- let def = quote! { #arg_name: #arg_dtype };
- let set = quote! {
- compute_encoder.set_value(&#arg_name, #arg_index);
+
+ let (def, set) = match constant_type {
+ MetalConstantType::Scalar => (
+ quote! { #arg_name: #arg_dtype },
+ quote! { compute_encoder.set_value(&#arg_name, #arg_count); },
+ ),
+ MetalConstantType::Array => (
+ quote! { #arg_name: &[#arg_dtype] },
+ quote! { compute_encoder.set_slice(#arg_name, #arg_count); },
+ ),
};
- (def, set, quote! { #arg_name })
+ arg_count += 1;
+
+ Some((def, set, quote! { #arg_name }))
}
- _ => unreachable!(),
+ MetalArgumentType::Groups(MetalGroupsType::Indirect) if !indirect_flag => {
+ indirect_flag = true;
+
+ Some((quote! { __dsl_indirect_dispatch_buffer: &Retained<ProtocolObject<dyn MTLBuffer>> }, quote! {}, quote! { __dsl_indirect_dispatch_buffer }))
+ }
+ _ => None,
}
})
.multiunzip();
- let dispatch = if kernel.has_axis() {
+ let (dispatch, elements) = if kernel.has_axis() {
if kernel.has_groups() || kernel.has_threads() {
anyhow::bail!("mixing groups/threads and axis is not supported");
}
@@ -99,27 +188,16 @@ pub fn bindgen(kernel: &MetalKernelInfo) -> anyhow::Result<TokenStream> {
let (threads, threads_per_group): (Vec<TokenStream>, Vec<TokenStream>) =
axis.into_iter().unzip();
- quote! {
- compute_encoder.dispatch_threads(
- crate::backends::metal::MTLSize::new(#((#threads) as usize, )*),
- crate::backends::metal::MTLSize::new(#((#threads_per_group) as usize, )*),
- );
- }
+ (
+ quote! {
+ compute_encoder.dispatch_threads(
+ MTLSize::new(#((#threads) as usize, )*),
+ MTLSize::new(#((#threads_per_group) as usize, )*),
+ );
+ },
+ threads.into_iter().chain(threads_per_group.into_iter()),
+ )
} else {
- let mut groups = kernel
- .arguments
- .iter()
- .filter_map(|a| {
- if let Ok(MetalArgumentType::Groups(rexprs)) = a.argument_type()
- {
- Some(parse_expr(&rexprs))
- } else {
- None
- }
- })
- .collect::<anyhow::Result<Vec<TokenStream>>>()?;
- groups.extend(repeat_n(quote! {1}, 3 - groups.len()));
-
let mut threads = kernel
.arguments
.iter()
@@ -135,44 +213,109 @@ pub fn bindgen(kernel: &MetalKernelInfo) -> anyhow::Result<TokenStream> {
.collect::<anyhow::Result<Vec<TokenStream>>>()?;
threads.extend(repeat_n(quote! {1}, 3 - threads.len()));
- quote! {
- compute_encoder.dispatch_threadgroups(
- crate::backends::metal::MTLSize::new(#((#groups) as usize, )*),
- crate::backends::metal::MTLSize::new(#((#threads) as usize, )*),
- );
+ if kernel.has_groups_indirect() {
+ if kernel.has_groups_direct() {
+ anyhow::bail!("cannot mix indirect and direct groups");
+ }
+
+ (
+ quote! {
+ compute_encoder.dispatch_threadgroups_indirect(
+ __dsl_indirect_dispatch_buffer,
+ 0,
+ MTLSize::new(#((#threads) as usize, )*),
+ );
+ },
+ vec![].into_iter().chain(threads.into_iter()),
+ )
+ } else {
+ let mut groups = kernel
+ .arguments
+ .iter()
+ .filter_map(|a| {
+ if let Ok(MetalArgumentType::Groups(
+ MetalGroupsType::Direct(rexprs),
+ )) = a.argument_type()
+ {
+ Some(parse_expr(&rexprs))
+ } else {
+ None
+ }
+ })
+ .collect::<anyhow::Result<Vec<TokenStream>>>()?;
+ groups.extend(repeat_n(quote! {1}, 3 - groups.len()));
+
+ (
+ quote! {
+ compute_encoder.dispatch_threadgroups(
+ MTLSize::new(#((#groups) as usize, )*),
+ MTLSize::new(#((#threads) as usize, )*),
+ );
+ },
+ groups.into_iter().chain(threads.into_iter()),
+ )
}
};
- Ok(quote! {
+ let guards = elements
+ .flat_map(|e| {
+ if let Ok(lit_int) = syn::parse2::<LitInt>(e.clone())
+ && let Ok(int) = lit_int.base10_parse::<u32>()
+ && int != 0
+ {
+ None
+ } else {
+ Some(quote! { (#e) == 0 })
+ }
+ })
+ .fold(quote! {}, |a, b| {
+ if !a.is_empty() && !b.is_empty() {
+ quote! {#a || #b}
+ } else {
+ quote! {#a #b}
+ }
+ });
+
+ let empty_dispatch_guards = if !guards.is_empty() {
+ quote! { if #guards { return; }; }
+ } else {
+ quote! {}
+ };
+
+ let kernel = quote! {
pub struct #struct_name {
- pipeline: crate::backends::metal::Retained<crate::backends::metal::ProtocolObject<dyn crate::backends::metal::MTLComputePipelineState>>,
+ pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
}
- impl #struct_name {
- pub fn new(context: &crate::backends::metal::MTLContext #specialize_extra_argument) -> Result<Self, crate::backends::metal::MTLError> {
- use crate::backends::metal::metal_extensions::LibraryPipelineExtensions;
- let pipeline = context.library.compute_pipeline_state(#specialize_kernel_format, None)?;
+ impl crate::backends::common::kernel::#trait_name for #struct_name {
+ type Backend = crate::backends::metal::Metal;
+
+ fn new(context: &MTLContext #(, #variants_extra_arguments)* #(, #specialize_args)*) -> Result<Self, MTLError> {
+ #function_constants_init
+ let pipeline = context.library.compute_pipeline_state(#variants_kernel_format, #function_constants_arg)?;
Ok(Self { pipeline })
}
- pub fn encode(&self, #(#encode_args_defs, )*compute_encoder: &crate::backends::metal::ProtocolObject<dyn crate::backends::metal::MTLComputeCommandEncoder>) {
- use metal::MTLComputeCommandEncoder;
- use crate::backends::metal::ComputeEncoderSetValue;
+ fn encode(&self, #(#encode_args_defs, )* compute_encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>) {
+ #empty_dispatch_guards
compute_encoder.set_compute_pipeline_state(&self.pipeline);
#(#encode_args_sets)*
#dispatch
}
-
- pub fn encode_if(&self, #(#encode_args_defs, )*compute_encoder: &crate::backends::metal::ProtocolObject<dyn crate::backends::metal::MTLComputeCommandEncoder>, predicate: Option<&crate::backends::metal::ProtocolObject<dyn crate::backends::metal::MTLBuffer>>) {
- use crate::backends::metal::metal_extensions::ComputeEncoderConditional;
+ fn encode_if(&self, #(#encode_args_defs, )* compute_encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>, predicate: Option<&Retained<ProtocolObject<dyn MTLBuffer>>>) {
+ #empty_dispatch_guards
compute_encoder.condition(
predicate,
|| {
- self.encode(#(#encode_args_names, )*compute_encoder);
+ self.encode(#(#encode_args_names, )* compute_encoder);
},
None::<fn()>,
);
}
}
- })
+ };
+
+ let associated_type = quote! { type #trait_name = #struct_name; };
+
+ Ok((kernel, associated_type))
}
diff --git a/crates/uzu/build/metal/compiler.rs b/crates/uzu/build/metal/compiler.rs
index 0b60958d7..cf232c231 100644
--- a/crates/uzu/build/metal/compiler.rs
+++ b/crates/uzu/build/metal/compiler.rs
@@ -1,35 +1,59 @@
-use std::collections::HashMap;
-use std::iter::{once, zip};
-use std::path::Path;
-use std::{env, fs, path::PathBuf};
-
-use anyhow::{Context, bail};
-use futures::future::try_join_all;
-use futures::{StreamExt, TryStreamExt, stream};
+use std::{
+ collections::HashMap,
+ env, fs,
+ path::{Path, PathBuf},
+};
+
+use anyhow::Context;
+use async_trait::async_trait;
+use futures::{StreamExt, TryStreamExt, future::try_join_all, stream};
use proc_macro2::TokenStream;
use quote::quote;
use serde::{Deserialize, Serialize};
-use tokio::io::AsyncReadExt;
-use tokio::task::spawn_blocking;
+use tokio::{io::AsyncReadExt, task::spawn_blocking};
use walkdir::WalkDir;
-use async_trait::async_trait;
-
-use crate::common::compiler::Compiler;
-use crate::common::{caching, envs};
-use crate::debug_log;
-
-use super::ast::{MetalArgumentType, MetalKernelInfo};
-use super::toolchain::MetalToolchain;
+use super::{
+ ast::MetalKernelInfo,
+ toolchain::MetalToolchain,
+ wrapper::{SpecializeBaseIndices, wrappers},
+};
+use crate::{
+ common::{
+ caching, codegen::write_tokens, compiler::Compiler, envs,
+ kernel::Kernel,
+ },
+ debug_log,
+};
#[derive(Serialize, Deserialize, Debug)]
struct ObjectInfo {
+ src_rel_path: PathBuf,
object_path: PathBuf,
kernels: Box<[MetalKernelInfo]>,
+ specialize_indices: SpecializeBaseIndices,
buildsystem_hash: [u8; blake3::OUT_LEN],
dependency_hashes: HashMap<Box<str>, [u8; blake3::OUT_LEN]>,
}
+impl ObjectInfo {
+ fn kernels(&self) -> (Box<[Box<str>]>, Box<[Kernel]>) {
+ let src_rel_path: Box<[Box<str>]> = self
+ .src_rel_path
+ .with_extension("")
+ .as_os_str()
+ .to_str()
+ .unwrap()
+ .split("/")
+ .map(|s| s.to_string().into_boxed_str())
+ .collect();
+
+ let kernels = self.kernels.iter().map(|ki| ki.to_kernel()).collect();
+
+ (src_rel_path, kernels)
+ }
+}
+
fn is_nax_source(path: &Path) -> bool {
path.file_name()
.and_then(|s| s.to_str())
@@ -37,155 +61,6 @@ fn is_nax_source(path: &Path) -> bool {
.unwrap_or(false)
}
-fn wrappers(kernel: &MetalKernelInfo) -> anyhow::Result<Box<[Box<str>]>> {
- let mut kernel_wrappers = Vec::new();
-
- for specialization_variant in kernel
- .specializations
- .as_ref()
- .map(|(t, v)| zip(std::iter::repeat(t), v.iter()).map(Some).collect())
- .unwrap_or(vec![None])
- {
- let (wrapper_name, underlying_name) =
- if let Some((_, variant)) = specialization_variant {
- (
- format!("{}_{}", kernel.name, variant),
- format!("{}<{}>", kernel.name, variant),
- )
- } else {
- (kernel.name.to_string(), kernel.name.to_string())
- };
-
- let max_total_threads_per_threadgroup = kernel
- .arguments
- .iter()
- .filter_map(|a| match a.argument_type() {
- Ok(MetalArgumentType::Axis(_, l))
- | Ok(MetalArgumentType::Threads(l)) => Some(format!("({l})")),
- _ => None,
- })
- .collect::<Vec<_>>()
- .join(" * ");
-
- let mut wrapper_arguments = kernel
- .arguments
- .iter()
- .filter_map(|a| match a.argument_type() {
- Ok(MetalArgumentType::Buffer)
- | Ok(MetalArgumentType::Constant(_)) => Some(format!(
- "{} {}",
- a.c_type
- .split_whitespace()
- .map(|token| {
- if let Some((typename, variant)) =
- specialization_variant
- && token == typename.as_ref()
- {
- variant
- } else {
- token
- }
- })
- .collect::<Vec<_>>()
- .join(" "),
- a.name
- )),
- _ => None,
- })
- .collect::<Vec<_>>();
-
- if kernel.has_axis() {
- if kernel.has_groups() || kernel.has_threads() {
- bail!("mixing groups/threads and axis is not supported");
- }
-
- wrapper_arguments.push(
- "uint3 __dsl_axis_idx [[thread_position_in_grid]]".into(),
- );
- }
-
- if kernel.has_groups() {
- wrapper_arguments.push(
- "uint3 __dsl_group_idx [[threadgroup_position_in_grid]]".into(),
- );
- }
-
- if kernel.has_threads() {
- wrapper_arguments.push(
- "uint3 __dsl_thread_idx [[thread_position_in_threadgroup]]"
- .into(),
- );
- }
-
- let wrapper_arguments = wrapper_arguments.join(", ");
-
- let shared_definitions = kernel.arguments.iter().filter_map(|a| {
- if let Ok(MetalArgumentType::Shared(len)) = a.argument_type() {
- Some(format!(
- "{} {}[{}]",
- a.c_type.replace('*', ""),
- a.name,
- len.as_ref(),
- ))
- } else {
- None
- }
- });
-
- let underlying_arguments = {
- let mut group_axis_letters = ["x", "y", "z"].iter();
- let mut thread_axis_letters = ["x", "y", "z"].iter();
-
- kernel
- .arguments
- .iter()
- .map(|a| match a.argument_type().unwrap() {
- MetalArgumentType::Buffer
- | MetalArgumentType::Constant(_)
- | MetalArgumentType::Shared(_) => a.name.to_string(),
- MetalArgumentType::Axis(..) => {
- format!(
- "__dsl_axis_idx.{}",
- group_axis_letters.next().unwrap()
- )
- },
- MetalArgumentType::Groups(_) => {
- format!(
- "__dsl_group_idx.{}",
- group_axis_letters.next().unwrap()
- )
- },
- MetalArgumentType::Threads(_) => {
- format!(
- "__dsl_thread_idx.{}",
- thread_axis_letters.next().unwrap()
- )
- },
- })
- .collect::<Vec<_>>()
- .join(", ")
- };
-
- let underlying_call =
- format!("{underlying_name}({underlying_arguments})");
-
- let wrapper_body = shared_definitions
- .chain(once(underlying_call))
- .map(|l| format!(" {l};\n"))
- .collect::<Vec<_>>()
- .join("");
-
- kernel_wrappers.push(
- format!(
- "\n[[kernel, max_total_threads_per_threadgroup({max_total_threads_per_threadgroup})]] void {wrapper_name}({wrapper_arguments}) {{\n{wrapper_body}}}\n"
- )
- .into(),
- );
- }
-
- Ok(kernel_wrappers.into())
-}
-
async fn hash_dependencies(
dependencies: impl Iterator<Item = Box<str>>
) -> anyhow::Result<HashMap<Box<str>, [u8; blake3::OUT_LEN]>> {
@@ -241,12 +116,12 @@ pub struct MetalCompiler {
}
impl MetalCompiler {
- pub fn new() -> anyhow::Result<Self> {
+ pub fn new_with_include_dir(include_dir: PathBuf) -> anyhow::Result<Self> {
let src_dir = PathBuf::from(
env::var("CARGO_MANIFEST_DIR")
.context("missing CARGO_MANIFEST_DIR")?,
)
- .join("src");
+ .join("src/backends/metal/kernel");
let out_dir =
PathBuf::from(env::var("OUT_DIR").context("missing OUT_DIR")?);
@@ -256,7 +131,8 @@ impl MetalCompiler {
})?;
let toolchain =
- MetalToolchain::from_env().context("cannot create toolchain")?;
+ MetalToolchain::from_env_with_include_dir(Some(include_dir))
+ .context("cannot create toolchain")?;
Ok(Self {
src_dir,
@@ -288,11 +164,14 @@ impl MetalCompiler {
.context("source path has no file name")?
.to_os_string();
+ let source_path_rel = source_path
+ .strip_prefix(&self.src_dir)
+ .context("source is not in src_dir")?;
+
let build_dir = self.build_dir.join(source_path_hash.to_string());
let objectinfo_path =
build_dir.join(&source_file_name).with_extension("objectinfo");
- // Check cache
if build_dir.exists() {
if let Ok(contents) = tokio::fs::read(&objectinfo_path).await
&& let Ok(cached) =
@@ -327,7 +206,6 @@ impl MetalCompiler {
format!("cannot create build directory {}", build_dir.display())
})?;
- // Analyze source
let (metal_kernel_infos, dependencies) =
self.toolchain.analyze(&source_path).await.with_context(|| {
format!("cannot analyze {}", source_path_display)
@@ -335,20 +213,14 @@ impl MetalCompiler {
let kernel_infos: Vec<MetalKernelInfo> = metal_kernel_infos.collect();
- // Generate footer with kernel wrappers
+ let (wrapper_strs, specialize_indices) = wrappers(&kernel_infos)
+ .context("cannot generate kernel wrappers")?;
+
let mut footer = String::new();
- for kernel_info in &kernel_infos {
- for wrapper in wrappers(kernel_info)
- .with_context(|| {
- format!("cannot generate wrappers for {}", kernel_info.name)
- })?
- .iter()
- {
- footer.push_str(wrapper);
- }
+ for wrapper in wrapper_strs.iter() {
+ footer.push_str(wrapper);
}
- // Compile
let object_path =
build_dir.join(&source_file_name).with_extension("air");
@@ -371,8 +243,10 @@ impl MetalCompiler {
.context("cannot hash dependencies")?;
let object_info = ObjectInfo {
+ src_rel_path: source_path_rel.into(),
object_path,
kernels: kernel_infos.into_boxed_slice(),
+ specialize_indices,
buildsystem_hash,
dependency_hashes,
};
@@ -463,23 +337,44 @@ impl MetalCompiler {
debug_log!("bindgen start");
- let bindings = objects
+ let (bindings, associated_types) = objects
.into_iter()
- .flat_map(|o| &o.kernels)
- .map(|k| {
- super::bindgen::bindgen(k).with_context(|| {
- format!("cannot generate bindings for {}", k.name)
- })
+ .flat_map(|o| o.kernels.iter().map(|k| (k, &o.specialize_indices)))
+ .map(|(k, specialize_indices)| {
+ super::bindgen::bindgen(k, specialize_indices).with_context(
+ || format!("cannot generate bindings for {}", k.name),
+ )
})
- .collect::<anyhow::Result<Vec<TokenStream>>>()?;
+ .collect::<anyhow::Result<(Vec<TokenStream>, Vec<TokenStream>)>>(
+ )?;
+
+ let tokens = quote! {
+ use crate::backends::metal::{
+ ComputeEncoderSetValue,
+ KernelDataType,
+ MTLContext,
+ MTLDataType,
+ MTLError,
+ MTLFunctionConstantValues,
+ MTLSize,
+ ProtocolObject,
+ Retained,
+ metal_extensions::{ComputeEncoderConditional, LibraryPipelineExtensions},
+ };
+ use metal::{MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState};
+
+ #(#bindings)*
- let tokens = quote! { #(#bindings)* };
+ pub struct MetalKernels;
- let parsed =
- syn::parse2(tokens).context("cannot parse generated bindings")?;
- fs::write(&out_path, prettyplease::unparse(&parsed)).with_context(
- || format!("cannot write bindings file {}", out_path.display()),
- )?;
+ impl crate::backends::common::kernel::Kernels for MetalKernels {
+ type Backend = crate::backends::metal::Metal;
+
+ #(#associated_types)*
+ }
+ };
+
+ write_tokens(tokens, &out_path).context("cannot write bindings")?;
fs::write(&hash_path, hash.to_string()).with_context(|| {
format!("cannot write hash file {}", hash_path.display())
@@ -493,7 +388,9 @@ impl MetalCompiler {
#[async_trait]
impl Compiler for MetalCompiler {
- async fn build(&self) -> anyhow::Result<()> {
+ async fn build(
+ &self
+ ) -> anyhow::Result<HashMap<Box<[Box<str>]>, Box<[Kernel]>>> {
let nax_enabled = cfg!(feature = "metal-nax");
debug_log!(
"metal nax {}",
@@ -530,6 +427,6 @@ impl Compiler for MetalCompiler {
self.link(&objects).await.context("cannot link objects")?;
self.bindgen(&objects).context("cannot generate bindings")?;
- Ok(())
+ Ok(objects.iter().map(|o| o.kernels()).collect())
}
}
diff --git a/crates/uzu/build/metal/mod.rs b/crates/uzu/build/metal/mod.rs
index ce9ce685c..7d2d4a094 100644
--- a/crates/uzu/build/metal/mod.rs
+++ b/crates/uzu/build/metal/mod.rs
@@ -2,5 +2,6 @@ mod ast;
mod bindgen;
mod compiler;
mod toolchain;
+mod wrapper;
pub use compiler::MetalCompiler;
diff --git a/crates/uzu/build/metal/toolchain.rs b/crates/uzu/build/metal/toolchain.rs
index a0682119a..0c3d540cb 100644
--- a/crates/uzu/build/metal/toolchain.rs
+++ b/crates/uzu/build/metal/toolchain.rs
@@ -2,7 +2,7 @@ use std::{
env::{self},
ffi::OsString,
fs,
- path::Path,
+ path::{Path, PathBuf},
process::Stdio,
};
@@ -78,18 +78,21 @@ impl MetalSdk {
#[derive(Debug)]
pub enum MetalStd {
+ Metal3_1,
Metal4_0,
}
impl MetalStd {
pub fn to_str(&self) -> &'static str {
match self {
+ MetalStd::Metal3_1 => "metal3.1",
MetalStd::Metal4_0 => "metal4.0",
}
}
pub fn min_os(&self) -> &'static str {
match self {
+ MetalStd::Metal3_1 => "14.0",
MetalStd::Metal4_0 => "26.0",
}
}
@@ -97,7 +100,11 @@ impl MetalStd {
impl Default for MetalStd {
fn default() -> Self {
- Self::Metal4_0
+ if cfg!(feature = "metal-nax") {
+ Self::Metal4_0
+ } else {
+ Self::Metal3_1
+ }
}
}
@@ -107,10 +114,13 @@ pub struct MetalToolchain {
std: MetalStd,
opt_flags: Box<[OsString]>,
extra_options: Box<[OsString]>,
+ include_dirs: Box<[PathBuf]>,
}
impl MetalToolchain {
- pub fn from_env() -> anyhow::Result<Self> {
+ pub fn from_env_with_include_dir(
+ include_dir: Option<PathBuf>
+ ) -> anyhow::Result<Self> {
let sdk = MetalSdk::from_env().context("cannot get sdk")?;
let std = MetalStd::default();
@@ -136,17 +146,21 @@ impl MetalToolchain {
sdk.os(),
std.min_os()
)),
- // NOTE: Previous build.rs script didn't forward metal compiler warnings to cargo warnings, the new one does. This is temporary to avoid warning spam without modifying unrelated things in the same pull request as introducing the dsl and converting kernels
+ // NOTE: Previous build.rs script didn't forward metal compiler warnings to cargo warnings, the new one does.
+ // This is temporary to avoid warning spam without modifying unrelated things in the same pull request as introducing the dsl and converting kernels
OsString::from("-Wno-sign-compare"),
OsString::from("-Wno-macro-redefined"),
OsString::from("-Wno-unused-variable"),
]);
+ let include_dirs = include_dir.into_iter().collect();
+
Ok(Self {
sdk,
std,
opt_flags,
extra_options,
+ include_dirs,
})
}
@@ -156,6 +170,15 @@ impl MetalToolchain {
cmd
}
+ fn add_include_dirs(
+ &self,
+ cmd: &mut Command,
+ ) {
+ for dir in self.include_dirs.iter() {
+ cmd.arg("-I").arg(dir);
+ }
+ }
+
pub async fn analyze(
&self,
path: impl AsRef<Path>,
@@ -168,23 +191,25 @@ impl MetalToolchain {
let depfile_path =
NamedTempFile::new().context("cannot create temporary file")?;
- let analyze_output = self
- .xcrun()
- .arg("metal")
+ let mut cmd = self.xcrun();
+ cmd.arg("metal")
.args(["-x", "metal"])
.arg(format!("-std={}", self.std.to_str()))
- .args(self.extra_options.as_ref())
- .arg("-DDSL_ANALYZE")
+ .args(self.extra_options.as_ref());
+
+ self.add_include_dirs(&mut cmd);
+
+ cmd.arg("-DDSL_ANALYZE")
.arg(path)
.arg("-fsyntax-only")
.args(["-MMD", "-MF"])
.arg(depfile_path.path())
.args(["-Xclang", "-ast-dump=json"])
.stdout(Stdio::piped())
- .stderr(Stdio::piped())
- .output()
- .await
- .context("cannot execute metal analyzer")?;
+ .stderr(Stdio::piped());
+
+ let analyze_output =
+ cmd.output().await.context("cannot execute metal analyzer")?;
if !analyze_output.status.success() {
bail!(
@@ -246,23 +271,26 @@ impl MetalToolchain {
footer: impl AsRef<str>,
output: impl AsRef<Path>,
) -> anyhow::Result<Option<Box<str>>> {
- let mut compile_child = self
- .xcrun()
- .arg("metal")
+ let mut cmd = self.xcrun();
+ cmd.arg("metal")
.arg("-c")
.args(["-x", "metal"])
.arg(format!("-std={}", self.std.to_str()))
.args(self.extra_options.as_ref())
- .args(self.opt_flags.as_ref())
- .arg("-include")
+ .args(self.opt_flags.as_ref());
+
+ self.add_include_dirs(&mut cmd);
+
+ cmd.arg("-include")
.arg(source.as_ref())
.arg("-")
.arg("-o")
.arg(output.as_ref())
.stdin(Stdio::piped())
- .stderr(Stdio::piped())
- .spawn()
- .context("cannot execute metal compiler")?;
+ .stderr(Stdio::piped());
+
+ let mut compile_child =
+ cmd.spawn().context("cannot execute metal compiler")?;
compile_child
.stdin
diff --git a/crates/uzu/build/metal/wrapper.rs b/crates/uzu/build/metal/wrapper.rs
new file mode 100644
index 000000000..6348506a1
--- /dev/null
+++ b/crates/uzu/build/metal/wrapper.rs
@@ -0,0 +1,264 @@
+use std::{collections::HashMap, iter::once};
+
+use anyhow::bail;
+use itertools::Itertools;
+
+use super::ast::{MetalArgumentType, MetalKernelInfo};
+
+pub type SpecializeBaseIndices = HashMap<Box<str>, usize>;
+
+pub fn wrappers(
+ kernels: &[MetalKernelInfo]
+) -> anyhow::Result<(Box<[Box<str>]>, SpecializeBaseIndices)> {
+ let mut all_wrappers = Vec::new();
+ let mut base_indices = SpecializeBaseIndices::new();
+ let mut next_index = 0usize;
+
+ for kernel in kernels {
+ let specialize_count = kernel
+ .arguments
+ .iter()
+ .filter(|a| {
+ matches!(
+ a.argument_type(),
+ Ok(MetalArgumentType::Specialize(_))
+ )
+ })
+ .count();
+
+ if specialize_count > 0 {
+ base_indices.insert(kernel.name.clone(), next_index);
+ next_index += specialize_count;
+ }
+
+ let kernel_wrappers =
+ kernel_wrappers(kernel, base_indices.get(&kernel.name))?;
+ all_wrappers.extend(kernel_wrappers.into_vec());
+ }
+
+ Ok((all_wrappers.into_boxed_slice(), base_indices))
+}
+
+fn kernel_wrappers(
+ kernel: &MetalKernelInfo,
+ base_index: Option<&usize>,
+) -> anyhow::Result<Box<[Box<str>]>> {
+ let mut kernel_wrappers = Vec::new();
+
+ let specialize_constants = if let Some(&base) = base_index {
+ kernel
+ .arguments
+ .iter()
+ .filter(|a| matches!(a.argument_type(), Ok(MetalArgumentType::Specialize(_))))
+ .enumerate()
+ .map(|(i, a)| {
+ let c_type = a.c_type.trim_start_matches("const ");
+ let idx = base + i;
+ format!(
+ "constant {c_type} __dsl_specialize_{}_{} [[function_constant({idx})]];\n",
+ kernel.name, a.name
+ )
+ })
+ .collect::<String>()
+ } else {
+ String::new()
+ };
+
+ for type_variant in if let Some(variants) = &kernel.variants {
+ variants
+ .iter()
+ .map(|type_parameter| type_parameter.variants.iter())
+ .multi_cartesian_product()
+ .map(|values| {
+ Some(
+ variants
+ .iter()
+ .map(|tp| tp.name.to_string())
+ .zip(values.iter().map(|v| v.to_string()))
+ .collect::<Vec<_>>(),
+ )
+ })
+ .collect()
+ } else {
+ vec![None]
+ } {
+ let (wrapper_name, underlying_name) =
+ if let Some(type_variant) = &type_variant {
+ (
+ format!(
+ "{}_{}",
+ kernel.name,
+ type_variant.iter().map(|(_k, v)| v).join("_")
+ ),
+ format!(
+ "{}<{}>",
+ kernel.name,
+ type_variant.iter().map(|(_k, v)| v).join(", ")
+ ),
+ )
+ } else {
+ (kernel.name.to_string(), kernel.name.to_string())
+ };
+
+ let max_total_threads_per_threadgroup = kernel
+ .arguments
+ .iter()
+ .filter_map(|a| match a.argument_type() {
+ Ok(MetalArgumentType::Axis(_, l))
+ | Ok(MetalArgumentType::Threads(l)) => Some(format!("({l})")),
+ _ => None,
+ })
+ .collect::<Vec<_>>();
+
+ let max_total_threads_per_threadgroup =
+ if !max_total_threads_per_threadgroup.is_empty() {
+ max_total_threads_per_threadgroup.join(" * ")
+ } else {
+ "1".to_string()
+ };
+
+ let wrapper_argument_replace = type_variant
+ .as_ref()
+ .map(|tv| tv.iter().cloned().collect::<HashMap<_, _>>());
+
+ let apply_replace = |s: &str| {
+ s.split_whitespace()
+ .map(|token| {
+ if let Some(wrapper_argument_replace) =
+ &wrapper_argument_replace
+ && let Some(replacement) =
+ wrapper_argument_replace.get(token)
+ {
+ replacement
+ } else {
+ token
+ }
+ })
+ .collect::<Vec<_>>()
+ .join(" ")
+ };
+
+ let mut wrapper_arguments = kernel
+ .arguments
+ .iter()
+ .filter(|a| {
+ matches!(
+ a.argument_type(),
+ Ok(MetalArgumentType::Buffer)
+ | Ok(MetalArgumentType::Constant(_))
+ )
+ })
+ .enumerate()
+ .map(|(i, a)| match a.argument_type() {
+ Ok(MetalArgumentType::Buffer)
+ | Ok(MetalArgumentType::Constant(_)) => format!(
+ "{} {} [[buffer({})]]",
+ apply_replace(&a.c_type),
+ a.name,
+ i
+ ),
+ _ => unreachable!(),
+ })
+ .collect::<Vec<_>>();
+
+ if kernel.has_axis() {
+ if kernel.has_groups() || kernel.has_threads() {
+ bail!("mixing groups/threads and axis is not supported");
+ }
+
+ wrapper_arguments.push(
+ "uint3 __dsl_axis_idx [[thread_position_in_grid]]".into(),
+ );
+ }
+
+ if kernel.has_groups() {
+ wrapper_arguments.push(
+ "uint3 __dsl_group_idx [[threadgroup_position_in_grid]]".into(),
+ );
+ }
+
+ if kernel.has_threads() {
+ wrapper_arguments.push(
+ "uint3 __dsl_thread_idx [[thread_position_in_threadgroup]]"
+ .into(),
+ );
+ }
+
+ let wrapper_arguments = wrapper_arguments.join(", ");
+
+ let shared_definitions =
+ kernel.arguments.iter().filter_map(|a| match a.argument_type() {
+ Ok(MetalArgumentType::Shared(Some(len))) => Some(format!(
+ "{} {}[{}]",
+ apply_replace(&a.c_type.replace('*', "")),
+ a.name,
+ len.as_ref(),
+ )),
+ Ok(MetalArgumentType::Shared(None)) => Some(format!(
+ "{} {}",
+ apply_replace(&a.c_type.replace('&', "")),
+ a.name
+ )),
+ _ => None,
+ });
+
+ let underlying_arguments = {
+ let mut group_axis_letters = ["x", "y", "z"].iter();
+ let mut thread_axis_letters = ["x", "y", "z"].iter();
+
+ kernel
+ .arguments
+ .iter()
+ .map(|a| match a.argument_type().unwrap() {
+ MetalArgumentType::Buffer
+ | MetalArgumentType::Constant(_)
+ | MetalArgumentType::Shared(_) => a.name.to_string(),
+ MetalArgumentType::Specialize(_) => {
+ format!("__dsl_specialize_{}_{}", kernel.name, a.name)
+ },
+ MetalArgumentType::Axis(..) => {
+ format!(
+ "__dsl_axis_idx.{}",
+ group_axis_letters.next().unwrap()
+ )
+ },
+ MetalArgumentType::Groups(_) => {
+ format!(
+ "__dsl_group_idx.{}",
+ group_axis_letters.next().unwrap()
+ )
+ },
+ MetalArgumentType::Threads(_) => {
+ format!(
+ "__dsl_thread_idx.{}",
+ thread_axis_letters.next().unwrap()
+ )
+ },
+ })
+ .collect::<Vec<_>>()
+ .join(", ")
+ };
+
+ let underlying_call =
+ format!("{underlying_name}({underlying_arguments})");
+
+ let wrapper_body = shared_definitions
+ .chain(once(underlying_call))
+ .map(|l| format!(" {l};\n"))
+ .collect::<Vec<_>>()
+ .join("");
+
+ kernel_wrappers.push(
+ format!(
+ "\n[[kernel, max_total_threads_per_threadgroup({max_total_threads_per_threadgroup})]] void {wrapper_name}({wrapper_arguments}) {{\n{wrapper_body}}}\n"
+ )
+ .into(),
+ );
+ }
+
+ if !specialize_constants.is_empty() {
+ kernel_wrappers.insert(0, specialize_constants.into());
+ }
+
+ Ok(kernel_wrappers.into())
+}
diff --git a/crates/uzu/build/shared_types/mod.rs b/crates/uzu/build/shared_types/mod.rs
deleted file mode 100644
index 7921cf6f3..000000000
--- a/crates/uzu/build/shared_types/mod.rs
+++ /dev/null
@@ -1,169 +0,0 @@
-use std::{env, fs, path::PathBuf};
-
-use anyhow::Context;
-use tokio::{task::spawn_blocking, try_join};
-
-use async_trait::async_trait;
-
-use crate::common::compiler::Compiler;
-use crate::debug_log;
-
-struct BindgenConfig {
- name: &'static str,
- kernel_subdir: &'static str,
- header_file: &'static str,
- output_file: &'static str,
- allowlist_types: &'static [&'static str],
- raw_lines: &'static [&'static str],
-}
-
-const MATMUL_CONFIG: BindgenConfig = BindgenConfig {
- name: "matmul",
- kernel_subdir: "matmul/common",
- header_file: "shared_types.h",
- output_file: "shared_types.rs",
- allowlist_types: &[
- "GEMMParams",
- "GEMMAddMMParams",
- "GEMMSpiltKParams",
- "GEMMSpiltKMlpFusedParams",
- ],
- raw_lines: &["#![allow(non_snake_case)]"],
-};
-
-const ATTENTION_CONFIG: BindgenConfig = BindgenConfig {
- name: "attention",
- kernel_subdir: "attention",
- header_file: "gemm_types.h",
- output_file: "gemm_types.rs",
- allowlist_types: &["AttnParams", "AttnMaskParams"],
- raw_lines: &[],
-};
-
-#[derive(Debug)]
-pub struct SharedTypesCompiler {
- src_dir: PathBuf,
- build_dir: PathBuf,
-}
-
-impl SharedTypesCompiler {
- pub fn new() -> anyhow::Result<Self> {
- let src_dir = PathBuf::from(
- env::var("CARGO_MANIFEST_DIR")
- .context("missing CARGO_MANIFEST_DIR")?,
- )
- .join("src");
-
- let build_dir =
- PathBuf::from(env::var("OUT_DIR").context("missing OUT_DIR")?)
- .join("shared_types");
- fs::create_dir_all(&build_dir).with_context(|| {
- format!("cannot create {}", build_dir.display())
- })?;
-
- Ok(Self {
- src_dir,
- build_dir,
- })
- }
-
- fn generate(
- &self,
- config: &BindgenConfig,
- ) -> anyhow::Result<()> {
- let kernel_dir = self
- .src_dir
- .join(format!("backends/metal/kernel/{}", config.kernel_subdir));
- let types_header = kernel_dir.join(config.header_file);
- let types_header_hash: [u8; blake3::OUT_LEN] =
- blake3::hash(&fs::read(&types_header).with_context(|| {
- format!("cannot read {}", types_header.display())
- })?)
- .into();
- let cached_types_header_hash =
- self.build_dir.join(format!("{}.hash", config.header_file));
-
- if let Some(cached_hash) = fs::read(&cached_types_header_hash).ok()
- && cached_hash == types_header_hash
- {
- debug_log!("{} bindgen cached", config.name);
- return Ok(());
- }
- debug_log!("{} bindgen started", config.name);
-
- let mut builder = bindgen::Builder::default()
- .header(types_header.to_string_lossy())
- .clang_arg("-x")
- .clang_arg("c")
- .derive_default(true)
- .derive_copy(true)
- .use_core();
-
- for raw_line in config.raw_lines {
- builder = builder.raw_line(*raw_line);
- }
-
- for ty in config.allowlist_types {
- builder = builder.allowlist_type(*ty);
- }
-
- let bindings = builder.generate().map_err(|e| {
- anyhow::anyhow!(
- "failed to generate bindings from {}: {e}",
- config.header_file
- )
- })?;
-
- let bindings_path = kernel_dir.join(config.output_file);
- bindings.write_to_file(&bindings_path).with_context(|| {
- format!("failed to write {}", bindings_path.display())
- })?;
-
- fs::write(&cached_types_header_hash, &types_header_hash).with_context(
- || {
- format!(
- "failed to write {}",
- cached_types_header_hash.display()
- )
- },
- )?;
- debug_log!("{} bindgen done", config.name);
-
- Ok(())
- }
-}
-
-#[async_trait]
-impl Compiler for SharedTypesCompiler {
- async fn build(&self) -> anyhow::Result<()> {
- let (matmul_result, attn_result) = try_join!(
- spawn_blocking({
- let src_dir = self.src_dir.clone();
- let build_dir = self.build_dir.clone();
- move || {
- SharedTypesCompiler {
- src_dir,
- build_dir,
- }
- .generate(&MATMUL_CONFIG)
- }
- }),
- spawn_blocking({
- let src_dir = self.src_dir.clone();
- let build_dir = self.build_dir.clone();
- move || {
- SharedTypesCompiler {
- src_dir,
- build_dir,
- }
- .generate(&ATTENTION_CONFIG)
- }
- })
- )?;
-
- matmul_result?;
- attn_result?;
-
- Ok(())
- }
-}
diff --git a/crates/uzu/src/array.rs b/crates/uzu/src/array.rs
new file mode 100644
index 000000000..c2ec9ba13
--- /dev/null
+++ b/crates/uzu/src/array.rs
@@ -0,0 +1,289 @@
+use std::{cell::RefCell, ops::Range, os::raw::c_void, ptr::NonNull};
+
+use ndarray::{ArrayView, Dimension, IxDyn};
+
+use crate::{
+ ArrayElement, DataType,
+ backends::common::{Backend, Context, NativeBuffer},
+};
+
+#[derive(Debug, Clone)]
+pub struct Array<B: Backend> {
+ buffer: B::NativeBuffer,
+ offset: usize,
+ shape: Box<[usize]>,
+ data_type: DataType,
+}
+
+pub fn size_for_shape(
+ shape: &[usize],
+ data_type: DataType,
+) -> usize {
+ let Some(last_dim) = shape.last() else {
+ return data_type.size_in_bytes();
+ };
+
+ let bits_per_row = last_dim * data_type.size_in_bits();
+ let padded_bytes_per_row = bits_per_row.div_ceil(8);
+
+ let num_rows: usize = shape.iter().rev().skip(1).product();
+
+ num_rows * padded_bytes_per_row
+}
+
+impl<B: Backend> Array<B> {
+ // Constructors
+ pub unsafe fn from_parts(
+ buffer: B::NativeBuffer,
+ offset: usize,
+ shape: &[usize],
+ data_type: DataType,
+ ) -> Self {
+ let required_bytes = size_for_shape(shape, data_type);
+ assert!(
+ offset + required_bytes <= buffer.length(),
+ "Shape {:?} with data type {:?} at offset {} requires {} bytes total, but buffer length is {} bytes",
+ shape,
+ data_type,
+ offset,
+ offset + required_bytes,
+ buffer.length()
+ );
+ Self {
+ buffer,
+ offset,
+ shape: shape.into(),
+ data_type,
+ }
+ }
+
+ pub fn view(
+ &self,
+ shape: &[usize],
+ ) -> Self {
+ unsafe {
+ Self::from_parts(
+ self.buffer.clone(),
+ self.offset,
+ shape,
+ self.data_type,
+ )
+ }
+ }
+
+ // Getters
+ pub fn buffer(&self) -> &B::NativeBuffer {
+ &self.buffer
+ }
+
+ pub fn offset(&self) -> usize {
+ self.offset
+ }
+
+ pub fn shape(&self) -> &[usize] {
+ &self.shape
+ }
+
+ pub fn data_type(&self) -> DataType {
+ self.data_type
+ }
+
+ // Utility
+ pub fn cpu_ptr(&self) -> NonNull<c_void> {
+ unsafe { self.buffer.cpu_ptr().add(self.offset) }
+ }
+
+ pub fn size(&self) -> usize {
+ size_for_shape(&self.shape, self.data_type)
+ }
+
+ pub fn num_elements(&self) -> usize {
+ self.shape.iter().product()
+ }
+
+ fn validate_element_type<T: ArrayElement>(&self) {
+ assert_eq!(
+ T::data_type(),
+ self.data_type,
+ "Invalid data type, expected {:?}, actual {:?}",
+ T::data_type(),
+ self.data_type
+ );
+ }
+
+ pub fn as_bytes(&self) -> &[u8] {
+ unsafe {
+ std::slice::from_raw_parts(
+ self.cpu_ptr().as_ptr() as *const u8,
+ self.size(),
+ )
+ }
+ }
+
+ pub fn as_bytes_mut(&mut self) -> &mut [u8] {
+ unsafe {
+ std::slice::from_raw_parts_mut(
+ self.cpu_ptr().as_ptr() as *mut u8,
+ self.size(),
+ )
+ }
+ }
+
+ pub fn as_slice<T: ArrayElement>(&self) -> &[T] {
+ self.validate_element_type::<T>();
+ bytemuck::cast_slice(self.as_bytes())
+ }
+
+ pub fn as_slice_mut<T: ArrayElement>(&mut self) -> &mut [T] {
+ self.validate_element_type::<T>();
+ bytemuck::cast_slice_mut(self.as_bytes_mut())
+ }
+
+ pub fn as_view<T: ArrayElement>(&self) -> ArrayView<'_, T, IxDyn> {
+ ArrayView::from_shape(IxDyn(self.shape()), self.as_slice::<T>())
+ .expect("Failed to create array view")
+ }
+
+ pub fn copy_from_array<C: Backend>(
+ &mut self,
+ other: &Array<C>,
+ ) {
+ assert_eq!(self.shape, other.shape);
+ assert_eq!(self.data_type, other.data_type);
+
+ self.as_bytes_mut().copy_from_slice(other.as_bytes());
+ }
+
+ pub fn copy_slice<C: Backend>(
+ &mut self,
+ source: &Array<C>,
+ axis: usize,
+ src_range: Range<usize>,
+ dst_offset: usize,
+ ) {
+ assert_eq!(self.shape().len(), source.shape().len(), "Rank mismatch");
+ assert!(axis < self.shape().len(), "Axis out of bounds");
+ for (i, (a, b)) in self.shape().iter().zip(source.shape()).enumerate() {
+ if i != axis {
+ assert_eq!(a, b, "Shapes must match on all non-sliced axes");
+ }
+ }
+ assert_eq!(
+ self.data_type(),
+ source.data_type(),
+ "Arrays must have the same data type"
+ );
+
+ let elem_size = self.data_type().size_in_bytes();
+
+ // The number of contiguous elements to copy for each slice operation
+ let block_size_elems =
+ self.shape().iter().skip(axis + 1).product::<usize>();
+ let block_size_bytes = block_size_elems * elem_size;
+
+ // The total number of blocks to copy
+ let num_blocks: usize = self.shape().iter().take(axis).product();
+
+ // Strides between the start of each block
+ let src_stride_bytes = source.shape()[axis] * block_size_bytes;
+ let dst_stride_bytes = self.shape()[axis] * block_size_bytes;
+
+ let rows_to_copy = src_range.end - src_range.start;
+ assert!(dst_offset + rows_to_copy <= self.shape()[axis]);
+ assert!(src_range.end <= source.shape()[axis]);
+ let src_buf = source.as_bytes();
+ let dst_buf = self.as_bytes_mut();
+ let copy_bytes = rows_to_copy * block_size_bytes;
+
+ for i in 0..num_blocks {
+ let src_block_start = i * src_stride_bytes;
+ let dst_block_start = i * dst_stride_bytes;
+
+ let src_start =
+ src_block_start + src_range.start * block_size_bytes;
+ let dst_start = dst_block_start + dst_offset * block_size_bytes;
+
+ let src_slice = &src_buf[src_start..src_start + copy_bytes];
+ let dst_slice = &mut dst_buf[dst_start..dst_start + copy_bytes];
+ dst_slice.copy_from_slice(src_slice);
+ }
+ }
+
+ pub fn copy_from_view<T: ArrayElement, D: Dimension>(
+ &mut self,
+ view: ArrayView<T, D>,
+ ) {
+ assert_eq!(self.data_type(), T::data_type());
+
+ let dst_slice = self.as_slice_mut::<T>();
+
+ if let Some(src_slice) = view.as_slice_memory_order() {
+ assert!(src_slice.len() <= dst_slice.len());
+ dst_slice[..src_slice.len()].copy_from_slice(src_slice);
+ } else {
+ assert!(view.len() <= dst_slice.len());
+ for (d, s) in dst_slice.iter_mut().zip(view.iter()) {
+ *d = *s;
+ }
+ }
+ }
+}
+
+pub trait ArrayCellExt<B: Backend> {
+ fn view(
+ &self,
+ shape: &[usize],
+ ) -> RefCell<Array<B>>;
+}
+
+impl<B: Backend> ArrayCellExt<B> for RefCell<Array<B>> {
+ fn view(
+ &self,
+ shape: &[usize],
+ ) -> RefCell<Array<B>> {
+ RefCell::new(self.borrow().view(shape))
+ }
+}
+
+pub trait ArrayContextExt {
+ type Backend: Backend;
+
+ fn create_array_uninitialized(
+ &self,
+ shape: &[usize],
+ data_type: DataType,
+ label: &str,
+ ) -> Array<Self::Backend>;
+
+ fn create_array(
+ &self,
+ shape: &[usize],
+ data_type: DataType,
+ label: &str,
+ ) -> Array<Self::Backend> {
+ let mut array =
+ self.create_array_uninitialized(shape, data_type, label);
+ array.as_bytes_mut().fill(0);
+ array
+ }
+}
+
+impl<C: Context> ArrayContextExt for C {
+ type Backend = C::Backend;
+
+ fn create_array_uninitialized(
+ &self,
+ shape: &[usize],
+ data_type: DataType,
+ label: &str,
+ ) -> Array<Self::Backend> {
+ let buffer_size_bytes = size_for_shape(shape, data_type);
+
+ let buffer = self
+ .create_buffer(buffer_size_bytes)
+ .expect("Failed to create buffer");
+ buffer.set_label(Some(label));
+
+ unsafe { Array::from_parts(buffer, 0, shape, data_type) }
+ }
+}
diff --git a/crates/uzu/src/backends/common/allocator/allocator.rs b/crates/uzu/src/backends/common/allocator/allocator.rs
new file mode 100644
index 000000000..57e1ad196
--- /dev/null
+++ b/crates/uzu/src/backends/common/allocator/allocator.rs
@@ -0,0 +1,192 @@
+use std::{
+ cell::{Cell, RefCell},
+ rc::Weak,
+};
+
+use super::{
+ AllocError,
+ scratch_pool::{ScratchPool, align_size},
+};
+use crate::backends::common::{
+ Backend, Buffer, BufferLifetime, Context, NativeBuffer,
+};
+
+pub struct Allocator<B: Backend> {
+ context: Weak<B::Context>,
+ scratch_pool: RefCell<ScratchPool<B::NativeBuffer>>,
+ active_memory: Cell<usize>,
+ peak_memory: Cell<usize>,
+ gc_limit: Cell<usize>,
+ max_pool_size: Cell<usize>,
+}
+
+impl<B: Backend> Allocator<B> {
+ pub fn new(context: Weak<B::Context>) -> Self {
+ Self {
+ context,
+ scratch_pool: RefCell::new(ScratchPool::new()),
+ active_memory: Cell::new(0),
+ peak_memory: Cell::new(0),
+ gc_limit: Cell::new(usize::MAX),
+ max_pool_size: Cell::new(usize::MAX),
+ }
+ }
+
+ fn create_native_buffer(
+ &self,
+ size: usize,
+ ) -> Result<B::NativeBuffer, AllocError> {
+ let context = self.context.upgrade().ok_or_else(|| {
+ AllocError::AllocationFailed {
+ size,
+ reason: "context has been dropped".to_string(),
+ }
+ })?;
+ context.create_buffer(size).map_err(|e| AllocError::AllocationFailed {
+ size,
+ reason: e.to_string(),
+ })
+ }
+
+ fn wrap_buffer(
+ &self,
+ native: B::NativeBuffer,
+ is_scratch: bool,
+ ) -> Buffer<B> {
+ Buffer::new(native, self.context.clone(), is_scratch)
+ }
+
+ pub fn set_gc_limit(
+ &self,
+ limit: usize,
+ ) {
+ self.gc_limit.set(limit);
+ }
+
+ pub fn set_max_pool_size(
+ &self,
+ limit: usize,
+ ) {
+ self.max_pool_size.set(limit);
+ }
+
+ pub fn gc_limit(&self) -> usize {
+ self.gc_limit.get()
+ }
+
+ pub fn max_pool_size(&self) -> usize {
+ self.max_pool_size.get()
+ }
+
+ #[inline]
+ fn track_allocation(
+ &self,
+ size: usize,
+ ) {
+ let new_active = self.active_memory.get() + size;
+ self.active_memory.set(new_active);
+ if new_active > self.peak_memory.get() {
+ self.peak_memory.set(new_active);
+ }
+ }
+
+ #[inline]
+ fn track_deallocation(
+ &self,
+ size: usize,
+ ) {
+ self.active_memory.set(self.active_memory.get().saturating_sub(size));
+ }
+
+ fn alloc_scratch_internal(
+ &self,
+ aligned_size: usize,
+ ) -> Result<B::NativeBuffer, AllocError> {
+ let mut pool = self.scratch_pool.borrow_mut();
+
+ if let Some((buffer, actual_size)) = pool.find_buffer(aligned_size) {
+ self.track_allocation(actual_size);
+ return Ok(buffer);
+ }
+
+ let gc_limit = self.gc_limit.get();
+ let active = self.active_memory.get();
+ let cache = pool.available_size();
+ let mem_required = active + cache + aligned_size;
+
+ if mem_required >= gc_limit {
+ let to_free = mem_required.saturating_sub(gc_limit);
+ pool.release_cached_buffers(to_free);
+ }
+
+ pool.total_allocated += aligned_size;
+ drop(pool);
+
+ let buffer = self.create_native_buffer(aligned_size)?;
+ self.track_allocation(aligned_size);
+
+ Ok(buffer)
+ }
+
+ pub fn alloc(
+ &self,
+ lifetime: BufferLifetime,
+ size: usize,
+ ) -> Result<Buffer<B>, AllocError> {
+ let aligned_size = align_size(size);
+
+ let (native, is_scratch) = match lifetime {
+ BufferLifetime::Permanent => {
+ let buffer = self.create_native_buffer(aligned_size)?;
+ self.track_allocation(aligned_size);
+ (buffer, false)
+ },
+ BufferLifetime::Scratch => {
+ (self.alloc_scratch_internal(aligned_size)?, true)
+ },
+ };
+
+ Ok(self.wrap_buffer(native, is_scratch))
+ }
+
+ pub fn handle_buffer_drop(
+ &self,
+ buffer: B::NativeBuffer,
+ is_scratch: bool,
+ ) {
+ let size = buffer.length();
+ self.track_deallocation(size);
+
+ if is_scratch {
+ let mut pool = self.scratch_pool.borrow_mut();
+ pool.return_buffer(buffer, size);
+
+ let max_pool_size = self.max_pool_size.get();
+ let current_size = pool.available_size();
+ if current_size > max_pool_size {
+ let to_free = current_size - max_pool_size;
+ pool.release_cached_buffers(to_free);
+ }
+ }
+ }
+
+ pub fn active_memory(&self) -> usize {
+ self.active_memory.get()
+ }
+
+ pub fn peak_memory(&self) -> usize {
+ self.peak_memory.get()
+ }
+
+ pub fn reset_peak_memory(&self) {
+ self.peak_memory.set(self.active_memory.get());
+ }
+
+ pub fn cache_memory(&self) -> usize {
+ self.scratch_pool.borrow().available_size()
+ }
+
+ pub fn clear_cache(&self) {
+ self.scratch_pool.borrow_mut().clear();
+ }
+}
diff --git a/crates/uzu/src/backends/common/allocator/error.rs b/crates/uzu/src/backends/common/allocator/error.rs
new file mode 100644
index 000000000..f6c4c949c
--- /dev/null
+++ b/crates/uzu/src/backends/common/allocator/error.rs
@@ -0,0 +1,41 @@
+use std::fmt;
+
+#[derive(Debug, Clone)]
+pub enum AllocError {
+ OutOfMemory {
+ requested: usize,
+ available: usize,
+ },
+ AllocationFailed {
+ size: usize,
+ reason: String,
+ },
+}
+
+impl fmt::Display for AllocError {
+ fn fmt(
+ &self,
+ f: &mut fmt::Formatter<'_>,
+ ) -> fmt::Result {
+ match self {
+ AllocError::OutOfMemory {
+ requested,
+ available,
+ } => {
+ write!(
+ f,
+ "Out of memory: requested {} bytes, {} available",
+ requested, available
+ )
+ },
+ AllocError::AllocationFailed {
+ size,
+ reason,
+ } => {
+ write!(f, "Failed to allocate {} bytes: {}", size, reason)
+ },
+ }
+ }
+}
+
+impl std::error::Error for AllocError {}
diff --git a/crates/uzu/src/backends/common/allocator/mod.rs b/crates/uzu/src/backends/common/allocator/mod.rs
new file mode 100644
index 000000000..a7ffa15d5
--- /dev/null
+++ b/crates/uzu/src/backends/common/allocator/mod.rs
@@ -0,0 +1,6 @@
+mod allocator;
+mod error;
+mod scratch_pool;
+
+pub use allocator::Allocator;
+pub use error::AllocError;
diff --git a/crates/uzu/src/backends/common/allocator/scratch_pool.rs b/crates/uzu/src/backends/common/allocator/scratch_pool.rs
new file mode 100644
index 000000000..1c62f71f9
--- /dev/null
+++ b/crates/uzu/src/backends/common/allocator/scratch_pool.rs
@@ -0,0 +1,257 @@
+use std::collections::BTreeMap;
+
+pub(super) const PAGE_SIZE: usize = 16384;
+
+pub(super) fn align_size(size: usize) -> usize {
+ if size > PAGE_SIZE {
+ (size + PAGE_SIZE - 1) & !(PAGE_SIZE - 1)
+ } else {
+ size
+ }
+}
+
+struct BufferNode<B> {
+ buffer: B,
+ size: usize,
+ bucket_idx: usize,
+ prev: Option<usize>,
+ next: Option<usize>,
+}
+
+pub(super) struct ScratchPool<B> {
+ nodes: Vec<Option<BufferNode<B>>>,
+ free_slots: Vec<usize>,
+ size_index: BTreeMap<usize, Vec<usize>>,
+ lru_head: Option<usize>,
+ lru_tail: Option<usize>,
+ pool_size: usize,
+ pub(super) total_allocated: usize,
+}
+
+impl<B> ScratchPool<B> {
+ pub(super) fn new() -> Self {
+ Self {
+ nodes: Vec::new(),
+ free_slots: Vec::new(),
+ size_index: BTreeMap::new(),
+ lru_head: None,
+ lru_tail: None,
+ pool_size: 0,
+ total_allocated: 0,
+ }
+ }
+
+ #[inline]
+ fn allocate_slot(&mut self) -> usize {
+ self.free_slots.pop().unwrap_or_else(|| {
+ let slot = self.nodes.len();
+ self.nodes.push(None);
+ slot
+ })
+ }
+
+ #[inline]
+ fn add_to_lru_head(
+ &mut self,
+ slot: usize,
+ ) {
+ if let Some(old_head) = self.lru_head {
+ if let Some(node) = self.nodes[old_head].as_mut() {
+ node.prev = Some(slot);
+ }
+ if let Some(node) = self.nodes[slot].as_mut() {
+ node.next = Some(old_head);
+ }
+ }
+
+ self.lru_head = Some(slot);
+
+ if self.lru_tail.is_none() {
+ self.lru_tail = Some(slot);
+ }
+ }
+
+ #[inline]
+ fn remove_from_lru(
+ &mut self,
+ slot: usize,
+ ) {
+ let (prev, next) = match self.nodes[slot].as_ref() {
+ Some(node) => (node.prev, node.next),
+ None => return,
+ };
+
+ match (prev, next) {
+ (Some(p), Some(n)) => {
+ if let Some(prev_node) = self.nodes[p].as_mut() {
+ prev_node.next = Some(n);
+ }
+ if let Some(next_node) = self.nodes[n].as_mut() {
+ next_node.prev = Some(p);
+ }
+ },
+ (Some(p), None) => {
+ if let Some(prev_node) = self.nodes[p].as_mut() {
+ prev_node.next = None;
+ }
+ self.lru_tail = Some(p);
+ },
+ (None, Some(n)) => {
+ if let Some(next_node) = self.nodes[n].as_mut() {
+ next_node.prev = None;
+ }
+ self.lru_head = Some(n);
+ },
+ (None, None) => {
+ self.lru_head = None;
+ self.lru_tail = None;
+ },
+ }
+
+ if let Some(node) = self.nodes[slot].as_mut() {
+ node.prev = None;
+ node.next = None;
+ }
+ }
+
+ #[inline]
+ fn remove_slot_from_bucket(
+ &mut self,
+ size: usize,
+ _slot: usize,
+ bucket_idx: usize,
+ ) {
+ if let Some(slots) = self.size_index.get_mut(&size) {
+ let last_idx = slots.len() - 1;
+ if bucket_idx != last_idx {
+ let swapped_slot = slots[last_idx];
+ slots[bucket_idx] = swapped_slot;
+ if let Some(swapped_node) = self.nodes[swapped_slot].as_mut() {
+ swapped_node.bucket_idx = bucket_idx;
+ }
+ }
+ slots.pop();
+ if slots.is_empty() {
+ self.size_index.remove(&size);
+ }
+ }
+ }
+
+ #[inline]
+ pub(super) fn find_buffer(
+ &mut self,
+ size: usize,
+ ) -> Option<(B, usize)> {
+ let max_size = (2 * size).min(size + 2 * PAGE_SIZE);
+
+ let mut found_key = None;
+ let mut found_slot = None;
+ let mut found_bucket_idx = None;
+
+ for (&key, slots) in self.size_index.range(size..=max_size) {
+ if let Some(&slot) = slots.last() {
+ found_key = Some(key);
+ found_slot = Some(slot);
+ found_bucket_idx = Some(slots.len() - 1);
+ break;
+ }
+ }
+
+ let (key, slot, bucket_idx) =
+ match (found_key, found_slot, found_bucket_idx) {
+ (Some(k), Some(s), Some(i)) => (k, s, i),
+ _ => return None,
+ };
+
+ self.remove_slot_from_bucket(key, slot, bucket_idx);
+ self.remove_from_lru(slot);
+
+ let node = self.nodes[slot].take()?;
+ self.free_slots.push(slot);
+ self.pool_size = self.pool_size.saturating_sub(key);
+
+ Some((node.buffer, key))
+ }
+
+ #[inline]
+ pub(super) fn return_buffer(
+ &mut self,
+ buffer: B,
+ size: usize,
+ ) {
+ let slot = self.allocate_slot();
+ let bucket_idx = self.size_index.get(&size).map_or(0, |v| v.len());
+
+ self.nodes[slot] = Some(BufferNode {
+ buffer,
+ size,
+ bucket_idx,
+ prev: None,
+ next: None,
+ });
+
+ self.add_to_lru_head(slot);
+ self.size_index.entry(size).or_default().push(slot);
+ self.pool_size += size;
+ }
+
+ pub(super) fn release_cached_buffers(
+ &mut self,
+ min_bytes_to_free: usize,
+ ) -> usize {
+ if min_bytes_to_free == 0 {
+ return 0;
+ }
+
+ if min_bytes_to_free >= (self.pool_size * 9) / 10 {
+ return self.clear();
+ }
+
+ let mut total_freed = 0usize;
+ let mut buffers_released = 0usize;
+
+ while total_freed < min_bytes_to_free {
+ let tail_slot = match self.lru_tail {
+ Some(s) => s,
+ None => break,
+ };
+
+ let (size, bucket_idx) = match self.nodes[tail_slot].as_ref() {
+ Some(n) => (n.size, n.bucket_idx),
+ None => break,
+ };
+
+ self.remove_slot_from_bucket(size, tail_slot, bucket_idx);
+ self.remove_from_lru(tail_slot);
+ self.nodes[tail_slot] = None;
+ self.free_slots.push(tail_slot);
+
+ total_freed += size;
+ buffers_released += 1;
+ self.pool_size = self.pool_size.saturating_sub(size);
+ self.total_allocated = self.total_allocated.saturating_sub(size);
+ }
+
+ buffers_released
+ }
+
+ #[inline]
+ pub(super) fn available_size(&self) -> usize {
+ self.pool_size
+ }
+
+ pub(super) fn clear(&mut self) -> usize {
+ let count = self.nodes.iter().filter(|n| n.is_some()).count();
+
+ self.nodes.clear();
+ self.free_slots.clear();
+ self.size_index.clear();
+ self.lru_head = None;
+ self.lru_tail = None;
+ self.total_allocated =
+ self.total_allocated.saturating_sub(self.pool_size);
+ self.pool_size = 0;
+
+ count
+ }
+}
diff --git a/crates/uzu/src/backends/common/backend.rs b/crates/uzu/src/backends/common/backend.rs
new file mode 100644
index 000000000..17a2e9782
--- /dev/null
+++ b/crates/uzu/src/backends/common/backend.rs
@@ -0,0 +1,12 @@
+use std::error::Error;
+
+use super::{CommandBuffer, Context, Kernels, NativeBuffer};
+
+pub trait Backend: Clone {
+ type Context: Context<Backend = Self>;
+ type CommandBuffer: CommandBuffer<Backend = Self>;
+ type NativeBuffer: NativeBuffer<Backend = Self>;
+ type EncoderRef;
+ type Kernels: Kernels<Backend = Self>;
+ type Error: Error;
+}
diff --git a/crates/uzu/src/backends/common/buffer.rs b/crates/uzu/src/backends/common/buffer.rs
new file mode 100644
index 000000000..8630de880
--- /dev/null
+++ b/crates/uzu/src/backends/common/buffer.rs
@@ -0,0 +1,38 @@
+use std::{mem::ManuallyDrop, rc::Weak};
+
+use super::{Backend, Context};
+
+pub struct Buffer<B: Backend> {
+ inner: ManuallyDrop<B::NativeBuffer>,
+ context: Weak<B::Context>,
+ is_scratch: bool,
+}
+
+impl<B: Backend> Buffer<B> {
+ pub fn new(
+ inner: B::NativeBuffer,
+ context: Weak<B::Context>,
+ is_scratch: bool,
+ ) -> Self {
+ Self {
+ inner: ManuallyDrop::new(inner),
+ context,
+ is_scratch,
+ }
+ }
+
+ pub fn inner(&self) -> &B::NativeBuffer {
+ &self.inner
+ }
+}
+
+impl<B: Backend> Drop for Buffer<B> {
+ fn drop(&mut self) {
+ // Safety: drop is only called once, inner is valid
+ let inner = unsafe { ManuallyDrop::take(&mut self.inner) };
+ if let Some(ctx) = self.context.upgrade() {
+ ctx.allocator().handle_buffer_drop(inner, self.is_scratch);
+ }
+ // If context is gone, inner drops naturally here
+ }
+}
diff --git a/crates/uzu/src/backends/common/buffer_lifetime.rs b/crates/uzu/src/backends/common/buffer_lifetime.rs
new file mode 100644
index 000000000..65cf6b5a7
--- /dev/null
+++ b/crates/uzu/src/backends/common/buffer_lifetime.rs
@@ -0,0 +1,5 @@
+#[derive(Clone, Copy, Debug, PartialEq, Eq)]
+pub enum BufferLifetime {
+ Permanent,
+ Scratch,
+}
diff --git a/crates/uzu/src/backends/common/command_buffer.rs b/crates/uzu/src/backends/common/command_buffer.rs
new file mode 100644
index 000000000..0315e12ad
--- /dev/null
+++ b/crates/uzu/src/backends/common/command_buffer.rs
@@ -0,0 +1,10 @@
+use super::Backend;
+
+pub trait CommandBuffer {
+ type Backend: Backend;
+
+ fn with_encoder<T>(
+ &self,
+ callback: impl FnOnce(&<Self::Backend as Backend>::EncoderRef) -> T,
+ ) -> T;
+}
diff --git a/crates/uzu/src/backends/common/context.rs b/crates/uzu/src/backends/common/context.rs
new file mode 100644
index 000000000..cb674cda9
--- /dev/null
+++ b/crates/uzu/src/backends/common/context.rs
@@ -0,0 +1,26 @@
+use std::rc::Rc;
+
+use super::{Allocator, Backend};
+
+pub trait Context: Sized {
+ type Backend: Backend<Context = Self>;
+
+ fn new() -> Result<Rc<Self>, <Self::Backend as Backend>::Error>;
+
+ fn allocator(&self) -> &Allocator<Self::Backend>;
+
+ fn create_buffer(
+ &self,
+ size: usize,
+ ) -> Result<
+ <Self::Backend as Backend>::NativeBuffer,
+ <Self::Backend as Backend>::Error,
+ >;
+
+ fn create_command_buffer(
+ &self
+ ) -> Result<
+ <Self::Backend as Backend>::CommandBuffer,
+ <Self::Backend as Backend>::Error,
+ >;
+}
diff --git a/crates/uzu/src/backends/common/gpu_types/attention.rs b/crates/uzu/src/backends/common/gpu_types/attention.rs
new file mode 100644
index 000000000..f608e13d1
--- /dev/null
+++ b/crates/uzu/src/backends/common/gpu_types/attention.rs
@@ -0,0 +1,49 @@
+//! Attention kernel parameter structs.
+
+/// Parameters for the GEMM attention kernel.
+///
+/// All strides are in **elements**, not bytes.
+/// Q/K/V/O are treated as 3D tensors with strides for:
+/// - `[0]` batch
+/// - `[1]` head (or kv-head for K/V)
+/// - `[2]` sequence (row stride for [seq, head_dim])
+#[repr(C)]
+#[derive(Debug, Default, Copy, Clone)]
+pub struct AttnParams {
+ pub q_strides: [i64; 3],
+ pub k_strides: [i64; 3],
+ pub v_strides: [i64; 3],
+ pub o_strides: [i64; 3],
+ pub gqa_factor: i32,
+ pub scale: f32,
+ /// Query length (suffix length)
+ pub q_len: i32,
+ /// Key length (prefix + suffix)
+ pub k_len: i32,
+ /// Absolute offset of the first query token in the full key sequence.
+ /// For LLM decode/prefill this is the segment prefix length.
+ pub q_off: i32,
+ /// Query tiling: q_len / BQ
+ pub nq_aligned: i32,
+ /// Query tiling: q_len % BQ
+ pub q_rem: i32,
+ /// Key tiling: ceil(k_len / BK)
+ pub nk: i32,
+ /// Key tiling: k_len / BK
+ pub nk_aligned: i32,
+ /// Key tiling: k_len % BK
+ pub k_rem: i32,
+}
+
+/// Parameters describing the additive attention mask.
+///
+/// Mask is treated as a 2D matrix [qL, kL] with:
+/// - per-batch base offset (`m_strides[0]`)
+/// - per-head base offset (`m_strides[1]`)
+/// - row stride in elements (`m_strides[2]`)
+/// Column stride is assumed to be 1.
+#[repr(C)]
+#[derive(Debug, Default, Copy, Clone)]
+pub struct AttnMaskParams {
+ pub m_strides: [i64; 3],
+}
diff --git a/crates/uzu/src/backends/common/gpu_types/matmul.rs b/crates/uzu/src/backends/common/gpu_types/matmul.rs
new file mode 100644
index 000000000..dbcc1d012
--- /dev/null
+++ b/crates/uzu/src/backends/common/gpu_types/matmul.rs
@@ -0,0 +1,74 @@
+//! GEMM kernel parameter structs.
+//!
+//! Field names use uppercase for matrix dimensions (M, N, K) to match
+//! standard BLAS/GEMM convention and generated Metal headers.
+
+#![allow(non_snake_case)]
+
+/// Main GEMM parameters passed to Metal kernels as constant buffer.
+#[repr(C)]
+#[derive(Debug, Default, Copy, Clone)]
+pub struct GEMMParams {
+ pub M: i32,
+ pub N: i32,
+ pub K: i32,
+ pub lda: i32,
+ pub ldb: i32,
+ pub ldd: i32,
+ pub tiles_n: i32,
+ pub tiles_m: i32,
+ pub batch_stride_a: i64,
+ pub batch_stride_b: i64,
+ pub batch_stride_d: i64,
+ pub swizzle_log: i32,
+ pub gemm_k_iterations_aligned: i32,
+ pub batch_ndim: i32,
+}
+
+/// Split-K GEMM parameters.
+#[repr(C)]
+#[derive(Debug, Default, Copy, Clone)]
+pub struct GEMMSpiltKParams {
+ pub M: i32,
+ pub N: i32,
+ pub K: i32,
+ pub lda: i32,
+ pub ldb: i32,
+ pub ldc: i32,
+ pub tiles_n: i32,
+ pub tiles_m: i32,
+ pub split_k_partitions: i32,
+ pub split_k_partition_stride: i32,
+ pub split_k_partition_size: i32,
+ pub gemm_k_iterations_aligned: i32,
+}
+
+/// Split-K MLP Fused GEMM parameters.
+#[repr(C)]
+#[derive(Debug, Default, Copy, Clone)]
+pub struct GEMMSpiltKMlpFusedParams {
+ pub M: i32,
+ pub N: i32,
+ pub K: i32,
+ pub lda: i32,
+ pub ldb: i32,
+ pub ldc: i32,
+ pub tiles_n: i32,
+ pub tiles_m: i32,
+ pub split_k_partitions: i32,
+ pub split_k_partition_stride: i32,
+ pub split_k_partition_size: i32,
+ pub gemm_k_iterations_aligned: i32,
+ pub hidden_dim: i32,
+}
+
+/// AddMM parameters (alpha * A @ B + beta * C).
+#[repr(C)]
+#[derive(Debug, Default, Copy, Clone)]
+pub struct GEMMAddMMParams {
+ pub ldc: i32,
+ pub fdc: i32,
+ pub batch_stride_c: i64,
+ pub alpha: f32,
+ pub beta: f32,
+}
diff --git a/crates/uzu/src/backends/common/gpu_types/mod.rs b/crates/uzu/src/backends/common/gpu_types/mod.rs
new file mode 100644
index 000000000..795fdc68b
--- /dev/null
+++ b/crates/uzu/src/backends/common/gpu_types/mod.rs
@@ -0,0 +1,10 @@
+//! GPU types shared between Rust and shader languages (Metal, HLSL, Slang).
+//!
+//! These `#[repr(C)]` structs are the source of truth. The build system uses
+//! cbindgen to generate C headers for Metal shaders.
+
+mod attention;
+mod matmul;
+
+pub use attention::*;
+pub use matmul::*;
diff --git a/crates/uzu/src/backends/common/kernel/mod.rs b/crates/uzu/src/backends/common/kernel/mod.rs
new file mode 100644
index 000000000..dc38e9b24
--- /dev/null
+++ b/crates/uzu/src/backends/common/kernel/mod.rs
@@ -0,0 +1,5 @@
+#![allow(non_snake_case)]
+
+include!(concat!(env!("OUT_DIR"), "/traits.rs"));
+
+pub mod sampling;
diff --git a/crates/uzu/src/backends/metal/kernel/sampling/mod.rs b/crates/uzu/src/backends/common/kernel/sampling.rs
similarity index 67%
rename from crates/uzu/src/backends/metal/kernel/sampling/mod.rs
rename to crates/uzu/src/backends/common/kernel/sampling.rs
index 463e502f6..bb8737f48 100644
--- a/crates/uzu/src/backends/metal/kernel/sampling/mod.rs
+++ b/crates/uzu/src/backends/common/kernel/sampling.rs
@@ -3,17 +3,20 @@ use std::mem::size_of;
use thiserror::Error;
use crate::{
- backends::metal::{
- KernelDataType, MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- MTLContext, MTLDeviceExt, MTLError, MTLResourceOptions, ProtocolObject, Retained,
- kernel::dsl::{
- ArgmaxFinalKernel, ArgmaxMainKernel, ArgmaxSingleKernel, BitmaskKernel, GumbelKernel,
- MinPKernel, TemperatureKernel, TopKKernel, TopPKernel,
+ DataType,
+ backends::common::{
+ Context,
+ kernel::{
+ ArgmaxFinalKernel, ArgmaxMainKernel, ArgmaxSingleKernel,
+ BitmaskKernel, GumbelKernel, MinPKernel, TemperatureKernel,
+ TopKKernel, TopPKernel,
},
},
session::parameter::SamplingMethod,
};
+use super::{Backend, Kernels};
+
#[derive(Debug, Clone, Copy, PartialEq)]
pub enum ArgmaxStrategy {
SinglePass, // One threadgroup per batch item
@@ -36,33 +39,33 @@ impl Default for ArgmaxPair {
}
}
-enum ArgmaxImplementation {
+enum ArgmaxImplementation<B: Backend> {
SinglePass {
- kernel: ArgmaxSingleKernel,
+ kernel: <B::Kernels as Kernels>::ArgmaxSingleKernel,
},
TwoPass {
- main_kernel: ArgmaxMainKernel,
- final_kernel: ArgmaxFinalKernel,
- partial_results_buffer: Retained<ProtocolObject<dyn MTLBuffer>>,
+ main_kernel: <B::Kernels as Kernels>::ArgmaxMainKernel,
+ final_kernel: <B::Kernels as Kernels>::ArgmaxFinalKernel,
+ partial_results_buffer: B::NativeBuffer,
},
}
-pub struct SamplingKernel {
- bitmask: BitmaskKernel,
- temperature: TemperatureKernel,
- topk: TopKKernel,
- topp: TopPKernel,
- minp: MinPKernel,
- gumbel: GumbelKernel,
- argmax_implementation: ArgmaxImplementation,
+pub struct SamplingKernel<B: Backend> {
+ bitmask: <B::Kernels as Kernels>::BitmaskKernel,
+ temperature: <B::Kernels as Kernels>::TemperatureKernel,
+ topk: <B::Kernels as Kernels>::TopKKernel,
+ topp: <B::Kernels as Kernels>::TopPKernel,
+ minp: <B::Kernels as Kernels>::MinPKernel,
+ gumbel: <B::Kernels as Kernels>::GumbelKernel,
+ argmax_implementation: ArgmaxImplementation<B>,
max_batch_size: usize,
max_vocab_size: usize,
}
#[derive(Debug, Error)]
-pub enum SamplingError {
- #[error("Metal error: {0}")]
- MetalError(#[from] MTLError),
+pub enum SamplingError<B: Backend> {
+ #[error("Backend error: {0}")]
+ BackendError(#[source] B::Error),
#[error("Function not found: {0}")]
FunctionNotFound(String),
#[error("Batch size {0} exceeds maximum {1}")]
@@ -73,13 +76,13 @@ pub enum SamplingError {
StochasticWithoutSeed,
}
-impl SamplingKernel {
+impl<B: Backend> SamplingKernel<B> {
pub fn new(
- context: &MTLContext,
- data_type: KernelDataType,
+ context: &B::Context,
+ data_type: DataType,
max_batch_size: usize,
max_vocab_size: usize,
- ) -> Result<Self, SamplingError> {
+ ) -> Result<Self, SamplingError<B>> {
Self::new_with_strategy(
context,
data_type,
@@ -90,30 +93,48 @@ impl SamplingKernel {
}
pub fn new_with_strategy(
- context: &MTLContext,
- data_type: KernelDataType,
+ context: &B::Context,
+ data_type: DataType,
max_batch_size: usize,
max_vocab_size: usize,
argmax_strategy: ArgmaxStrategy,
- ) -> Result<Self, SamplingError> {
- let bitmask = BitmaskKernel::new(context, data_type)?;
- let temperature = TemperatureKernel::new(context, data_type)?;
- let topk = TopKKernel::new(context, data_type)?;
- let topp = TopPKernel::new(context, data_type)?;
- let minp = MinPKernel::new(context, data_type)?;
- let gumbel = GumbelKernel::new(context, data_type)?;
+ ) -> Result<Self, SamplingError<B>> {
+ let bitmask =
+ <B::Kernels as Kernels>::BitmaskKernel::new(context, data_type)
+ .map_err(SamplingError::BackendError)?;
+ let temperature =
+ <B::Kernels as Kernels>::TemperatureKernel::new(context, data_type)
+ .map_err(SamplingError::BackendError)?;
+ let topk = <B::Kernels as Kernels>::TopKKernel::new(context, data_type)
+ .map_err(SamplingError::BackendError)?;
+ let topp = <B::Kernels as Kernels>::TopPKernel::new(context, data_type)
+ .map_err(SamplingError::BackendError)?;
+ let minp = <B::Kernels as Kernels>::MinPKernel::new(context, data_type)
+ .map_err(SamplingError::BackendError)?;
+ let gumbel =
+ <B::Kernels as Kernels>::GumbelKernel::new(context, data_type)
+ .map_err(SamplingError::BackendError)?;
let argmax_implementation = match argmax_strategy {
ArgmaxStrategy::SinglePass => {
- let kernel = ArgmaxSingleKernel::new(context, data_type)?;
+ let kernel = <B::Kernels as Kernels>::ArgmaxSingleKernel::new(
+ context, data_type,
+ )
+ .map_err(SamplingError::BackendError)?;
ArgmaxImplementation::SinglePass {
kernel,
}
},
ArgmaxStrategy::TwoPass => {
- let main_kernel = ArgmaxMainKernel::new(context, data_type)?;
- let final_kernel = ArgmaxFinalKernel::new(context)?;
+ let main_kernel =
+ <B::Kernels as Kernels>::ArgmaxMainKernel::new(
+ context, data_type,
+ )
+ .map_err(SamplingError::BackendError)?;
+ let final_kernel =
+ <B::Kernels as Kernels>::ArgmaxFinalKernel::new(context)
+ .map_err(SamplingError::BackendError)?;
let block_size = 1024;
let grain_size = 4;
@@ -126,10 +147,8 @@ impl SamplingKernel {
max_batch_size * max_vocab_groups_per_batch;
let partial_results_buffer = context
- .device
- .new_buffer(
+ .create_buffer(
max_partial_results * size_of::<ArgmaxPair>(),
- MTLResourceOptions::STORAGE_MODE_SHARED,
)
.expect("Failed to create partial results buffer");
@@ -154,51 +173,19 @@ impl SamplingKernel {
})
}
- pub fn encode(
- &self,
- logits_buffer: &ProtocolObject<dyn MTLBuffer>,
- seeds_buffer: Option<&ProtocolObject<dyn MTLBuffer>>,
- seeds_offset: usize,
- bitmask_buffer: Option<&ProtocolObject<dyn MTLBuffer>>,
- bitmask_offset: usize,
- sampled_tokens_buffer: &ProtocolObject<dyn MTLBuffer>,
- sampling_method: SamplingMethod,
- batch_size: usize,
- vocab_size: usize,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- ) -> Result<(), SamplingError> {
- let compute_encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- self.encode_with_encoder(
- logits_buffer,
- seeds_buffer,
- seeds_offset,
- bitmask_buffer,
- bitmask_offset,
- sampled_tokens_buffer,
- sampling_method,
- batch_size,
- vocab_size,
- &compute_encoder,
- )?;
- compute_encoder.end_encoding();
- Ok(())
- }
-
pub fn encode_with_encoder(
&self,
- logits_buffer: &ProtocolObject<dyn MTLBuffer>,
- seeds_buffer: Option<&ProtocolObject<dyn MTLBuffer>>,
+ logits_buffer: &B::NativeBuffer,
+ seeds_buffer: Option<&B::NativeBuffer>,
seeds_offset: usize,
- bitmask_buffer: Option<&ProtocolObject<dyn MTLBuffer>>,
+ bitmask_buffer: Option<&B::NativeBuffer>,
bitmask_offset: usize,
- sampled_tokens_buffer: &ProtocolObject<dyn MTLBuffer>,
+ sampled_tokens_buffer: &B::NativeBuffer,
sampling_method: SamplingMethod,
batch_size: usize,
vocab_size: usize,
- compute_encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- ) -> Result<(), SamplingError> {
+ compute_encoder: &B::EncoderRef,
+ ) -> Result<(), SamplingError<B>> {
if batch_size > self.max_batch_size {
return Err(SamplingError::BatchSizeExceeded(
batch_size,
diff --git a/crates/uzu/src/backends/common/mod.rs b/crates/uzu/src/backends/common/mod.rs
new file mode 100644
index 000000000..392acdf4c
--- /dev/null
+++ b/crates/uzu/src/backends/common/mod.rs
@@ -0,0 +1,18 @@
+mod allocator;
+mod backend;
+mod buffer;
+mod buffer_lifetime;
+mod command_buffer;
+mod context;
+pub mod gpu_types;
+pub mod kernel;
+mod native_buffer;
+
+pub use allocator::{AllocError, Allocator};
+pub use backend::Backend;
+pub use buffer::Buffer;
+pub use buffer_lifetime::BufferLifetime;
+pub use command_buffer::CommandBuffer;
+pub use context::Context;
+pub use kernel::Kernels;
+pub use native_buffer::NativeBuffer;
diff --git a/crates/uzu/src/backends/common/native_buffer.rs b/crates/uzu/src/backends/common/native_buffer.rs
new file mode 100644
index 000000000..80d09ae5c
--- /dev/null
+++ b/crates/uzu/src/backends/common/native_buffer.rs
@@ -0,0 +1,16 @@
+use std::{os::raw::c_void, ptr::NonNull};
+
+use super::Backend;
+
+pub trait NativeBuffer: Send + Sync + Clone {
+ type Backend: Backend;
+
+ fn set_label(
+ &self,
+ label: Option<&str>,
+ );
+
+ fn cpu_ptr(&self) -> NonNull<c_void>;
+ fn length(&self) -> usize;
+ fn id(&self) -> usize;
+}
diff --git a/crates/uzu/src/backends/cpu/common.rs b/crates/uzu/src/backends/cpu/common.rs
deleted file mode 100644
index b160f8c21..000000000
--- a/crates/uzu/src/backends/cpu/common.rs
+++ /dev/null
@@ -1,77 +0,0 @@
-use crate::{Array, DataType, DeviceContext, array::array_size_in_bytes};
-
-#[derive(Debug, Clone)]
-pub struct CPUArray {
- buffer: Box<[u8]>,
- shape: Box<[usize]>,
- data_type: DataType,
- label: String,
-}
-
-impl Array for CPUArray {
- type BackendBuffer = Box<[u8]>;
-
- fn shape(&self) -> &[usize] {
- &self.shape
- }
-
- fn label(&self) -> String {
- self.label.clone()
- }
-
- fn data_type(&self) -> DataType {
- self.data_type
- }
-
- fn buffer(&self) -> &[u8] {
- &self.buffer
- }
-
- fn buffer_mut(&mut self) -> &mut [u8] {
- &mut self.buffer
- }
-
- fn backend_buffer(&self) -> &Box<[u8]> {
- &self.buffer
- }
-}
-
-impl CPUArray {
- /// Create a new Array
- pub fn new(
- shape: &[usize],
- data_type: DataType,
- label: String,
- ) -> Self {
- let buffer_size_bytes = array_size_in_bytes(shape, data_type);
- let buffer =
- std::iter::repeat(0 as u8).take(buffer_size_bytes).collect();
- Self {
- buffer,
- shape: shape.into(),
- data_type,
- label,
- }
- }
-}
-
-pub struct CPUContext {}
-
-impl CPUContext {
- pub fn new() -> Self {
- Self {}
- }
-}
-
-impl DeviceContext for CPUContext {
- type DeviceArray = CPUArray;
-
- unsafe fn array_uninitialized(
- &self,
- shape: &[usize],
- data_type: DataType,
- label: String,
- ) -> CPUArray {
- CPUArray::new(shape, data_type, label)
- }
-}
diff --git a/crates/uzu/src/backends/cpu/mod.rs b/crates/uzu/src/backends/cpu/mod.rs
index 2613d1531..5d259bb5f 100644
--- a/crates/uzu/src/backends/cpu/mod.rs
+++ b/crates/uzu/src/backends/cpu/mod.rs
@@ -1,3 +1 @@
pub mod argmax;
-pub mod common;
-pub use common::{CPUArray, CPUContext};
diff --git a/crates/uzu/src/backends/metal/allocator.rs b/crates/uzu/src/backends/metal/allocator.rs
new file mode 100644
index 000000000..8db8a15ce
--- /dev/null
+++ b/crates/uzu/src/backends/metal/allocator.rs
@@ -0,0 +1,31 @@
+use std::{ops::Deref, os::raw::c_void, ptr::NonNull};
+
+use metal::{MTLBuffer, MTLResourceExt};
+use objc2::{rc::Retained, runtime::ProtocolObject};
+
+use crate::backends::common::NativeBuffer;
+
+use super::Metal;
+
+impl NativeBuffer for Retained<ProtocolObject<dyn MTLBuffer>> {
+ type Backend = Metal;
+
+ fn set_label(
+ &self,
+ label: Option<&str>,
+ ) {
+ MTLResourceExt::set_label(self.deref(), label);
+ }
+
+ fn cpu_ptr(&self) -> NonNull<c_void> {
+ self.contents()
+ }
+
+ fn length(&self) -> usize {
+ self.deref().length()
+ }
+
+ fn id(&self) -> usize {
+ Retained::as_ptr(self) as usize
+ }
+}
diff --git a/crates/uzu/src/backends/metal/array.rs b/crates/uzu/src/backends/metal/array.rs
index babae4ca4..a68cffecf 100644
--- a/crates/uzu/src/backends/metal/array.rs
+++ b/crates/uzu/src/backends/metal/array.rs
@@ -1,278 +1,5 @@
-use std::ops::Range;
+use crate::array::Array;
-use half::{bf16, f16};
+use super::Metal;
-use crate::{
- Array, ArrayElement, DataType, array::array_size_in_bytes,
- backends::metal::{
- MTLBuffer, MTLDeviceExt, MTLResource, MTLResourceExt, MTLResourceOptions, ProtocolObject,
- Retained,
- },
-};
-
-/// Type alias for owned MTLBuffer
-type MTLBufferObj = Retained<ProtocolObject<dyn MTLBuffer>>;
-
-#[derive(Debug, Clone)]
-pub struct MetalArray {
- buffer: MTLBufferObj,
- shape: Box<[usize]>,
- data_type: DataType,
- offset: usize,
-}
-
-impl Array for MetalArray {
- type BackendBuffer = ProtocolObject<dyn MTLBuffer>;
-
- fn shape(&self) -> &[usize] {
- &self.shape
- }
-
- fn data_type(&self) -> DataType {
- self.data_type
- }
-
- fn label(&self) -> String {
- self.buffer.label().unwrap_or_default()
- }
-
- fn buffer(&self) -> &[u8] {
- unsafe {
- std::slice::from_raw_parts(
- (self.buffer.contents().as_ptr() as *const u8).add(self.offset),
- self.size_in_bytes(),
- )
- }
- }
-
- fn buffer_mut(&mut self) -> &mut [u8] {
- unsafe {
- std::slice::from_raw_parts_mut(
- (self.buffer.contents().as_ptr() as *mut u8).add(self.offset),
- self.size_in_bytes(),
- )
- }
- }
-
- fn backend_buffer(&self) -> &ProtocolObject<dyn MTLBuffer> {
- &self.buffer
- }
-}
-
-impl MetalArray {
- pub unsafe fn new(
- buffer: MTLBufferObj,
- shape: &[usize],
- data_type: DataType,
- ) -> Self {
- unsafe { Self::new_with_offset(buffer, shape, data_type, 0) }
- }
-
- /// Returns a clone of the owned MTL buffer (increments reference count)
- pub fn mtl_buffer_cloned(&self) -> MTLBufferObj {
- self.buffer.clone()
- }
-
- pub unsafe fn new_with_offset(
- buffer: MTLBufferObj,
- shape: &[usize],
- data_type: DataType,
- offset: usize,
- ) -> Self {
- let required_bytes = array_size_in_bytes(shape, data_type);
- assert!(
- offset + required_bytes <= buffer.length(),
- "Shape {:?} with data type {:?} at offset {} requires {} bytes total, but buffer length is {} bytes",
- shape,
- data_type,
- offset,
- offset + required_bytes,
- buffer.length()
- );
- Self {
- buffer,
- shape: shape.into(),
- data_type,
- offset,
- }
- }
-
- pub fn buffer_offset(&self) -> usize {
- self.offset
- }
-
- pub unsafe fn mtl_buffer(&mut self) -> &ProtocolObject<dyn MTLBuffer> {
- &self.buffer
- }
-
- /// Splits the array into multiple `MetalArray` objects along the first dimension (rows) only.
- /// This is a zero-copy operation that creates child buffers pointing to the same memory.
- ///
- /// # Arguments
- /// * `row_lengths` - Array of row counts for each split. Must sum to total rows.
- ///
- /// # Example
- /// ```ignore
- /// use uzu::backends::metal::MetalArray;
- /// use uzu::DataType;
- /// use metal::{MTLDevice, MTLDeviceExt, MTLResourceOptions};
- ///
- /// # let device = MTLDevice::system_default().unwrap();
- /// # let shape = [10usize, 6usize];
- /// # let num_elems = shape[0] * shape[1];
- /// # let data = vec![0.0f32; num_elems];
- /// # let buffer = device.new_buffer_with_data(
- /// # bytemuck::cast_slice(&data),
- /// # MTLResourceOptions::STORAGE_MODE_SHARED,
- /// # ).unwrap();
- /// # let mut array = unsafe { MetalArray::new(buffer, &shape, DataType::F32) };
- ///
- /// // Original: [10, 6] tensor (10 rows × 6 columns)
- /// let splits = unsafe { array.row_split_no_copy(&[2, 5, 3]) };
- /// // Results: [2,6], [5,6], [3,6] - each split has full 6-column width
- /// assert_eq!(splits.len(), 3);
- /// ```
- pub unsafe fn row_split_no_copy(
- &mut self,
- row_lengths: &[usize],
- ) -> Box<[MetalArray]> {
- use std::ptr::NonNull;
-
- assert!(!self.shape.is_empty(), "Cannot split scalar or empty tensor");
- assert_eq!(
- row_lengths.iter().sum::<usize>(),
- self.shape[0],
- "Sum of row lengths must equal number of rows"
- );
-
- let elem_size = self.data_type.size_in_bytes();
- let row_size_bytes =
- self.shape.iter().skip(1).product::<usize>() * elem_size;
- let device = self.buffer.device();
-
- let mut results = Vec::with_capacity(row_lengths.len());
- let mut current_row_offset = 0usize;
-
- for &num_rows in row_lengths {
- let offset_bytes = current_row_offset * row_size_bytes;
- let required_len = (num_rows * row_size_bytes).max(48);
-
- // Create no-copy child buffer
- let base_ptr = self.buffer.contents().as_ptr() as *mut u8;
- let slice_ptr =
- unsafe { base_ptr.add(offset_bytes) } as *mut std::ffi::c_void;
-
- let child_buffer = unsafe {
- device.new_buffer_with_bytes_no_copy(
- NonNull::new_unchecked(slice_ptr),
- required_len,
- MTLResourceOptions::STORAGE_MODE_SHARED,
- )
- }.expect("Failed to create child buffer");
-
- // Create split shape
- let mut split_shape = self.shape.to_vec();
- split_shape[0] = num_rows;
-
- // Create MetalArray instead of TensorData
- let split_array = unsafe {
- MetalArray::new(child_buffer, &split_shape, self.data_type)
- };
-
- results.push(split_array);
- current_row_offset += num_rows;
- }
-
- results.into_boxed_slice()
- }
-}
-
-impl MetalArray {
- pub fn copy_from_array(
- &mut self,
- array: &MetalArray,
- ) {
- match self.data_type {
- DataType::BF16 => self.copy_from_array_with_type::<bf16>(array),
- DataType::F16 => self.copy_from_array_with_type::<f16>(array),
- DataType::F32 => self.copy_from_array_with_type::<f32>(array),
- DataType::F64 => self.copy_from_array_with_type::<f64>(array),
- DataType::I8 => self.copy_from_array_with_type::<i8>(array),
- DataType::U8 => self.copy_from_array_with_type::<u8>(array),
- DataType::I16 => self.copy_from_array_with_type::<i16>(array),
- DataType::U16 => self.copy_from_array_with_type::<u16>(array),
- DataType::I32 => self.copy_from_array_with_type::<i32>(array),
- DataType::U32 => self.copy_from_array_with_type::<u32>(array),
- DataType::I64 => self.copy_from_array_with_type::<i64>(array),
- DataType::U64 => self.copy_from_array_with_type::<u64>(array),
- _ => panic!("Unsupported data type"),
- }
- }
-
- fn copy_from_array_with_type<T: ArrayElement>(
- &mut self,
- array: &MetalArray,
- ) {
- assert_eq!(self.shape, array.shape);
- assert_eq!(self.data_type, array.data_type);
- self.as_slice_mut::<T>()
- .unwrap()
- .copy_from_slice(array.as_slice::<T>().unwrap());
- }
-
- pub fn copy_slice(
- &mut self,
- source: &Self,
- axis: usize,
- src_range: Range<usize>,
- dst_offset: usize,
- ) {
- assert_eq!(self.shape().len(), source.shape().len(), "Rank mismatch");
- assert!(axis < self.shape().len(), "Axis out of bounds");
- for (i, (a, b)) in self.shape().iter().zip(source.shape()).enumerate() {
- if i != axis {
- assert_eq!(a, b, "Shapes must match on all non-sliced axes");
- }
- }
- assert_eq!(
- self.data_type(),
- source.data_type(),
- "Arrays must have the same data type"
- );
-
- let elem_size = self.data_type().size_in_bytes();
-
- // The number of contiguous elements to copy for each slice operation
- let block_size_elems =
- self.shape().iter().skip(axis + 1).product::<usize>();
- let block_size_bytes = block_size_elems * elem_size;
-
- // The total number of blocks to copy
- let num_blocks: usize = self.shape().iter().take(axis).product();
-
- // Strides between the start of each block
- let src_stride_bytes = source.shape()[axis] * block_size_bytes;
- let dst_stride_bytes = self.shape()[axis] * block_size_bytes;
-
- let rows_to_copy = src_range.end - src_range.start;
- assert!(dst_offset + rows_to_copy <= self.shape()[axis]);
- assert!(src_range.end <= source.shape()[axis]);
-
- let src_buf = source.buffer();
- let dst_buf = self.buffer_mut();
- let copy_bytes = rows_to_copy * block_size_bytes;
-
- for i in 0..num_blocks {
- let src_block_start = i * src_stride_bytes;
- let dst_block_start = i * dst_stride_bytes;
-
- let src_start =
- src_block_start + src_range.start * block_size_bytes;
- let dst_start = dst_block_start + dst_offset * block_size_bytes;
-
- let src_slice = &src_buf[src_start..src_start + copy_bytes];
- let dst_slice = &mut dst_buf[dst_start..dst_start + copy_bytes];
- dst_slice.copy_from_slice(src_slice);
- }
- }
-}
+pub type MetalArray = Array<Metal>;
diff --git a/crates/uzu/src/backends/metal/backend.rs b/crates/uzu/src/backends/metal/backend.rs
new file mode 100644
index 000000000..9d98fdc4a
--- /dev/null
+++ b/crates/uzu/src/backends/metal/backend.rs
@@ -0,0 +1,17 @@
+use metal::{MTLBuffer, MTLCommandBuffer, MTLComputeCommandEncoder};
+use objc2::{rc::Retained, runtime::ProtocolObject};
+
+use super::{MTLContext, MTLError, MetalKernels};
+use crate::backends::common::Backend;
+
+#[derive(Debug, Clone)]
+pub struct Metal;
+
+impl Backend for Metal {
+ type NativeBuffer = Retained<ProtocolObject<dyn MTLBuffer>>;
+ type Context = MTLContext;
+ type CommandBuffer = Retained<ProtocolObject<dyn MTLCommandBuffer>>;
+ type EncoderRef = ProtocolObject<dyn MTLComputeCommandEncoder>;
+ type Kernels = MetalKernels;
+ type Error = MTLError;
+}
diff --git a/crates/uzu/src/backends/metal/buffer_allocator.rs b/crates/uzu/src/backends/metal/buffer_allocator.rs
deleted file mode 100644
index b958e3244..000000000
--- a/crates/uzu/src/backends/metal/buffer_allocator.rs
+++ /dev/null
@@ -1,38 +0,0 @@
-use super::{
- MTLBuffer, MTLDevice, MTLDeviceExt, MTLHeap, MTLResourceOptions, ProtocolObject, Retained,
-};
-
-/// Type alias for owned MTLBuffer
-pub type MTLBufferObj = Retained<ProtocolObject<dyn MTLBuffer>>;
-
-pub trait BufferAllocator {
- fn allocate_buffer(
- &self,
- size_in_bytes: usize,
- options: MTLResourceOptions,
- ) -> MTLBufferObj;
-}
-
-impl BufferAllocator for ProtocolObject<dyn MTLDevice> {
- #[inline]
- fn allocate_buffer(
- &self,
- size_in_bytes: usize,
- options: MTLResourceOptions,
- ) -> MTLBufferObj {
- self.new_buffer(size_in_bytes, options)
- .expect("Failed to allocate buffer")
- }
-}
-
-impl BufferAllocator for ProtocolObject<dyn MTLHeap> {
- #[inline]
- fn allocate_buffer(
- &self,
- size_in_bytes: usize,
- options: MTLResourceOptions,
- ) -> MTLBufferObj {
- self.new_buffer(size_in_bytes, options)
- .expect("Failed to allocate buffer from heap")
- }
-}
diff --git a/crates/uzu/src/backends/metal/classifier_context.rs b/crates/uzu/src/backends/metal/classifier_context.rs
index 038561ccd..593d70497 100644
--- a/crates/uzu/src/backends/metal/classifier_context.rs
+++ b/crates/uzu/src/backends/metal/classifier_context.rs
@@ -1,19 +1,26 @@
+use crate::backends::metal::Metal;
use std::{cell::RefCell, fs::File, io::BufReader, path::Path, rc::Rc};
use super::{
- KernelDataType, MTLCommandBuffer, MTLCommandQueue, MTLContext, MTLDevice, MTLDeviceExt,
- ModelShape, ProtocolObject, Retained,
+ KernelDataType, MTLCommandBuffer, MTLCommandQueue, MTLContext, ModelShape,
+ ProtocolObject, Retained,
compilation_parameters::CompilationConfig,
encodable_block::{
- Activation, ClassifierLayer, ClassifierPredictionHead, Normalization, Pooling, Rope,
+ Activation, ClassifierLayer, ClassifierPredictionHead, Normalization,
+ Pooling, Rope,
transformer_layer::{embed_block, linear_block},
},
- forward_pass::{ArrayId, EncodableBlock, RopeType, ScratchBuffers, SharedBuffers},
- kernel::{PoolingKernel, SigmoidKernel},
+ forward_pass::{
+ ArrayId, EncodableBlock, RopeType, ScratchBuffers, SharedBuffers,
+ },
+ kernel::dsl::SigmoidMetalKernel,
};
use crate::{
DataType,
- backends::metal::error::ClassifierError,
+ backends::{
+ common::Context, common::kernel::SigmoidKernel,
+ metal::error::ClassifierError,
+ },
config::{ClassifierModelConfig, ModelMetadata},
parameters::ParameterLoader,
session::types::Error,
@@ -24,35 +31,32 @@ pub struct ClassifierContext {
pub command_buffer: Retained<ProtocolObject<dyn MTLCommandBuffer>>,
pub shared_buffers: Rc<RefCell<SharedBuffers>>,
- pub scratch_buffers: ScratchBuffers<Rc<MTLContext>>,
+ pub scratch_buffers: ScratchBuffers<MTLContext>,
pub model_config: ClassifierModelConfig,
pub model_shape: ModelShape,
- pub embed: Box<dyn EncodableBlock>,
+ pub embed: Box<dyn EncodableBlock<Metal>>,
pub embedding_norm: Normalization,
pub layers: Box<[ClassifierLayer]>,
pub output_norm: Normalization,
- pub global_rope: Rc<Box<dyn EncodableBlock>>,
- pub local_rope: Option<Rc<Box<dyn EncodableBlock>>>,
+ pub global_rope: Rc<Box<dyn EncodableBlock<Metal>>>,
+ pub local_rope: Option<Rc<Box<dyn EncodableBlock<Metal>>>>,
- pub pooling: Box<dyn EncodableBlock>,
- pub prediction_head: Box<dyn EncodableBlock>,
+ pub pooling: Box<dyn EncodableBlock<Metal>>,
+ pub prediction_head: Box<dyn EncodableBlock<Metal>>,
- pub sigmoid_kernel: SigmoidKernel,
+ pub sigmoid_kernel: SigmoidMetalKernel,
}
impl ClassifierContext {
pub fn new(model_path: &Path) -> Result<Self, Error> {
- let mtl_device = <dyn MTLDevice>::system_default()
- .ok_or(Error::UnableToCreateMetalContext)?;
- let mtl_command_queue = mtl_device
- .new_command_queue_with_max_command_buffer_count(1024)
- .ok_or(Error::UnableToCreateMetalContext)?;
+ let context =
+ MTLContext::new().map_err(|_| Error::UnableToCreateMetalContext)?;
- let command_buffer = mtl_command_queue
- .command_buffer()
- .ok_or(Error::UnableToCreateMetalContext)?;
+ let command_buffer = context
+ .create_command_buffer()
+ .map_err(|_| Error::UnableToCreateMetalContext)?;
let config_path = model_path.join("config.json");
if !config_path.exists() {
@@ -77,11 +81,6 @@ impl ClassifierContext {
);
let model_shape = ModelShape::from_decoder_config(&decoder_config);
- let mtl_context = Rc::new(
- MTLContext::new(mtl_device, mtl_command_queue)
- .map_err(|_| Error::UnableToCreateMetalContext)?,
- );
-
let compilation_config = Rc::new(CompilationConfig::default());
let weights_path = model_path.join("model.safetensors");
if !weights_path.exists() {
@@ -89,12 +88,12 @@ impl ClassifierContext {
}
let weights_file = File::open(&weights_path)
.map_err(|_| Error::UnableToLoadWeights)?;
- let loader = ParameterLoader::new(&weights_file, &mtl_context)
+ let loader = ParameterLoader::new(&weights_file, context.as_ref())
.map_err(|_| Error::UnableToLoadWeights)?;
let root_loader_view = loader.tree();
let shared_buffers = Rc::new(RefCell::new(SharedBuffers::new(
- &mtl_context,
+ &context,
&decoder_config,
&model_shape,
)));
@@ -108,14 +107,10 @@ impl ClassifierContext {
{
let mut shared_bufs = shared_buffers.borrow_mut();
if let Some(global_rope) = &mut shared_bufs.global_rope {
- global_rope.update_data(
- &transformer_tree,
- String::from("global_rope"),
- );
+ global_rope.update_data(&transformer_tree, "global_rope");
}
if let Some(local_rope) = &mut shared_bufs.local_rope {
- local_rope
- .update_data(&transformer_tree, String::from("local_rope"));
+ local_rope.update_data(&transformer_tree, "local_rope");
}
}
@@ -128,11 +123,10 @@ impl ClassifierContext {
.activation_precision()
.into();
- let embed =
- embed_block(&decoder_config, &mtl_context, &root_loader_view);
+ let embed = embed_block(&decoder_config, &context, &root_loader_view);
let global_rope =
- Self::create_rope_block(&mtl_context, data_type, RopeType::Global)
+ Self::create_rope_block(&context, data_type, RopeType::Global)
.map_err(Error::Classifier)?;
let local_rope = classifier_model_config
.model_config
@@ -140,11 +134,7 @@ impl ClassifierContext {
.local_rope_config
.as_ref()
.map(|_| {
- Self::create_rope_block(
- &mtl_context,
- data_type,
- RopeType::Local,
- )
+ Self::create_rope_block(&context, data_type, RopeType::Local)
})
.transpose()
.map_err(Error::Classifier)?;
@@ -196,7 +186,7 @@ impl ClassifierContext {
})?;
Ok(ClassifierLayer::new(
- mtl_context.clone(),
+ context.clone(),
layer_config,
compilation_config.clone(),
layer_index,
@@ -224,7 +214,7 @@ impl ClassifierContext {
))
})?;
let output_norm = Normalization::new(
- &mtl_context,
+ &context,
data_type,
classifier_model_config
.model_config
@@ -245,7 +235,7 @@ impl ClassifierContext {
let context_length =
classifier_model_config.model_config.context_length;
let scratch_buffers = ScratchBuffers::new(
- &mtl_context,
+ context.as_ref(),
&decoder_config,
&model_shape,
context_length,
@@ -259,7 +249,7 @@ impl ClassifierContext {
))
})?;
let embedding_norm = Normalization::new(
- &mtl_context,
+ &context,
data_type,
classifier_model_config.model_config.embedding_norm_config.clone(),
ArrayId::Main,
@@ -298,7 +288,7 @@ impl ClassifierContext {
prediction_head_config.use_dense_bias,
model_dim,
[model_dim],
- &mtl_context,
+ &context,
&prediction_head_dense_tree,
ArrayId::ClassifierPooling,
ArrayId::ClassifierPredictionHeadDense,
@@ -306,7 +296,7 @@ impl ClassifierContext {
let prediction_head_activation = Box::new(
Activation::new(
- &mtl_context,
+ &context,
prediction_head_data_type,
prediction_head_config.activation.clone(),
ArrayId::ClassifierPredictionHeadDense,
@@ -326,7 +316,7 @@ impl ClassifierContext {
))
})?;
let prediction_head_norm = Normalization::new(
- &mtl_context,
+ &context,
prediction_head_data_type,
prediction_head_config.normalization_config.clone(),
ArrayId::ClassifierPredictionHeadDense,
@@ -351,7 +341,7 @@ impl ClassifierContext {
true,
model_dim,
[num_labels],
- &mtl_context,
+ &context,
&prediction_head_readout_tree,
ArrayId::ClassifierPredictionHeadNorm,
ArrayId::ClassifierPredictionHeadLogits,
@@ -363,22 +353,26 @@ impl ClassifierContext {
)))
})?;
- let pooling_kernel = PoolingKernel::new(&mtl_context, data_type)
- .map_err(|e| {
- eprintln!("Failed to create pooling kernel: {:?}", e);
- Error::UnableToCreateMetalContext
- })?;
- let sigmoid_kernel = SigmoidKernel::new(&mtl_context, data_type)
+ let sigmoid_kernel =
+ SigmoidMetalKernel::new(&context, data_type.into()).map_err(
+ |e| {
+ eprintln!("Failed to create sigmoid kernel: {:?}", e);
+ Error::UnableToCreateMetalContext
+ },
+ )?;
+
+ let pooling = Box::new(
+ Pooling::new(
+ &context,
+ data_type.into(),
+ classifier_model_config.model_config.classifier_pooling.clone(),
+ model_dim,
+ )
.map_err(|e| {
- eprintln!("Failed to create sigmoid kernel: {:?}", e);
+ eprintln!("Failed to create pooling: {:?}", e);
Error::UnableToCreateMetalContext
- })?;
-
- let pooling = Box::new(Pooling::new(
- pooling_kernel,
- classifier_model_config.model_config.classifier_pooling.clone(),
- model_dim,
- ));
+ })?,
+ );
let prediction_head = Box::new(ClassifierPredictionHead::new(
prediction_head_dense.map_err(|e| {
@@ -393,7 +387,7 @@ impl ClassifierContext {
));
Ok(Self {
- mtl_context,
+ mtl_context: context,
command_buffer,
shared_buffers,
scratch_buffers,
@@ -415,10 +409,10 @@ impl ClassifierContext {
mtl_context: &MTLContext,
data_type: DataType,
rope_type: RopeType,
- ) -> Result<Rc<Box<dyn EncodableBlock>>, ClassifierError> {
+ ) -> Result<Rc<Box<dyn EncodableBlock<Metal>>>, ClassifierError> {
let kernel_data_type: KernelDataType = data_type.into();
- let rotation: Box<dyn EncodableBlock> = Box::new(
+ let rotation: Box<dyn EncodableBlock<Metal>> = Box::new(
Rope::new(mtl_context, kernel_data_type, rope_type).map_err(
|e| {
ClassifierError::KernelCreationFailed(format!(
diff --git a/crates/uzu/src/backends/metal/command_buffer.rs b/crates/uzu/src/backends/metal/command_buffer.rs
new file mode 100644
index 000000000..b7cfd391a
--- /dev/null
+++ b/crates/uzu/src/backends/metal/command_buffer.rs
@@ -0,0 +1,27 @@
+use metal::{MTLCommandBuffer, MTLCommandEncoder};
+use objc2::{rc::Retained, runtime::ProtocolObject};
+
+use crate::backends::common::CommandBuffer;
+
+use super::Metal;
+
+impl CommandBuffer for Retained<ProtocolObject<dyn MTLCommandBuffer>> {
+ type Backend = Metal;
+
+ fn with_encoder<T>(
+ &self,
+ callback: impl FnOnce(
+ &<Self::Backend as crate::backends::common::Backend>::EncoderRef,
+ ) -> T,
+ ) -> T {
+ let encoder = self
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+
+ let ret = callback(&encoder);
+
+ encoder.end_encoding();
+
+ ret
+ }
+}
diff --git a/crates/uzu/src/backends/metal/context.rs b/crates/uzu/src/backends/metal/context.rs
index 5b4a5e977..be16f6d3b 100644
--- a/crates/uzu/src/backends/metal/context.rs
+++ b/crates/uzu/src/backends/metal/context.rs
@@ -1,20 +1,16 @@
-const MTLB: &[u8] =
- include_bytes!(concat!(env!("OUT_DIR"), "/default.metallib"));
-
use std::{cell::RefCell, collections::HashMap, env, rc::Rc};
+use metal::{MTLBuffer, MTLCommandBuffer};
use objc2::{rc::Retained, runtime::ProtocolObject};
use super::{
- MetalArray, error::MTLError, metal_extensions::LibraryPipelineExtensions,
+ Metal, error::MTLError, kernel, metal_extensions::LibraryPipelineExtensions,
};
-use crate::{
- DataType, DeviceContext,
- array::array_size_in_bytes,
- backends::metal::{
- MTLCommandQueue, MTLComputePipelineState, MTLDevice, MTLResourceExt,
- MTLDeviceExt, MTLFunctionConstantValues, MTLLibrary,
- MTLResourceOptions,
+use crate::backends::{
+ common::{Allocator, Context},
+ metal::{
+ MTLCommandQueue, MTLComputePipelineState, MTLDevice, MTLDeviceExt,
+ MTLFunctionConstantValues, MTLLibrary, MTLResourceOptions,
},
};
@@ -140,34 +136,10 @@ pub struct MTLContext {
pipeline_cache: RefCell<
HashMap<String, Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
>,
+ allocator: Allocator<Metal>,
}
impl MTLContext {
- pub fn new(
- device: Retained<ProtocolObject<dyn MTLDevice>>,
- command_queue: Retained<ProtocolObject<dyn MTLCommandQueue>>,
- ) -> Result<Self, MTLError> {
- let library = match device.new_library_with_data(MTLB) {
- Ok(lib) => lib,
- Err(e) => {
- return Err(MTLError::Generic(format!(
- "Failed to create Metal library: {}",
- e
- )));
- },
- };
-
- let architecture = DeviceArchitecture::from_device(&device);
-
- Ok(Self {
- device,
- command_queue,
- architecture,
- library,
- pipeline_cache: RefCell::new(HashMap::new()),
- })
- }
-
/// Returns true if NAX kernels are available on this device.
pub fn is_nax_available(&self) -> bool {
cfg!(feature = "metal-nax") && self.architecture.is_nax_available()
@@ -230,43 +202,59 @@ impl MTLContext {
Ok(pipeline)
}
-
}
-impl DeviceContext for MTLContext {
- type DeviceArray = MetalArray;
+impl Context for MTLContext {
+ type Backend = Metal;
- unsafe fn array_uninitialized(
- &self,
- shape: &[usize],
- data_type: DataType,
- label: String,
- ) -> MetalArray {
- unsafe {
- let buffer_size_bytes = array_size_in_bytes(shape, data_type);
-
- let buffer = self
- .device
- .new_buffer(
- buffer_size_bytes,
- MTLResourceOptions::STORAGE_MODE_SHARED,
- )
- .expect("Failed to create buffer");
- buffer.set_label(Some(&label));
- MetalArray::new(buffer, shape, data_type)
- }
+ fn new() -> Result<Rc<Self>, MTLError> {
+ let device: Retained<ProtocolObject<dyn MTLDevice>> =
+ <dyn metal::MTLDevice>::system_default().ok_or(
+ MTLError::Generic("cannot open system default device".into()),
+ )?;
+
+ let command_queue = device
+ .new_command_queue_with_max_command_buffer_count(1024)
+ .ok_or(MTLError::Generic("cannot create command queue".into()))?;
+
+ let library =
+ device.new_library_with_data(kernel::MTLB).map_err(|nserror| {
+ MTLError::Generic(format!(
+ "Failed to create Metal library: {}",
+ nserror
+ ))
+ })?;
+
+ let architecture = DeviceArchitecture::from_device(&device);
+
+ Ok(Rc::new_cyclic(|weak_self| Self {
+ device,
+ command_queue,
+ architecture,
+ library,
+ pipeline_cache: RefCell::new(HashMap::new()),
+ allocator: Allocator::new(weak_self.clone()),
+ }))
}
-}
-impl DeviceContext for Rc<MTLContext> {
- type DeviceArray = MetalArray;
+ fn allocator(&self) -> &Allocator<Metal> {
+ &self.allocator
+ }
- unsafe fn array_uninitialized(
+ fn create_buffer(
&self,
- shape: &[usize],
- data_type: DataType,
- label: String,
- ) -> MetalArray {
- unsafe { (**self).array_uninitialized(shape, data_type, label) }
+ size: usize,
+ ) -> Result<Retained<ProtocolObject<dyn MTLBuffer>>, MTLError> {
+ self.device
+ .new_buffer(size, MTLResourceOptions::STORAGE_MODE_SHARED)
+ .ok_or(MTLError::Generic("cannot create buffer".into()))
+ }
+
+ fn create_command_buffer(
+ &self
+ ) -> Result<Retained<ProtocolObject<dyn MTLCommandBuffer>>, MTLError> {
+ self.command_queue
+ .command_buffer()
+ .ok_or(MTLError::Generic("cannot create command buffer".into()))
}
}
diff --git a/crates/uzu/src/backends/metal/encodable_block/activation/mod.rs b/crates/uzu/src/backends/metal/encodable_block/activation/mod.rs
index a8c00495e..78bd80f67 100644
--- a/crates/uzu/src/backends/metal/encodable_block/activation/mod.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/activation/mod.rs
@@ -1,19 +1,21 @@
-use crate::backends::metal::{ProtocolObject,
- MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
+use crate::backends::metal::{
+ MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder, Metal,
+ ProtocolObject, Retained,
};
use super::{
super::{
MTLContext, MTLError,
forward_pass::{ArrayId, ForwardPassState},
- kernel::activation::ActivationKernel,
+ kernel::dsl::ActivationMetalKernel,
},
EncodableBlock, EncodingParameters,
};
-use crate::{Array, DataType, config::Activation as ActivationConfig};
+use crate::backends::common::kernel::ActivationKernel;
+use crate::{DataType, config::Activation as ActivationConfig};
pub struct Activation {
- kernel: ActivationKernel,
+ kernel: ActivationMetalKernel,
config: ActivationConfig,
input_array_id: ArrayId,
output_array_id: ArrayId,
@@ -27,7 +29,7 @@ impl Activation {
input_array_id: ArrayId,
output_array_id: ArrayId,
) -> Result<Self, MTLError> {
- let kernel = ActivationKernel::new(context, data_type)?;
+ let kernel = ActivationMetalKernel::new(context, data_type.into())?;
Ok(Self {
kernel,
config,
@@ -37,28 +39,18 @@ impl Activation {
}
}
-impl EncodableBlock for Activation {
+impl EncodableBlock<Metal> for Activation {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
_parameters: &EncodingParameters,
) {
- let arrays = state.arrays(&[self.input_array_id, self.output_array_id]);
- let mut input_array = arrays[0].borrow_mut();
- let mut output_array = arrays[1].borrow_mut();
-
- let n = input_array.shape().iter().product();
-
- let input_buffer = unsafe { input_array.mtl_buffer() };
- let output_buffer = unsafe { output_array.mtl_buffer() };
-
- let encoder = command_buffer.new_compute_command_encoder()
+ let encoder = command_buffer
+ .new_compute_command_encoder()
.expect("Failed to create compute command encoder");
- self.kernel
- .encode(&encoder, &self.config, input_buffer, output_buffer, n)
- .expect("Failed to encode activation kernel");
+ self.encode_with_shared_encoder(state, &encoder, _parameters);
encoder.end_encoding();
}
@@ -74,16 +66,30 @@ impl EncodableBlock for Activation {
_parameters: &EncodingParameters,
) {
let arrays = state.arrays(&[self.input_array_id, self.output_array_id]);
- let mut input_array = arrays[0].borrow_mut();
- let mut output_array = arrays[1].borrow_mut();
+ let input_array = arrays[0].borrow_mut();
+ let output_array = arrays[1].borrow_mut();
+
+ let n = input_array.shape().iter().product::<usize>();
- let n = input_array.shape().iter().product();
+ let input_buffer = input_array.buffer();
+ let output_buffer = output_array.buffer();
- let input_buffer = unsafe { input_array.mtl_buffer() };
- let output_buffer = unsafe { output_array.mtl_buffer() };
+ let act_type = match self.config {
+ ActivationConfig::SiLU {
+ ..
+ } => 0,
+ ActivationConfig::Gelu => 1,
+ ActivationConfig::Identity => {
+ panic!("Identity activation is not supported for kernel")
+ },
+ };
- self.kernel
- .encode(encoder, &self.config, input_buffer, output_buffer, n)
- .expect("Failed to encode activation kernel");
+ self.kernel.encode(
+ input_buffer,
+ output_buffer,
+ n as u32,
+ act_type,
+ encoder,
+ );
}
}
diff --git a/crates/uzu/src/backends/metal/encodable_block/attention.rs b/crates/uzu/src/backends/metal/encodable_block/attention.rs
index 9a5a5b39e..8ecff4b22 100644
--- a/crates/uzu/src/backends/metal/encodable_block/attention.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/attention.rs
@@ -1,9 +1,9 @@
//! Attention kernel encodable.
-use super::{EncodableBlock, EncodingParameters};
+use super::{EncodableBlock, EncodingParameters, Metal};
use crate::backends::metal::{
KernelDataType, MTLCommandBuffer, MTLCommandEncoder,
- MTLComputeCommandEncoder, MTLContext, ProtocolObject,
+ MTLComputeCommandEncoder, MTLContext, ProtocolObject, Retained,
forward_pass::{ArrayId, ForwardPassState, HashMapId},
kernel::attention::{
AttentionError, AttentionGemmArguments, AttentionKernel,
@@ -59,11 +59,11 @@ impl Attention {
}
}
-impl EncodableBlock for Attention {
+impl EncodableBlock<Metal> for Attention {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
let compute_encoder = command_buffer
@@ -95,25 +95,24 @@ impl EncodableBlock for Attention {
num_groups,
max_sequence_length,
) = {
- use crate::Array;
let qkv_binding = state.arrays(&[ArrayId::QKV]);
let qkv_array = qkv_binding[0].borrow();
- let suffix_length = Array::shape(&*qkv_array)[0];
+ let suffix_length = qkv_array.shape()[0];
let queries_binding = state.arrays(&[ArrayId::RotatedQueries]);
let queries_array = queries_binding[0].borrow();
- let num_heads = Array::shape(&*queries_array)[0];
- let head_dim = Array::shape(&*queries_array)[2];
+ let num_heads = queries_array.shape()[0];
+ let head_dim = queries_array.shape()[2];
let keys_binding = state.arrays(&[ArrayId::RotatedKeys]);
let keys_array = keys_binding[0].borrow();
- let num_groups = Array::shape(&*keys_array)[0];
+ let num_groups = keys_array.shape()[0];
let max_sequence_length = if let Some(_kv) = state.cache_layers() {
let key_cache_binding =
state.arrays(&[ArrayId::Keys(self.layer_index)]);
let key_cache_array = key_cache_binding[0].borrow();
- Array::shape(&*key_cache_array)[1]
+ key_cache_array.shape()[1]
} else {
// For classifiers without KV cache, max_sequence_length is just suffix_length
suffix_length
@@ -193,11 +192,11 @@ impl EncodableBlock for Attention {
None
};
- let mut rotated_keys_array = rotated_keys_binding[0].borrow_mut();
- let rotated_keys_buffer = unsafe { rotated_keys_array.mtl_buffer() };
+ let rotated_keys_array = rotated_keys_binding[0].borrow_mut();
+ let rotated_keys_buffer = rotated_keys_array.buffer();
- let mut qkv_array = qkv_binding[0].borrow_mut();
- let qkv_buffer = unsafe { qkv_array.mtl_buffer() };
+ let qkv_array = qkv_binding[0].borrow_mut();
+ let qkv_buffer = qkv_array.buffer();
// Get KV cache buffers only if KV cache exists (LLM mode)
let has_kv_cache = state.cache_layers().is_some();
@@ -208,10 +207,10 @@ impl EncodableBlock for Attention {
state.arrays(&[ArrayId::Values(self.layer_index)]);
let key_cache_array = key_cache_binding[0].borrow_mut();
- let key_cache_buf = key_cache_array.mtl_buffer_cloned();
+ let key_cache_buf = key_cache_array.buffer().clone();
let value_cache_array = value_cache_binding[0].borrow_mut();
- let value_cache_buf = value_cache_array.mtl_buffer_cloned();
+ let value_cache_buf = value_cache_array.buffer().clone();
(key_cache_buf, value_cache_buf)
} else {
@@ -221,8 +220,7 @@ impl EncodableBlock for Attention {
state.arrays(&[ArrayId::ExtractedValues]);
let extracted_values_array =
extracted_values_binding[0].borrow_mut();
- let extracted_values_buf =
- extracted_values_array.mtl_buffer_cloned();
+ let extracted_values_buf = extracted_values_array.buffer().clone();
// Reuse the KV cache update kernel to write values into extracted_values_buf.
if let Err(e) = self.kernel.encode_kv_cache_update(
@@ -243,24 +241,14 @@ impl EncodableBlock for Attention {
eprintln!("Failed to prepare rotated values buffer: {:?}", e);
}
- (
- unsafe {
- objc2::rc::Retained::retain(
- rotated_keys_buffer as *const _ as *mut _,
- )
- .unwrap()
- },
- extracted_values_buf,
- )
+ (rotated_keys_buffer.clone(), extracted_values_buf)
};
- let mut queries_array = rotated_queries_binding[0].borrow_mut();
- let queries_buffer = unsafe { queries_array.mtl_buffer() };
+ let queries_array = rotated_queries_binding[0].borrow_mut();
+ let queries_buffer = queries_array.buffer();
- let mut attention_output_array =
- attention_output_binding[0].borrow_mut();
- let attention_output_buffer =
- unsafe { attention_output_array.mtl_buffer() };
+ let attention_output_array = attention_output_binding[0].borrow_mut();
+ let attention_output_buffer = attention_output_array.buffer();
let attention_bias_buffer = if use_mask {
let attention_bias_map = attention_bias_binding[0].clone();
@@ -269,7 +257,7 @@ impl EncodableBlock for Attention {
.get(&window_length)
.map(|array| {
let array_ref = array.borrow();
- array_ref.mtl_buffer_cloned()
+ array_ref.buffer().clone()
})
.unwrap_or_else(|| {
panic!(
@@ -286,18 +274,18 @@ impl EncodableBlock for Attention {
let sums_binding = state.arrays(&[ArrayId::AttentionSums]);
let maxs_binding = state.arrays(&[ArrayId::AttentionMaxs]);
- let mut partials_array = partials_binding[0].borrow_mut();
- let partials_buffer = unsafe { partials_array.mtl_buffer() };
+ let partials_array = partials_binding[0].borrow_mut();
+ let partials_buffer = partials_array.buffer();
- let mut sums_array = sums_binding[0].borrow_mut();
- let sums_buffer = unsafe { sums_array.mtl_buffer() };
+ let sums_array = sums_binding[0].borrow_mut();
+ let sums_buffer = sums_array.buffer();
- let mut maxs_array = maxs_binding[0].borrow_mut();
- let maxs_buffer = unsafe { maxs_array.mtl_buffer() };
+ let maxs_array = maxs_binding[0].borrow_mut();
+ let maxs_buffer = maxs_array.buffer();
let sinks_buffer = sinks_binding
.as_ref()
- .map(|binding| binding[0].borrow().mtl_buffer_cloned());
+ .map(|binding| binding[0].borrow().buffer().clone());
// Only update KV cache for LLM mode (not for classifiers)
if has_kv_cache {
diff --git a/crates/uzu/src/backends/metal/encodable_block/classifier_layer.rs b/crates/uzu/src/backends/metal/encodable_block/classifier_layer.rs
index 51f0084c7..b697968c4 100644
--- a/crates/uzu/src/backends/metal/encodable_block/classifier_layer.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/classifier_layer.rs
@@ -1,7 +1,10 @@
+use crate::backends::metal::Metal;
use std::rc::Rc;
-use crate::backends::metal::{ProtocolObject,
- MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
+#[cfg(not(feature = "tracing"))]
+use crate::backends::metal::MTLCommandEncoder;
+use crate::backends::metal::{
+ MTLCommandBuffer, MTLComputeCommandEncoder, ProtocolObject, Retained,
};
use objc2::rc::autoreleasepool;
@@ -22,20 +25,20 @@ use crate::{
pub struct ClassifierLayer {
#[cfg_attr(not(feature = "tracing"), allow(dead_code))]
layer_index: usize,
- copy_main_to_shortcut_mixer: Box<dyn EncodableBlock>,
- pre_attention_norm: Option<Box<dyn EncodableBlock>>,
- qkv_projection: Box<dyn EncodableBlock>,
- qk_norm: Option<Box<dyn EncodableBlock>>,
- rope: Rc<Box<dyn EncodableBlock>>,
- attention: Box<dyn EncodableBlock>,
- out_projection: Box<dyn EncodableBlock>,
- post_attention_norm: Option<Box<dyn EncodableBlock>>,
- mixer_residual_add: Box<dyn EncodableBlock>,
- copy_main_to_shortcut_mlp: Box<dyn EncodableBlock>,
- pre_mlp_norm: Box<dyn EncodableBlock>,
- mlp: Box<dyn EncodableBlock>,
- post_mlp_norm: Option<Box<dyn EncodableBlock>>,
- mlp_residual_add: Box<dyn EncodableBlock>,
+ copy_main_to_shortcut_mixer: Box<dyn EncodableBlock<Metal>>,
+ pre_attention_norm: Option<Box<dyn EncodableBlock<Metal>>>,
+ qkv_projection: Box<dyn EncodableBlock<Metal>>,
+ qk_norm: Option<Box<dyn EncodableBlock<Metal>>>,
+ rope: Rc<Box<dyn EncodableBlock<Metal>>>,
+ attention: Box<dyn EncodableBlock<Metal>>,
+ out_projection: Box<dyn EncodableBlock<Metal>>,
+ post_attention_norm: Option<Box<dyn EncodableBlock<Metal>>>,
+ mixer_residual_add: Box<dyn EncodableBlock<Metal>>,
+ copy_main_to_shortcut_mlp: Box<dyn EncodableBlock<Metal>>,
+ pre_mlp_norm: Box<dyn EncodableBlock<Metal>>,
+ mlp: Box<dyn EncodableBlock<Metal>>,
+ post_mlp_norm: Option<Box<dyn EncodableBlock<Metal>>>,
+ mlp_residual_add: Box<dyn EncodableBlock<Metal>>,
}
impl ClassifierLayer {
@@ -50,8 +53,8 @@ impl ClassifierLayer {
head_dim: usize,
num_groups: usize,
attention_scale: Option<f32>,
- layer_loader: &ParameterTree<Rc<MTLContext>>,
- rope: Rc<Box<dyn EncodableBlock>>,
+ layer_loader: &ParameterTree<MTLContext>,
+ rope: Rc<Box<dyn EncodableBlock<Metal>>>,
) -> Self {
autoreleasepool(|_| {
let ctx = &*mtl_context; // Reference for functions expecting &MTLContext
@@ -66,16 +69,18 @@ impl ClassifierLayer {
let kernel_data_type: KernelDataType =
intermediate_data_type.into();
- let copy_main_to_shortcut_mixer: Box<dyn EncodableBlock> = Box::new(
- TensorCopy::new(
- ctx,
- kernel_data_type,
- vec![ArrayId::Main, ArrayId::Shortcut].into_boxed_slice(),
- )
- .unwrap(),
- );
+ let copy_main_to_shortcut_mixer: Box<dyn EncodableBlock<Metal>> =
+ Box::new(
+ TensorCopy::new(
+ ctx,
+ kernel_data_type,
+ vec![ArrayId::Main, ArrayId::Shortcut]
+ .into_boxed_slice(),
+ )
+ .unwrap(),
+ );
- let pre_attention_norm: Option<Box<dyn EncodableBlock>> =
+ let pre_attention_norm: Option<Box<dyn EncodableBlock<Metal>>> =
if let Some(norm_config) =
&layer_config.pre_attention_norm_config
{
@@ -118,31 +123,30 @@ impl ClassifierLayer {
)
.expect("Failed to create qkv projection");
- let qk_norm: Option<Box<dyn EncodableBlock>> = if attention_config
- .query_norm_config
- .is_some()
- || attention_config.key_norm_config.is_some()
- {
- match QKNorm::new(
- ctx,
- intermediate_data_type,
- attention_config.query_norm_config.clone(),
- attention_config.key_norm_config.clone(),
- ArrayId::QKV,
- &layer_loader.subtree("mixer").unwrap(),
- num_heads,
- num_groups,
- head_dim,
- ) {
- Ok(norm) => Some(Box::new(norm)),
- Err(e) => panic!(
- "Failed to create QK norm kernel for layer {}: {:?}",
- layer_index, e
- ),
- }
- } else {
- None
- };
+ let qk_norm: Option<Box<dyn EncodableBlock<Metal>>> =
+ if attention_config.query_norm_config.is_some()
+ || attention_config.key_norm_config.is_some()
+ {
+ match QKNorm::new(
+ ctx,
+ intermediate_data_type,
+ attention_config.query_norm_config.clone(),
+ attention_config.key_norm_config.clone(),
+ ArrayId::QKV,
+ &layer_loader.subtree("mixer").unwrap(),
+ num_heads,
+ num_groups,
+ head_dim,
+ ) {
+ Ok(norm) => Some(Box::new(norm)),
+ Err(e) => panic!(
+ "Failed to create QK norm kernel for layer {}: {:?}",
+ layer_index, e
+ ),
+ }
+ } else {
+ None
+ };
let out_projection = transformer_layer::linear_block(
&attention_config.out_projection_config,
@@ -156,7 +160,7 @@ impl ClassifierLayer {
)
.expect("Failed to create out projection");
- let post_attention_norm: Option<Box<dyn EncodableBlock>> =
+ let post_attention_norm: Option<Box<dyn EncodableBlock<Metal>>> =
if let Some(norm_config) =
&layer_config.post_attention_norm_config
{
@@ -175,7 +179,7 @@ impl ClassifierLayer {
None
};
- let mixer_residual_add: Box<dyn EncodableBlock> = Box::new(
+ let mixer_residual_add: Box<dyn EncodableBlock<Metal>> = Box::new(
TensorAddSwap::new(
ctx,
kernel_data_type,
@@ -184,16 +188,18 @@ impl ClassifierLayer {
.unwrap(),
);
- let copy_main_to_shortcut_mlp: Box<dyn EncodableBlock> = Box::new(
- TensorCopy::new(
- ctx,
- kernel_data_type,
- vec![ArrayId::Main, ArrayId::Shortcut].into_boxed_slice(),
- )
- .unwrap(),
- );
+ let copy_main_to_shortcut_mlp: Box<dyn EncodableBlock<Metal>> =
+ Box::new(
+ TensorCopy::new(
+ ctx,
+ kernel_data_type,
+ vec![ArrayId::Main, ArrayId::Shortcut]
+ .into_boxed_slice(),
+ )
+ .unwrap(),
+ );
- let pre_mlp_norm: Box<dyn EncodableBlock> = Box::new(
+ let pre_mlp_norm: Box<dyn EncodableBlock<Metal>> = Box::new(
Normalization::new(
ctx,
intermediate_data_type,
@@ -214,7 +220,7 @@ impl ClassifierLayer {
)
.expect("Failed to create mlp block");
- let post_mlp_norm: Option<Box<dyn EncodableBlock>> =
+ let post_mlp_norm: Option<Box<dyn EncodableBlock<Metal>>> =
if let Some(norm_config) = &layer_config.post_mlp_norm_config {
Some(Box::new(
Normalization::new(
@@ -231,7 +237,7 @@ impl ClassifierLayer {
None
};
- let attention: Box<dyn EncodableBlock> = Box::new(
+ let attention: Box<dyn EncodableBlock<Metal>> = Box::new(
Attention::new(
ctx,
kernel_data_type,
@@ -244,7 +250,7 @@ impl ClassifierLayer {
.expect("Failed to create attention kernel"),
);
- let mlp_residual_add: Box<dyn EncodableBlock> = Box::new(
+ let mlp_residual_add: Box<dyn EncodableBlock<Metal>> = Box::new(
TensorAddSwap::new(
ctx,
kernel_data_type,
@@ -274,18 +280,19 @@ impl ClassifierLayer {
}
}
-impl EncodableBlock for ClassifierLayer {
+impl EncodableBlock<Metal> for ClassifierLayer {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
#[cfg(not(feature = "tracing"))]
{
if self.supports_shared_encoder() {
- let encoder = command_buffer.new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
self.encode_with_shared_encoder(state, &encoder, parameters);
encoder.end_encoding();
return;
diff --git a/crates/uzu/src/backends/metal/encodable_block/decoder.rs b/crates/uzu/src/backends/metal/encodable_block/decoder.rs
index feebf5796..a24b15f66 100644
--- a/crates/uzu/src/backends/metal/encodable_block/decoder.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/decoder.rs
@@ -1,5 +1,6 @@
//! Decoder executables - combines embedding, layers, normalization, and readout.
+use crate::backends::metal::Metal;
use std::rc::Rc;
use super::{
@@ -8,8 +9,8 @@ use super::{
use crate::{
DataType, DecoderConfig,
backends::metal::{
- KernelDataType, MTLCommandBuffer, MTLContext, ModelShape,
- ProtocolObject,
+ KernelDataType, MTLCommandBuffer, MTLComputeCommandEncoder, MTLContext,
+ ModelShape, ProtocolObject, Retained,
compilation_parameters::CompilationConfig,
encodable_block::transformer_layer::{embed_block, readout_block},
forward_pass::{ArrayId, ForwardPassState, RopeType},
@@ -20,19 +21,19 @@ use crate::{
/// Full decoder executable with all layers and components.
pub struct Decoder {
- pub embed: Box<dyn EncodableBlock>,
+ pub embed: Box<dyn EncodableBlock<Metal>>,
pub layers: Box<[LayerExecutables]>,
- pub norm: Box<dyn EncodableBlock>,
- pub readout: Box<dyn EncodableBlock>,
- pub global_rope: Option<Rc<Box<dyn EncodableBlock>>>,
- pub local_rope: Option<Rc<Box<dyn EncodableBlock>>>,
+ pub norm: Box<dyn EncodableBlock<Metal>>,
+ pub readout: Box<dyn EncodableBlock<Metal>>,
+ pub global_rope: Option<Rc<Box<dyn EncodableBlock<Metal>>>>,
+ pub local_rope: Option<Rc<Box<dyn EncodableBlock<Metal>>>>,
}
impl Decoder {
pub fn new(
mtl_context: Rc<MTLContext>,
decoder_config: Rc<DecoderConfig>,
- root_weight_loader: &ParameterTree<Rc<MTLContext>>,
+ root_weight_loader: &ParameterTree<MTLContext>,
compilation_config: Rc<CompilationConfig>,
) -> Self {
let decoder_weight_loader = root_weight_loader
@@ -145,7 +146,7 @@ impl Decoder {
})
.collect::<Vec<_>>();
- let norm_block: Box<dyn EncodableBlock> = Box::new(
+ let norm_block: Box<dyn EncodableBlock<Metal>> = Box::new(
RMSNorm::new(
&mtl_context,
norm_data_type,
@@ -172,8 +173,8 @@ impl Decoder {
mtl_context: &MTLContext,
kernel_data_type: KernelDataType,
rope_type: RopeType,
- ) -> Rc<Box<dyn EncodableBlock>> {
- let rotation: Box<dyn EncodableBlock> = Box::new(
+ ) -> Rc<Box<dyn EncodableBlock<Metal>>> {
+ let rotation: Box<dyn EncodableBlock<Metal>> = Box::new(
Rope::new(mtl_context, kernel_data_type, rope_type)
.expect("Failed to create Rope"),
);
@@ -197,11 +198,20 @@ impl Decoder {
}
}
-impl EncodableBlock for Decoder {
+impl EncodableBlock<Metal> for Decoder {
+ fn encode_with_shared_encoder(
+ &self,
+ _state: &mut ForwardPassState,
+ _encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
+ _parameters: &EncodingParameters,
+ ) {
+ unimplemented!("Decoder does not support shared encoder")
+ }
+
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
self.embed.encode(state, command_buffer, parameters);
diff --git a/crates/uzu/src/backends/metal/encodable_block/embedding/full_precision_lookup.rs b/crates/uzu/src/backends/metal/encodable_block/embedding/full_precision_lookup.rs
index 0558aedf8..6728697e8 100644
--- a/crates/uzu/src/backends/metal/encodable_block/embedding/full_precision_lookup.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/embedding/full_precision_lookup.rs
@@ -1,29 +1,24 @@
-use std::rc::Rc;
-
-use crate::backends::metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- ProtocolObject, Retained,
-};
-
use super::{
- super::{EncodableBlock, EncodingParameters},
+ super::{EncodableBlock, EncodingParameters, Metal},
EmbeddingError,
};
use crate::{
- Array, DataType,
- backends::metal::{
- MTLContext, MTLError,
- forward_pass::{ArrayId, ForwardPassState},
- kernel::embedding::{
- FullPrecisionEmbeddingLookupArguments,
- FullPrecisionEmbeddingLookupKernel,
+ DataType,
+ backends::{
+ common::kernel::FullPrecisionEmbeddingLookupKernel,
+ metal::{
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
+ MTLComputeCommandEncoder, MTLContext, MTLError, ProtocolObject,
+ Retained,
+ forward_pass::{ArrayId, ForwardPassState},
+ kernel::dsl::FullPrecisionEmbeddingLookupMetalKernel,
},
},
parameters::ParameterTree,
};
pub struct FullPrecisionEmbeddingLookup {
- kernel: FullPrecisionEmbeddingLookupKernel,
+ kernel: FullPrecisionEmbeddingLookupMetalKernel,
weights_buffer: Retained<ProtocolObject<dyn MTLBuffer>>,
vocab_size: u32,
model_dim: u32,
@@ -37,12 +32,14 @@ impl FullPrecisionEmbeddingLookup {
vocab_size: usize,
model_dim: usize,
input_scale: Option<f32>,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
) -> Result<Self, EmbeddingError> {
- let kernel =
- FullPrecisionEmbeddingLookupKernel::new(mtl_context, data_type)?;
+ let kernel = FullPrecisionEmbeddingLookupMetalKernel::new(
+ mtl_context,
+ data_type.into(),
+ )?;
- let mut weights = match parameter_tree.leaf("weights") {
+ let weights = match parameter_tree.leaf("weights") {
Ok(weights) => weights,
Err(_) => parameter_tree.leaf("input_weights").map_err(|e| {
EmbeddingError::MetalError(MTLError::Generic(format!(
@@ -74,7 +71,7 @@ impl FullPrecisionEmbeddingLookup {
)));
}
- let weights_buffer = unsafe { weights.mtl_buffer().to_owned() };
+ let weights_buffer = weights.buffer().to_owned();
Ok(Self {
kernel,
@@ -86,14 +83,15 @@ impl FullPrecisionEmbeddingLookup {
}
}
-impl EncodableBlock for FullPrecisionEmbeddingLookup {
+impl EncodableBlock<Metal> for FullPrecisionEmbeddingLookup {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
- let encoder = command_buffer.new_compute_command_encoder()
+ let encoder = command_buffer
+ .new_compute_command_encoder()
.expect("Failed to create compute command encoder");
self.encode_with_shared_encoder(state, &encoder, parameters);
encoder.end_encoding();
@@ -116,24 +114,21 @@ impl EncodableBlock for FullPrecisionEmbeddingLookup {
) {
let arrays = state.arrays(&[ArrayId::TokenIds, ArrayId::Main]);
let batch_size = state.active_suffix_length();
- let mut token_ids_array_mut = arrays[0].borrow_mut();
- let mut output_array_mut = arrays[1].borrow_mut();
+ let token_ids_array_mut = arrays[0].borrow_mut();
+ let output_array_mut = arrays[1].borrow_mut();
- let token_ids_buffer = unsafe { token_ids_array_mut.mtl_buffer() };
- let output_buffer = unsafe { output_array_mut.mtl_buffer() };
+ let token_ids_buffer = token_ids_array_mut.buffer();
+ let output_buffer = output_array_mut.buffer();
- let args = FullPrecisionEmbeddingLookupArguments {
+ self.kernel.encode(
token_ids_buffer,
- weights_buffer: &self.weights_buffer,
+ &self.weights_buffer,
output_buffer,
- batch_size: batch_size as u32,
- vocab_size: self.vocab_size,
- model_dim: self.model_dim,
- input_scale: self.input_scale,
- };
-
- self.kernel
- .encode(encoder, args)
- .expect("Failed to encode full precision embedding lookup kernel");
+ batch_size as u32,
+ self.vocab_size,
+ self.model_dim,
+ self.input_scale,
+ encoder,
+ )
}
}
diff --git a/crates/uzu/src/backends/metal/encodable_block/embedding/full_precision_readout.rs b/crates/uzu/src/backends/metal/encodable_block/embedding/full_precision_readout.rs
index b22e9f79c..4fe565361 100644
--- a/crates/uzu/src/backends/metal/encodable_block/embedding/full_precision_readout.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/embedding/full_precision_readout.rs
@@ -1,4 +1,4 @@
-use std::{cell::RefCell, rc::Rc};
+use std::cell::RefCell;
use crate::backends::metal::{
MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
@@ -6,11 +6,11 @@ use crate::backends::metal::{
};
use super::{
- super::{EncodableBlock, EncodingParameters},
+ super::{EncodableBlock, EncodingParameters, Metal},
EmbeddingError,
};
use crate::{
- Array, DataType,
+ DataType,
backends::metal::{
MTLContext, MTLError,
forward_pass::{ArrayId, ForwardPassState},
@@ -32,14 +32,14 @@ impl FullPrecisionEmbeddingReadout {
data_type: DataType,
vocab_size: usize,
model_dim: usize,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
) -> Result<Self, EmbeddingError> {
if !matches!(data_type, DataType::F16 | DataType::BF16 | DataType::F32)
{
return Err(EmbeddingError::UnsupportedDataType(data_type));
}
- let mut weights = match parameter_tree.leaf("weights") {
+ let weights = match parameter_tree.leaf("weights") {
Ok(weights) => weights,
Err(_) => parameter_tree.leaf("output_weights").map_err(|e| {
EmbeddingError::MetalError(MTLError::Generic(format!(
@@ -70,7 +70,7 @@ impl FullPrecisionEmbeddingReadout {
)));
}
- let weights_buffer = unsafe { weights.mtl_buffer().to_owned() };
+ let weights_buffer = weights.buffer().to_owned();
let mut kernel =
MatmulKernel::new(data_type).map_err(EmbeddingError::MetalError)?;
@@ -85,14 +85,15 @@ impl FullPrecisionEmbeddingReadout {
}
}
-impl EncodableBlock for FullPrecisionEmbeddingReadout {
+impl EncodableBlock<Metal> for FullPrecisionEmbeddingReadout {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
- let encoder = command_buffer.new_compute_command_encoder()
+ let encoder = command_buffer
+ .new_compute_command_encoder()
.expect("Failed to create compute command encoder");
self.encode_with_shared_encoder(state, &encoder, parameters);
encoder.end_encoding();
@@ -119,12 +120,12 @@ impl EncodableBlock for FullPrecisionEmbeddingReadout {
return;
}
let sampling_start = state.sampling_start();
- let mut input_array_mut = arrays[0].borrow_mut();
- let mut output_array_mut = arrays[1].borrow_mut();
+ let input_array_mut = arrays[0].borrow_mut();
+ let output_array_mut = arrays[1].borrow_mut();
let elem_size = input_array_mut.data_type().size_in_bytes();
- let input_buffer = unsafe { input_array_mut.mtl_buffer() };
- let output_buffer = unsafe { output_array_mut.mtl_buffer() };
+ let input_buffer = input_array_mut.buffer();
+ let output_buffer = output_array_mut.buffer();
let a_offset = (sampling_start * self.model_dim * elem_size) as u64;
let args = MatmulArguments {
diff --git a/crates/uzu/src/backends/metal/encodable_block/embedding/quantized_lookup.rs b/crates/uzu/src/backends/metal/encodable_block/embedding/quantized_lookup.rs
index 13fa1d897..1fb215cb0 100644
--- a/crates/uzu/src/backends/metal/encodable_block/embedding/quantized_lookup.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/embedding/quantized_lookup.rs
@@ -1,18 +1,17 @@
-use std::rc::Rc;
-
-
use super::{
- super::{EncodableBlock, EncodingParameters},
+ super::{EncodableBlock, EncodingParameters, Metal},
EmbeddingError,
};
use crate::{
- Array, DataType,
- backends::metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- MTLContext, MTLDeviceExt, MTLError, MTLResourceOptions, ProtocolObject, Retained,
- forward_pass::{ArrayId, ForwardPassState},
- kernel::embedding::{
- QuantizedEmbeddingLookupArguments, QuantizedEmbeddingLookupKernel,
+ DataType,
+ backends::{
+ common::{Context, kernel::QuantizedEmbeddingLookupKernel},
+ metal::{
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
+ MTLComputeCommandEncoder, MTLContext, MTLError, ProtocolObject,
+ Retained,
+ forward_pass::{ArrayId, ForwardPassState},
+ kernel::dsl::QuantizedEmbeddingLookupMetalKernel,
},
},
config::QuantizationMode,
@@ -20,10 +19,11 @@ use crate::{
};
pub struct QuantizedEmbeddingLookup {
- kernel: QuantizedEmbeddingLookupKernel,
+ kernel: QuantizedEmbeddingLookupMetalKernel,
weights_buffer: Retained<ProtocolObject<dyn MTLBuffer>>,
scales_buffer: Retained<ProtocolObject<dyn MTLBuffer>>,
biases_buffer: Retained<ProtocolObject<dyn MTLBuffer>>,
+ mode: QuantizationMode,
input_scale: f32,
vocab_size: u32,
model_dim: u32,
@@ -39,7 +39,7 @@ impl QuantizedEmbeddingLookup {
group_size: usize,
mode: QuantizationMode,
input_scale: f32,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
) -> Result<Self, EmbeddingError> {
Self::new_with_names(
mtl_context,
@@ -64,7 +64,7 @@ impl QuantizedEmbeddingLookup {
group_size: usize,
mode: QuantizationMode,
input_scale: f32,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
) -> Result<Self, EmbeddingError> {
Self::new_with_names(
mtl_context,
@@ -92,15 +92,17 @@ impl QuantizedEmbeddingLookup {
weights_name: &str,
scales_name: &str,
biases_name: &str,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
) -> Result<Self, EmbeddingError> {
let packing_divisor = mode.packing_divisor();
- let kernel =
- QuantizedEmbeddingLookupKernel::new(mtl_context, data_type, mode)?;
+ let kernel = QuantizedEmbeddingLookupMetalKernel::new(
+ mtl_context,
+ data_type.into(),
+ )?;
// Load weights [vocab_size, model_dim/packing_divisor] as storage_type
- let mut weights = parameter_tree.leaf(weights_name).map_err(|e| {
+ let weights = parameter_tree.leaf(weights_name).map_err(|e| {
EmbeddingError::MetalError(MTLError::Generic(format!(
"Failed to load weights: {:?}",
e
@@ -118,7 +120,7 @@ impl QuantizedEmbeddingLookup {
}
// Load scales [vocab_size, num_groups]
- let mut scales = parameter_tree.leaf(scales_name).map_err(|e| {
+ let scales = parameter_tree.leaf(scales_name).map_err(|e| {
EmbeddingError::MetalError(MTLError::Generic(format!(
"Failed to load scales: {:?}",
e
@@ -154,60 +156,60 @@ impl QuantizedEmbeddingLookup {
}
// Load or create biases buffer [vocab_size, num_groups] (MLX key: "biases")
- let biases_buffer: Retained<ProtocolObject<dyn MTLBuffer>> = match parameter_tree.leaf(biases_name) {
- Ok(mut deq_biases) => {
- if deq_biases.shape() != [vocab_size, num_groups] {
- return Err(EmbeddingError::MetalError(MTLError::Generic(
- format!(
- "Embedding lookup deq_biases shape mismatch: got {:?}, expected [{}, {}]",
- deq_biases.shape(),
- vocab_size,
- num_groups
- ),
- )));
- }
- if deq_biases.data_type() != data_type {
- return Err(EmbeddingError::UnsupportedDataType(
- deq_biases.data_type(),
- ));
- }
- unsafe { deq_biases.mtl_buffer().to_owned().into() }
- },
- Err(_) => {
- let elem_size: usize = match data_type {
- DataType::F16 | DataType::BF16 => 2,
- DataType::F32 => 4,
- other => {
- return Err(EmbeddingError::UnsupportedDataType(other));
- },
- };
- let size_bytes = (vocab_size * num_groups * elem_size) as u64;
- let buf = mtl_context
- .device
- .new_buffer(
- size_bytes.try_into().unwrap(),
- MTLResourceOptions::STORAGE_MODE_SHARED,
- )
- .expect("Failed to allocate buffer");
- unsafe {
- std::ptr::write_bytes(
- metal::MTLBuffer::contents(&*buf).as_ptr(),
- 0,
- size_bytes as usize,
- );
- }
- buf
- },
- };
+ let biases_buffer: Retained<ProtocolObject<dyn MTLBuffer>> =
+ match parameter_tree.leaf(biases_name) {
+ Ok(deq_biases) => {
+ if deq_biases.shape() != [vocab_size, num_groups] {
+ return Err(EmbeddingError::MetalError(
+ MTLError::Generic(format!(
+ "Embedding lookup deq_biases shape mismatch: got {:?}, expected [{}, {}]",
+ deq_biases.shape(),
+ vocab_size,
+ num_groups
+ )),
+ ));
+ }
+ if deq_biases.data_type() != data_type {
+ return Err(EmbeddingError::UnsupportedDataType(
+ deq_biases.data_type(),
+ ));
+ }
+ deq_biases.buffer().to_owned().into()
+ },
+ Err(_) => {
+ let elem_size: usize = match data_type {
+ DataType::F16 | DataType::BF16 => 2,
+ DataType::F32 => 4,
+ other => {
+ return Err(EmbeddingError::UnsupportedDataType(
+ other,
+ ));
+ },
+ };
+ let size_bytes = vocab_size * num_groups * elem_size;
+ let buf = mtl_context
+ .create_buffer(size_bytes)
+ .expect("Failed to allocate buffer");
+ unsafe {
+ std::ptr::write_bytes(
+ metal::MTLBuffer::contents(&*buf).as_ptr(),
+ 0,
+ size_bytes as usize,
+ );
+ }
+ buf
+ },
+ };
- let weights_buffer = unsafe { weights.mtl_buffer().to_owned().into() };
- let scales_buffer = unsafe { scales.mtl_buffer().to_owned().into() };
+ let weights_buffer = weights.buffer().to_owned().into();
+ let scales_buffer = scales.buffer().to_owned().into();
Ok(Self {
kernel,
weights_buffer,
scales_buffer,
biases_buffer,
+ mode,
input_scale,
vocab_size: vocab_size as u32,
model_dim: model_dim as u32,
@@ -216,11 +218,11 @@ impl QuantizedEmbeddingLookup {
}
}
-impl EncodableBlock for QuantizedEmbeddingLookup {
+impl EncodableBlock<Metal> for QuantizedEmbeddingLookup {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
let encoder = command_buffer
@@ -247,27 +249,27 @@ impl EncodableBlock for QuantizedEmbeddingLookup {
) {
let arrays = state.arrays(&[ArrayId::TokenIds, ArrayId::Main]);
let batch_size = state.active_suffix_length();
- let mut token_ids_array_mut = arrays[0].borrow_mut();
- let mut output_array_mut = arrays[1].borrow_mut();
-
- let token_ids_buffer = unsafe { token_ids_array_mut.mtl_buffer() };
- let output_buffer = unsafe { output_array_mut.mtl_buffer() };
+ let token_ids_array_mut = arrays[0].borrow_mut();
+ let output_array_mut = arrays[1].borrow_mut();
- let args = QuantizedEmbeddingLookupArguments {
- token_ids_buffer,
- weights_buffer: &self.weights_buffer,
- scales_buffer: &self.scales_buffer,
- biases_buffer: &self.biases_buffer,
- output_buffer,
- batch_size: batch_size as u32,
- vocab_size: self.vocab_size,
- model_dim: self.model_dim,
- group_size: self.group_size,
- input_scale: self.input_scale,
+ let quant_mode = match self.mode {
+ QuantizationMode::UInt4 => 0,
+ QuantizationMode::Int8 => 1,
+ QuantizationMode::UInt8 => 2,
};
-
- self.kernel
- .encode(encoder, args)
- .expect("Failed to encode quantized embedding lookup kernel");
+ self.kernel.encode(
+ token_ids_array_mut.buffer(),
+ &self.weights_buffer,
+ &self.scales_buffer,
+ &self.biases_buffer,
+ output_array_mut.buffer(),
+ batch_size as u32,
+ self.vocab_size,
+ self.model_dim,
+ self.group_size,
+ self.input_scale,
+ quant_mode,
+ encoder,
+ )
}
}
diff --git a/crates/uzu/src/backends/metal/encodable_block/embedding/quantized_readout.rs b/crates/uzu/src/backends/metal/encodable_block/embedding/quantized_readout.rs
index 37aa9bfda..576018b03 100644
--- a/crates/uzu/src/backends/metal/encodable_block/embedding/quantized_readout.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/embedding/quantized_readout.rs
@@ -1,18 +1,20 @@
-use std::rc::Rc;
-
-
use super::{
- super::{EncodableBlock, EncodingParameters},
+ super::{EncodableBlock, EncodingParameters, Metal},
EmbeddingError,
};
use crate::{
- Array, DataType,
- backends::metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- MTLContext, MTLDeviceExt, MTLError, MTLResourceOptions, ProtocolObject, Retained,
- forward_pass::{ArrayId, ForwardPassState},
- kernel::quant_matmul::{
- QuantizationType, QuantizedMatmulArguments, QuantizedMatmulKernel,
+ DataType,
+ backends::{
+ common::Context,
+ metal::{
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
+ MTLComputeCommandEncoder, MTLContext, MTLError, ProtocolObject,
+ Retained,
+ forward_pass::{ArrayId, ForwardPassState},
+ kernel::quant_matmul::{
+ QuantizationType, QuantizedMatmulArguments,
+ QuantizedMatmulKernel,
+ },
},
},
config::QuantizationMode,
@@ -36,7 +38,7 @@ impl QuantizedEmbeddingReadout {
model_dim: usize,
group_size: usize,
mode: QuantizationMode,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
) -> Result<Self, EmbeddingError> {
Self::new_with_names(
mtl_context,
@@ -59,7 +61,7 @@ impl QuantizedEmbeddingReadout {
model_dim: usize,
group_size: usize,
mode: QuantizationMode,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
) -> Result<Self, EmbeddingError> {
Self::new_with_names(
mtl_context,
@@ -85,10 +87,10 @@ impl QuantizedEmbeddingReadout {
weights_name: &str,
scales_name: &str,
biases_name: &str,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
) -> Result<Self, EmbeddingError> {
// Load weights [vocab_size, model_dim/2] as U8
- let mut weights = parameter_tree.leaf(weights_name).map_err(|e| {
+ let weights = parameter_tree.leaf(weights_name).map_err(|e| {
EmbeddingError::MetalError(MTLError::Generic(format!(
"Failed to load weights: {:?}",
e
@@ -96,7 +98,7 @@ impl QuantizedEmbeddingReadout {
})?;
// Load scales [vocab_size, num_groups]
- let mut scales = parameter_tree.leaf(scales_name).map_err(|e| {
+ let scales = parameter_tree.leaf(scales_name).map_err(|e| {
EmbeddingError::MetalError(MTLError::Generic(format!(
"Failed to load scales: {:?}",
e
@@ -137,55 +139,54 @@ impl QuantizedEmbeddingReadout {
}
// MLX requires per-group biases; if missing, create a zero buffer of shape [vocab_size, num_groups]
- let biases_buffer: Retained<ProtocolObject<dyn MTLBuffer>> = match parameter_tree.leaf(biases_name) {
- Ok(mut deq_biases) => {
- if deq_biases.shape() != [vocab_size, num_groups] {
- return Err(EmbeddingError::MetalError(MTLError::Generic(
- format!(
- "Embedding readout deq_biases shape mismatch: got {:?}, expected [{}, {}]",
- deq_biases.shape(),
- vocab_size,
- num_groups
- ),
- )));
- }
- if deq_biases.data_type() != data_type {
- return Err(EmbeddingError::UnsupportedDataType(
- deq_biases.data_type(),
- ));
- }
- unsafe { deq_biases.mtl_buffer().to_owned().into() }
- },
- Err(_) => {
- // Allocate zero-initialized biases buffer
- let elem_size: usize = match data_type {
- DataType::F16 | DataType::BF16 => 2,
- DataType::F32 => 4,
- other => {
- return Err(EmbeddingError::UnsupportedDataType(other));
- },
- };
- let size_bytes = (vocab_size * num_groups * elem_size) as u64;
- let buf = mtl_context
- .device
- .new_buffer(
- size_bytes.try_into().unwrap(),
- MTLResourceOptions::STORAGE_MODE_SHARED,
- )
- .expect("Failed to allocate buffer");
- unsafe {
- std::ptr::write_bytes(
- metal::MTLBuffer::contents(&*buf).as_ptr(),
- 0,
- size_bytes as usize,
- );
- }
- buf
- },
- };
+ let biases_buffer: Retained<ProtocolObject<dyn MTLBuffer>> =
+ match parameter_tree.leaf(biases_name) {
+ Ok(deq_biases) => {
+ if deq_biases.shape() != [vocab_size, num_groups] {
+ return Err(EmbeddingError::MetalError(
+ MTLError::Generic(format!(
+ "Embedding readout deq_biases shape mismatch: got {:?}, expected [{}, {}]",
+ deq_biases.shape(),
+ vocab_size,
+ num_groups
+ )),
+ ));
+ }
+ if deq_biases.data_type() != data_type {
+ return Err(EmbeddingError::UnsupportedDataType(
+ deq_biases.data_type(),
+ ));
+ }
+ deq_biases.buffer().to_owned().into()
+ },
+ Err(_) => {
+ // Allocate zero-initialized biases buffer
+ let elem_size: usize = match data_type {
+ DataType::F16 | DataType::BF16 => 2,
+ DataType::F32 => 4,
+ other => {
+ return Err(EmbeddingError::UnsupportedDataType(
+ other,
+ ));
+ },
+ };
+ let size_bytes = vocab_size * num_groups * elem_size;
+ let buf = mtl_context
+ .create_buffer(size_bytes)
+ .expect("Failed to allocate buffer");
+ unsafe {
+ std::ptr::write_bytes(
+ metal::MTLBuffer::contents(&*buf).as_ptr(),
+ 0,
+ size_bytes as usize,
+ );
+ }
+ buf
+ },
+ };
- let weights_buffer = unsafe { weights.mtl_buffer().to_owned().into() };
- let scales_buffer = unsafe { scales.mtl_buffer().to_owned().into() };
+ let weights_buffer = weights.buffer().to_owned().into();
+ let scales_buffer = scales.buffer().to_owned().into();
let kernel = QuantizedMatmulKernel::new(
mtl_context,
@@ -215,11 +216,11 @@ impl QuantizedEmbeddingReadout {
}
}
-impl EncodableBlock for QuantizedEmbeddingReadout {
+impl EncodableBlock<Metal> for QuantizedEmbeddingReadout {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
let encoder = command_buffer
@@ -250,12 +251,12 @@ impl EncodableBlock for QuantizedEmbeddingReadout {
return;
}
let sampling_start = state.sampling_start();
- let mut input_array_mut = arrays[0].borrow_mut();
- let mut output_array_mut = arrays[1].borrow_mut();
+ let input_array_mut = arrays[0].borrow_mut();
+ let output_array_mut = arrays[1].borrow_mut();
let elem_size = input_array_mut.data_type().size_in_bytes();
- let input_buffer = unsafe { input_array_mut.mtl_buffer() };
- let output_buffer = unsafe { output_array_mut.mtl_buffer() };
+ let input_buffer = input_array_mut.buffer();
+ let output_buffer = output_array_mut.buffer();
let a_offset = (sampling_start * self.model_dim * elem_size) as u64;
let args = QuantizedMatmulArguments {
diff --git a/crates/uzu/src/backends/metal/encodable_block/encoding_parameters.rs b/crates/uzu/src/backends/metal/encodable_block/encoding_parameters.rs
index f000841e9..1b28f7704 100644
--- a/crates/uzu/src/backends/metal/encodable_block/encoding_parameters.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/encoding_parameters.rs
@@ -1,48 +1,6 @@
//! Encoding parameters for forward pass operations.
-use crate::backends::metal::{MTLBuffer, ProtocolObject, Retained};
+use crate::backends::metal::Metal;
-#[derive(Clone)]
-pub struct EncodingParameters<'a> {
- pub warmup: bool,
- pub enable_commit: bool,
- pub wait_until_completed: bool,
- pub projection_step: Option<usize>,
- pub predicate: Option<&'a Retained<ProtocolObject<dyn MTLBuffer>>>,
-}
-
-impl<'a> EncodingParameters<'a> {
- pub fn new(
- warmup: bool,
- enable_commit: bool,
- wait_until_completed: bool,
- ) -> Self {
- Self {
- warmup,
- enable_commit,
- wait_until_completed,
- projection_step: None,
- predicate: None,
- }
- }
-
- pub fn with_projection(
- mut self,
- projection_step: usize,
- ) -> Self {
- self.projection_step = Some(projection_step);
- self
- }
-
- pub fn with_predicate(
- mut self,
- predicate: &'a Retained<ProtocolObject<dyn MTLBuffer>>,
- ) -> Self {
- self.predicate = Some(predicate);
- self
- }
-
- pub fn predicate_ref(&self) -> Option<&ProtocolObject<dyn MTLBuffer>> {
- self.predicate.map(|buffer| &**buffer)
- }
-}
+pub type EncodingParameters<'a> =
+ crate::encodable_block::EncodingParameters<'a, Metal>;
diff --git a/crates/uzu/src/backends/metal/encodable_block/layer/executables.rs b/crates/uzu/src/backends/metal/encodable_block/layer/executables.rs
index 2a1925437..7bb5a9e15 100644
--- a/crates/uzu/src/backends/metal/encodable_block/layer/executables.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/layer/executables.rs
@@ -1,9 +1,12 @@
//! Layer executables - a single decoder layer with mixer, norms, and MLP.
+use crate::backends::metal::Metal;
use std::rc::Rc;
-use crate::backends::metal::{ProtocolObject,
- MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
+#[cfg(not(feature = "tracing"))]
+use crate::backends::metal::MTLCommandEncoder;
+use crate::backends::metal::{
+ MTLCommandBuffer, MTLComputeCommandEncoder, ProtocolObject, Retained,
};
use objc2::rc::autoreleasepool;
@@ -29,14 +32,14 @@ use crate::{
/// A single decoder layer with all its components.
pub struct LayerExecutables {
pub layer_index: usize,
- pub copy_main_to_shortcut: Box<dyn EncodableBlock>,
- pub pre_attention_norm: Box<dyn EncodableBlock>,
+ pub copy_main_to_shortcut: Box<dyn EncodableBlock<Metal>>,
+ pub pre_attention_norm: Box<dyn EncodableBlock<Metal>>,
pub(crate) mixer: MixerExecutables,
- pub post_attention_norm: Option<Box<dyn EncodableBlock>>,
- pub main_shortcut_add_swap: Box<dyn EncodableBlock>,
- pub pre_mlp_norm: Box<dyn EncodableBlock>,
- pub mlp: Box<dyn EncodableBlock>,
- pub post_mlp_norm: Option<Box<dyn EncodableBlock>>,
+ pub post_attention_norm: Option<Box<dyn EncodableBlock<Metal>>>,
+ pub main_shortcut_add_swap: Box<dyn EncodableBlock<Metal>>,
+ pub pre_mlp_norm: Box<dyn EncodableBlock<Metal>>,
+ pub mlp: Box<dyn EncodableBlock<Metal>>,
+ pub post_mlp_norm: Option<Box<dyn EncodableBlock<Metal>>>,
}
impl LayerExecutables {
@@ -53,8 +56,8 @@ impl LayerExecutables {
head_dim: usize,
num_groups: usize,
attention_scale: Option<f32>,
- decoder_layer_loader: &ParameterTree<Rc<MTLContext>>,
- rope: Option<Rc<Box<dyn EncodableBlock>>>,
+ decoder_layer_loader: &ParameterTree<MTLContext>,
+ rope: Option<Rc<Box<dyn EncodableBlock<Metal>>>>,
) -> Self {
autoreleasepool(|_| {
let ctx = &*mtl_context; // Reference for functions expecting &MTLContext
@@ -75,16 +78,18 @@ impl LayerExecutables {
let kernel_data_type: KernelDataType =
intermediate_data_type.into();
- let copy_main_to_shortcut: Box<dyn EncodableBlock> = Box::new(
- TensorCopy::new(
- ctx,
- kernel_data_type,
- vec![ArrayId::Main, ArrayId::Shortcut].into_boxed_slice(),
- )
- .unwrap(),
- );
+ let copy_main_to_shortcut: Box<dyn EncodableBlock<Metal>> =
+ Box::new(
+ TensorCopy::new(
+ ctx,
+ kernel_data_type,
+ vec![ArrayId::Main, ArrayId::Shortcut]
+ .into_boxed_slice(),
+ )
+ .unwrap(),
+ );
- let pre_attention_norm: Box<dyn EncodableBlock> = Box::new(
+ let pre_attention_norm: Box<dyn EncodableBlock<Metal>> = Box::new(
RMSNorm::new(
ctx,
intermediate_data_type,
@@ -120,7 +125,7 @@ impl LayerExecutables {
)
.expect("Failed to create qkv projection");
- let qk_norm: Option<Box<dyn EncodableBlock>> =
+ let qk_norm: Option<Box<dyn EncodableBlock<Metal>>> =
if attention_config.query_norm_config.is_some()
|| attention_config.key_norm_config.is_some()
{
@@ -136,7 +141,7 @@ impl LayerExecutables {
head_dim,
) {
Ok(qk_norm) => Some(Box::new(qk_norm)
- as Box<dyn EncodableBlock>),
+ as Box<dyn EncodableBlock<Metal>>),
Err(e) => panic!(
"Failed to create QK norm kernel for layer {}: {:?}",
layer_index, e
@@ -182,7 +187,7 @@ impl LayerExecutables {
}
},
MixerConfig::Mamba(mamba_config) => {
- let mixer: Box<dyn EncodableBlock> =
+ let mixer: Box<dyn EncodableBlock<Metal>> =
Box::new(MambaMixer::new(
ctx,
layer_type.clone(),
@@ -200,7 +205,7 @@ impl LayerExecutables {
}
},
MixerConfig::ShortConv(short_conv_config) => {
- let mixer: Box<dyn EncodableBlock> =
+ let mixer: Box<dyn EncodableBlock<Metal>> =
Box::new(ShortConvMixer::new(
ctx,
layer_type.clone(),
@@ -216,7 +221,7 @@ impl LayerExecutables {
},
};
- let post_attention_norm: Option<Box<dyn EncodableBlock>> =
+ let post_attention_norm: Option<Box<dyn EncodableBlock<Metal>>> =
if let Some(norm_config) =
&layer_config.post_attention_norm_config
{
@@ -237,16 +242,18 @@ impl LayerExecutables {
None
};
- let main_shortcut_add_swap: Box<dyn EncodableBlock> = Box::new(
- TensorAddSwap::new(
- ctx,
- kernel_data_type,
- vec![ArrayId::Shortcut, ArrayId::Main].into_boxed_slice(),
- )
- .unwrap(),
- );
+ let main_shortcut_add_swap: Box<dyn EncodableBlock<Metal>> =
+ Box::new(
+ TensorAddSwap::new(
+ ctx,
+ kernel_data_type,
+ vec![ArrayId::Shortcut, ArrayId::Main]
+ .into_boxed_slice(),
+ )
+ .unwrap(),
+ );
- let pre_mlp_norm: Box<dyn EncodableBlock> = Box::new(
+ let pre_mlp_norm: Box<dyn EncodableBlock<Metal>> = Box::new(
RMSNorm::new(
ctx,
intermediate_data_type,
@@ -267,7 +274,7 @@ impl LayerExecutables {
)
.expect("Failed to create mlp block");
- let post_mlp_norm: Option<Box<dyn EncodableBlock>> =
+ let post_mlp_norm: Option<Box<dyn EncodableBlock<Metal>>> =
if let Some(norm_config) = &layer_config.post_mlp_norm_config {
Some(Box::new(
RMSNorm::new(
@@ -301,11 +308,11 @@ impl LayerExecutables {
}
}
-impl EncodableBlock for LayerExecutables {
+impl EncodableBlock<Metal> for LayerExecutables {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
// In non-tracing builds, if every sub-block supports shared encoding,
@@ -313,8 +320,9 @@ impl EncodableBlock for LayerExecutables {
#[cfg(not(feature = "tracing"))]
{
if self.supports_shared_encoder() {
- let encoder = command_buffer.new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
self.encode_with_shared_encoder(state, &encoder, parameters);
encoder.end_encoding();
return;
diff --git a/crates/uzu/src/backends/metal/encodable_block/layer/mixer.rs b/crates/uzu/src/backends/metal/encodable_block/layer/mixer.rs
index f4f0ef4c5..16b9eea8d 100644
--- a/crates/uzu/src/backends/metal/encodable_block/layer/mixer.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/layer/mixer.rs
@@ -3,20 +3,21 @@
use std::rc::Rc;
use super::super::EncodableBlock;
+use crate::backends::metal::Metal;
/// Mixer component - either attention, state space, or short conv.
pub(crate) enum MixerExecutables {
Attention {
- qkv_projection: Box<dyn EncodableBlock>,
- qk_norm: Option<Box<dyn EncodableBlock>>,
- rope: Rc<Box<dyn EncodableBlock>>,
- attention: Box<dyn EncodableBlock>,
- out_projection: Box<dyn EncodableBlock>,
+ qkv_projection: Box<dyn EncodableBlock<Metal>>,
+ qk_norm: Option<Box<dyn EncodableBlock<Metal>>>,
+ rope: Rc<Box<dyn EncodableBlock<Metal>>>,
+ attention: Box<dyn EncodableBlock<Metal>>,
+ out_projection: Box<dyn EncodableBlock<Metal>>,
},
StateSpace {
- mixer: Box<dyn EncodableBlock>,
+ mixer: Box<dyn EncodableBlock<Metal>>,
},
ShortConv {
- mixer: Box<dyn EncodableBlock>,
+ mixer: Box<dyn EncodableBlock<Metal>>,
},
}
diff --git a/crates/uzu/src/backends/metal/encodable_block/linear/full_precision.rs b/crates/uzu/src/backends/metal/encodable_block/linear/full_precision.rs
index f1811f9c0..32d7279e4 100644
--- a/crates/uzu/src/backends/metal/encodable_block/linear/full_precision.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/linear/full_precision.rs
@@ -1,15 +1,16 @@
-use std::{cell::RefCell, rc::Rc};
+use crate::backends::metal::Metal;
+use std::cell::RefCell;
use crate::{
DataType,
backends::metal::{
MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
- MTLComputeCommandEncoder, MTLContext, MTLError, ProtocolObject, Retained,
+ MTLComputeCommandEncoder, MTLContext, MTLError, ProtocolObject,
+ Retained,
encodable_block::{EncodableBlock, EncodingParameters},
forward_pass::{ArrayId, ForwardPassState},
kernel::matmul::{MatmulArguments, MatmulKernel},
},
- device::array::Array,
parameters::ParameterTree,
};
@@ -29,7 +30,7 @@ impl FullPrecisionLinear {
precision: DataType,
input_dim: usize,
output_dim: usize,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
input_array_id: ArrayId,
output_array_id: ArrayId,
) -> Result<Self, MTLError> {
@@ -41,7 +42,7 @@ impl FullPrecisionLinear {
)));
}
- let mut weights = parameter_tree.leaf("weights").map_err(|e| {
+ let weights = parameter_tree.leaf("weights").map_err(|e| {
MTLError::Generic(format!("Failed to load weights: {:?}", e))
})?;
@@ -62,10 +63,10 @@ impl FullPrecisionLinear {
}
let weights_buffer: Retained<ProtocolObject<dyn MTLBuffer>> =
- unsafe { weights.mtl_buffer() }.to_owned().into();
+ weights.buffer().to_owned().into();
let bias_buffer = match parameter_tree.leaf("biases") {
- Ok(mut biases) => {
+ Ok(biases) => {
if biases.shape() != [output_dim] {
return Err(MTLError::Generic(format!(
"Bias shape mismatch: got {:?}, expected [{:?}]",
@@ -81,7 +82,7 @@ impl FullPrecisionLinear {
)));
}
let bias_buffer: Retained<ProtocolObject<dyn MTLBuffer>> =
- unsafe { biases.mtl_buffer() }.to_owned().into();
+ biases.buffer().to_owned().into();
Some(bias_buffer)
},
Err(_) => None,
@@ -102,21 +103,21 @@ impl FullPrecisionLinear {
}
}
-impl EncodableBlock for FullPrecisionLinear {
+impl EncodableBlock<Metal> for FullPrecisionLinear {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
let arrays = state.arrays(&[self.input_array_id, self.output_array_id]);
let batch_size = state.active_suffix_length();
- let mut input_array_mut = arrays[0].borrow_mut();
- let mut output_array_mut = arrays[1].borrow_mut();
+ let input_array_mut = arrays[0].borrow_mut();
+ let output_array_mut = arrays[1].borrow_mut();
- let input_buffer = unsafe { input_array_mut.mtl_buffer() };
- let output_buffer = unsafe { output_array_mut.mtl_buffer() };
+ let input_buffer = input_array_mut.buffer();
+ let output_buffer = output_array_mut.buffer();
let encoder = command_buffer
.new_compute_command_encoder()
@@ -128,7 +129,7 @@ impl EncodableBlock for FullPrecisionLinear {
b: &self.weights_buffer,
c: None,
d: output_buffer,
- bias: self.bias_buffer.as_ref().map(|v| v.as_ref()),
+ bias: self.bias_buffer.as_ref(),
batch: batch_size as i32,
input_dim: self.input_dim as i32,
output_dim: self.output_dim as i32,
@@ -168,11 +169,11 @@ impl EncodableBlock for FullPrecisionLinear {
let arrays = state.arrays(&[self.input_array_id, self.output_array_id]);
let batch_size = state.active_suffix_length();
- let mut input_array_mut = arrays[0].borrow_mut();
- let mut output_array_mut = arrays[1].borrow_mut();
+ let input_array_mut = arrays[0].borrow_mut();
+ let output_array_mut = arrays[1].borrow_mut();
- let input_buffer = unsafe { input_array_mut.mtl_buffer() };
- let output_buffer = unsafe { output_array_mut.mtl_buffer() };
+ let input_buffer = input_array_mut.buffer();
+ let output_buffer = output_array_mut.buffer();
let args = MatmulArguments {
a: input_buffer,
@@ -180,7 +181,7 @@ impl EncodableBlock for FullPrecisionLinear {
b: &self.weights_buffer,
c: None,
d: output_buffer,
- bias: self.bias_buffer.as_ref().map(|v| v.as_ref()),
+ bias: self.bias_buffer.as_ref(),
batch: batch_size as i32,
input_dim: self.input_dim as i32,
output_dim: self.output_dim as i32,
diff --git a/crates/uzu/src/backends/metal/encodable_block/linear/quantized.rs b/crates/uzu/src/backends/metal/encodable_block/linear/quantized.rs
index d24df6386..77adc96ff 100644
--- a/crates/uzu/src/backends/metal/encodable_block/linear/quantized.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/linear/quantized.rs
@@ -1,8 +1,11 @@
-use std::rc::Rc;
+use crate::backends::metal::Metal;
-use crate::backends::metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- ProtocolObject, Retained,
+use crate::backends::{
+ common::kernel::TensorAddBiasKernel,
+ metal::{
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
+ MTLComputeCommandEncoder, ProtocolObject, Retained,
+ },
};
use crate::{
@@ -12,7 +15,7 @@ use crate::{
encodable_block::{EncodableBlock, EncodingParameters},
forward_pass::{ArrayId, ForwardPassState},
kernel::{
- KernelDataType, TensorAddBias,
+ dsl::TensorAddBiasMetalKernel,
quant_matmul::{
QuantizationType, QuantizedMatmulArguments,
QuantizedMatmulKernel,
@@ -20,13 +23,12 @@ use crate::{
},
},
config::QuantizationConfig,
- device::array::Array,
parameters::ParameterTree,
};
pub struct QuantizedLinear {
kernel: QuantizedMatmulKernel,
- bias_add_kernel: Option<TensorAddBias>,
+ bias_add_kernel: Option<TensorAddBiasMetalKernel>,
biases_buffer: Option<Retained<ProtocolObject<dyn MTLBuffer>>>,
weights_buffer: Retained<ProtocolObject<dyn MTLBuffer>>,
scales_buffer: Retained<ProtocolObject<dyn MTLBuffer>>,
@@ -44,7 +46,7 @@ impl QuantizedLinear {
config: &QuantizationConfig,
input_dim: usize,
output_dim: usize,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
input_array_id: ArrayId,
output_array_id: ArrayId,
) -> Result<Self, MTLError> {
@@ -60,7 +62,7 @@ impl QuantizedLinear {
)));
}
- let mut weights = parameter_tree.leaf("weights").map_err(|e| {
+ let weights = parameter_tree.leaf("weights").map_err(|e| {
MTLError::Generic(format!("Failed to load weights: {:?}", e))
})?;
@@ -75,7 +77,7 @@ impl QuantizedLinear {
)));
}
- let mut scales = parameter_tree.leaf("scales").map_err(|e| {
+ let scales = parameter_tree.leaf("scales").map_err(|e| {
MTLError::Generic(format!("Failed to load scales: {:?}", e))
})?;
let k_g = (input_dim + config.group_size - 1) / config.group_size;
@@ -96,7 +98,7 @@ impl QuantizedLinear {
)));
}
match parameter_tree.leaf("deq_biases") {
- Ok(mut deq_biases) => {
+ Ok(deq_biases) => {
let db_shape = deq_biases.shape();
if !(w_shape == [output_dim, input_dim / packing_divisor]
&& s_shape == [output_dim, k_g]
@@ -115,13 +117,13 @@ impl QuantizedLinear {
)));
}
let scales_buffer: Retained<ProtocolObject<dyn MTLBuffer>> =
- unsafe { objc2::rc::Retained::retain(scales.mtl_buffer() as *const _ as *mut _).unwrap() };
+ scales.buffer().clone();
let biases_buf: Retained<ProtocolObject<dyn MTLBuffer>> =
- unsafe { objc2::rc::Retained::retain(deq_biases.mtl_buffer() as *const _ as *mut _).unwrap() };
+ deq_biases.buffer().clone();
(QuantizationType::Mlx, biases_buf, scales_buffer)
},
Err(_) => {
- let mut zero_points =
+ let zero_points =
parameter_tree.leaf("zero_points").map_err(|e| {
MTLError::Generic(format!(
"Failed to load zero_points: {:?}",
@@ -153,21 +155,21 @@ impl QuantizedLinear {
)));
}
let scales_buffer: Retained<ProtocolObject<dyn MTLBuffer>> =
- unsafe { objc2::rc::Retained::retain(scales.mtl_buffer() as *const _ as *mut _).unwrap() };
+ scales.buffer().clone();
let zps_buf: Retained<ProtocolObject<dyn MTLBuffer>> =
- unsafe { objc2::rc::Retained::retain(std::ptr::from_ref(&*zero_points.mtl_buffer()) as *mut _).unwrap() };
+ zero_points.buffer().clone();
(QuantizationType::ZeroPoint, zps_buf, scales_buffer)
},
}
};
let weights_buffer: Retained<ProtocolObject<dyn MTLBuffer>> =
- unsafe { weights.mtl_buffer() }.to_owned().into();
+ weights.buffer().to_owned().into();
// Optional trainable bias support
let (bias_add_kernel, biases_buffer) =
match parameter_tree.leaf("biases") {
- Ok(mut biases) => {
+ Ok(biases) => {
if biases.shape() != [output_dim] {
return Err(MTLError::Generic(format!(
"Bias shape mismatch: got {:?}, expected [{:?}]",
@@ -182,12 +184,12 @@ impl QuantizedLinear {
kernel_data_type
)));
}
- let bias_add_kernel = Some(TensorAddBias::new(
+ let bias_add_kernel = Some(TensorAddBiasMetalKernel::new(
mtl_context,
- KernelDataType::from(kernel_data_type),
+ kernel_data_type,
)?);
let biases_buffer: Retained<ProtocolObject<dyn MTLBuffer>> =
- unsafe { biases.mtl_buffer() }.to_owned().into();
+ biases.buffer().to_owned().into();
(bias_add_kernel, Some(biases_buffer))
},
Err(_) => (None, None),
@@ -224,23 +226,24 @@ impl QuantizedLinear {
}
}
-impl EncodableBlock for QuantizedLinear {
+impl EncodableBlock<Metal> for QuantizedLinear {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
let arrays = state.arrays(&[self.input_array_id, self.output_array_id]);
let batch_size = state.active_suffix_length();
- let mut input_array_mut = arrays[0].borrow_mut();
- let mut output_array_mut = arrays[1].borrow_mut();
+ let input_array_mut = arrays[0].borrow_mut();
+ let output_array_mut = arrays[1].borrow_mut();
- let input_buffer = unsafe { input_array_mut.mtl_buffer() };
- let output_buffer = unsafe { output_array_mut.mtl_buffer() };
+ let input_buffer = input_array_mut.buffer();
+ let output_buffer = output_array_mut.buffer();
- let encoder = command_buffer.new_compute_command_encoder()
+ let encoder = command_buffer
+ .new_compute_command_encoder()
.expect("Failed to create compute command encoder");
let args = QuantizedMatmulArguments {
@@ -266,14 +269,14 @@ impl EncodableBlock for QuantizedLinear {
(&self.bias_add_kernel, &self.biases_buffer)
{
let total_len = batch_size * self.output_dim;
- bias_add.encode_into_command_buffer(
+ bias_add.encode_if(
output_buffer,
&bias_buf,
output_buffer,
- self.output_dim,
- total_len,
- command_buffer,
- parameters.predicate.map(|v| &**v),
+ self.output_dim as u32,
+ total_len as u32,
+ &encoder,
+ parameters.predicate,
);
}
@@ -296,11 +299,11 @@ impl EncodableBlock for QuantizedLinear {
let arrays = state.arrays(&[self.input_array_id, self.output_array_id]);
let batch_size = state.active_suffix_length();
- let mut input_array_mut = arrays[0].borrow_mut();
- let mut output_array_mut = arrays[1].borrow_mut();
+ let input_array_mut = arrays[0].borrow_mut();
+ let output_array_mut = arrays[1].borrow_mut();
- let input_buffer = unsafe { input_array_mut.mtl_buffer() };
- let output_buffer = unsafe { output_array_mut.mtl_buffer() };
+ let input_buffer = input_array_mut.buffer();
+ let output_buffer = output_array_mut.buffer();
let args = QuantizedMatmulArguments {
a_buffer: input_buffer,
@@ -323,14 +326,14 @@ impl EncodableBlock for QuantizedLinear {
(&self.bias_add_kernel, &self.biases_buffer)
{
let total_len = batch_size * self.output_dim;
- bias_add.encode_with_encoder(
+ bias_add.encode_if(
output_buffer,
&bias_buf,
output_buffer,
- self.output_dim,
- total_len,
+ self.output_dim as u32,
+ total_len as u32,
encoder,
- parameters.predicate.map(|v| &**v),
+ parameters.predicate,
);
}
}
diff --git a/crates/uzu/src/backends/metal/encodable_block/mamba_mixer.rs b/crates/uzu/src/backends/metal/encodable_block/mamba_mixer.rs
index 75d30df2d..99e47ffd9 100644
--- a/crates/uzu/src/backends/metal/encodable_block/mamba_mixer.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/mamba_mixer.rs
@@ -1,38 +1,40 @@
//! Mamba2 SSM mixer encodable.
-use std::{env, rc::Rc};
-
-use crate::backends::metal::{ProtocolObject,
- MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
+use super::{EncodableBlock, EncodingParameters, Metal, transformer_layer};
+use crate::backends::metal::kernel::ssm::ssd_prefill::{
+ SSDPrefillArguments, SSDPrefillKernels, SSDPrefillMode,
};
-
-use super::{EncodableBlock, EncodingParameters, transformer_layer};
use crate::{
DataType,
- backends::metal::{
- KernelDataType, MTLContext, MetalArray,
- compilation_parameters::CompilationConfig,
- forward_pass::{ArrayId, ForwardPassState},
- kernel::ssm::{
- Conv1dPackArguments, Conv1dScanArguments, Conv1dScanKernel,
- SSDPrefillArguments, SSDPrefillKernel, SSDPrefillMode,
- SSDUpdateArguments, SSDUpdateKernel, SplitInProjArguments,
- SplitInProjKernel, conv1d_scan::Conv1dDecodeArguments,
+ backends::{
+ common::kernel::{SSDUpdateKernel, SplitInProjKernel},
+ metal::{
+ KernelDataType, MTLCommandBuffer, MTLCommandEncoder,
+ MTLComputeCommandEncoder, MTLContext, MetalArray, ProtocolObject,
+ Retained,
+ compilation_parameters::CompilationConfig,
+ forward_pass::{ArrayId, ForwardPassState},
+ kernel::dsl::{SSDUpdateMetalKernel, SplitInProjMetalKernel},
+ kernel::ssm::{
+ Conv1dPackArguments, Conv1dScanArguments, Conv1dScanKernel,
+ conv1d_scan::Conv1dDecodeArguments,
+ },
},
},
config::{DecoderLayerType, Mamba2Config},
parameters::ParameterTree,
};
+use std::{env, rc::Rc};
pub(crate) struct MambaMixer {
layer_index: usize,
config: Mamba2Config,
- in_projection: Box<dyn EncodableBlock>,
- out_projection: Box<dyn EncodableBlock>,
- split_inproj: SplitInProjKernel,
+ in_projection: Box<dyn EncodableBlock<Metal>>,
+ out_projection: Box<dyn EncodableBlock<Metal>>,
+ split_inproj: SplitInProjMetalKernel,
conv_scan: Conv1dScanKernel,
- ssm_prefill: SSDPrefillKernel,
- ssd_update: SSDUpdateKernel,
+ ssd_prefill: SSDPrefillKernels,
+ ssd_update: SSDUpdateMetalKernel,
conv_weight: MetalArray,
conv_bias: Option<MetalArray>,
gate_bias: MetalArray,
@@ -52,7 +54,7 @@ impl MambaMixer {
num_heads: usize,
head_dim: usize,
num_groups: usize,
- decoder_layer_loader: &ParameterTree<Rc<MTLContext>>,
+ decoder_layer_loader: &ParameterTree<MTLContext>,
) -> Self {
let _ = (num_heads, head_dim, num_groups);
if !matches!(layer_type, DecoderLayerType::StateSpace { .. }) {
@@ -113,7 +115,7 @@ impl MambaMixer {
split_tree.leaf("skip_connection_weight").unwrap().clone();
let split_inproj =
- SplitInProjKernel::new(mtl_context, kernel_data_type)
+ SplitInProjMetalKernel::new(mtl_context, kernel_data_type.into())
.expect("Failed to create split in-projection kernel");
let conv_scan = Conv1dScanKernel::new(
mtl_context,
@@ -121,10 +123,11 @@ impl MambaMixer {
&mamba_config.activation,
)
.expect("Failed to create conv scan kernel");
- let ssm_prefill = SSDPrefillKernel::new(mtl_context, kernel_data_type)
+ let ssd_prefill = SSDPrefillKernels::new(mtl_context, kernel_data_type)
.expect("Failed to create SSD prefill kernel");
- let ssd_update = SSDUpdateKernel::new(mtl_context, kernel_data_type)
- .expect("Failed to create SSD decode kernel");
+ let ssd_update =
+ SSDUpdateMetalKernel::new(mtl_context, kernel_data_type.into())
+ .expect("Failed to create SSD decode kernel");
let prefill_mode = resolve_prefill_mode_from_env();
Self {
@@ -134,7 +137,7 @@ impl MambaMixer {
out_projection,
split_inproj,
conv_scan,
- ssm_prefill,
+ ssd_prefill,
ssd_update,
conv_weight,
conv_bias,
@@ -182,40 +185,36 @@ impl MambaMixer {
ArrayId::SsmZ(self.layer_index),
ArrayId::SsmDt(self.layer_index),
]);
- let mut in_proj = arrays[0].borrow_mut();
- let mut conv_inputs = arrays[1].borrow_mut();
- let mut gate = arrays[2].borrow_mut();
- let mut dt = arrays[3].borrow_mut();
-
- let input_buf = unsafe { in_proj.mtl_buffer().to_owned() };
- let conv_buf = unsafe { conv_inputs.mtl_buffer().to_owned() };
- let gate_buf = unsafe { gate.mtl_buffer().to_owned() };
- let dt_buf = unsafe { dt.mtl_buffer().to_owned() };
- let mut gate_bias = self.gate_bias.clone();
- let bias_buf = unsafe { gate_bias.mtl_buffer().to_owned() };
+ let in_proj = arrays[0].borrow_mut();
+ let conv_inputs = arrays[1].borrow_mut();
+ let gate = arrays[2].borrow_mut();
+ let dt = arrays[3].borrow_mut();
+
+ let input_buf = in_proj.buffer().to_owned();
+ let conv_buf = conv_inputs.buffer().to_owned();
+ let gate_buf = gate.buffer().to_owned();
+ let dt_buf = dt.buffer().to_owned();
+ let gate_bias = self.gate_bias.clone();
+ let bias_buf = gate_bias.buffer().to_owned();
let conv_dim = self.config.conv_dim();
let inner_dim = self.config.inner_dim();
let num_heads = self.config.num_heads;
let total_dim = conv_dim + inner_dim + num_heads;
- self.split_inproj
- .encode(
- encoder,
- SplitInProjArguments {
- input: &input_buf,
- conv_out: &conv_buf,
- z_out: &gate_buf,
- dt_out: &dt_buf,
- z_bias: &bias_buf,
- total_dim,
- conv_dim,
- inner_dim,
- num_heads,
- suffix_length,
- },
- )
- .expect("Failed to encode split in-projection kernel");
+ self.split_inproj.encode(
+ &input_buf,
+ &conv_buf,
+ &gate_buf,
+ &dt_buf,
+ &bias_buf,
+ suffix_length as u32,
+ total_dim as u32,
+ conv_dim as u32,
+ inner_dim as u32,
+ num_heads as u32,
+ encoder,
+ )
}
fn run_conv_scan(
@@ -231,23 +230,23 @@ impl MambaMixer {
ArrayId::SsmB(self.layer_index),
ArrayId::SsmC(self.layer_index),
]);
- let mut conv_inputs = arrays[0].borrow_mut();
- let mut conv_state = arrays[1].borrow_mut();
- let mut x_arr = arrays[2].borrow_mut();
- let mut b_arr = arrays[3].borrow_mut();
- let mut c_arr = arrays[4].borrow_mut();
-
- let input_buf = unsafe { conv_inputs.mtl_buffer().to_owned() };
- let state_buf = unsafe { objc2::rc::Retained::retain(std::ptr::from_ref(&*conv_state.mtl_buffer()) as *mut _).unwrap() };
- let x_buf = unsafe { x_arr.mtl_buffer().to_owned() };
- let b_buf = unsafe { b_arr.mtl_buffer().to_owned() };
- let c_buf = unsafe { c_arr.mtl_buffer().to_owned() };
+ let conv_inputs = arrays[0].borrow_mut();
+ let conv_state = arrays[1].borrow_mut();
+ let x_arr = arrays[2].borrow_mut();
+ let b_arr = arrays[3].borrow_mut();
+ let c_arr = arrays[4].borrow_mut();
+
+ let input_buf = conv_inputs.buffer().to_owned();
+ let state_buf = conv_state.buffer().clone();
+ let x_buf = x_arr.buffer().to_owned();
+ let b_buf = b_arr.buffer().to_owned();
+ let c_buf = c_arr.buffer().to_owned();
let weight_storage = self.conv_weight.clone();
- let weight_buf = weight_storage.mtl_buffer_cloned();
+ let weight_buf = weight_storage.buffer().clone();
let bias_buf = self.conv_bias.as_ref().map(|arr| {
let storage = arr.clone();
- storage.mtl_buffer_cloned()
+ storage.buffer().clone()
});
let conv_dim = self.config.conv_dim();
@@ -284,8 +283,8 @@ impl MambaMixer {
let array = state
.conv_padded_buffer()
.expect("Missing conv padded buffer");
- let mut borrow = array.borrow_mut();
- let buf = unsafe { objc2::rc::Retained::retain(std::ptr::from_ref(&*borrow.mtl_buffer()) as *mut _).unwrap() };
+ let borrow = array.borrow_mut();
+ let buf = borrow.buffer().clone();
drop(borrow);
self.conv_scan
@@ -350,68 +349,66 @@ impl MambaMixer {
ArrayId::AttentionOutput,
]);
let x = base_arrays[0].borrow();
- let x_buf = x.mtl_buffer_cloned();
+ let x_buf = x.buffer().clone();
drop(x);
let b = base_arrays[1].borrow();
- let b_buf = b.mtl_buffer_cloned();
+ let b_buf = b.buffer().clone();
drop(b);
let c = base_arrays[2].borrow();
- let c_buf = c.mtl_buffer_cloned();
+ let c_buf = c.buffer().clone();
drop(c);
let dt = base_arrays[3].borrow();
- let dt_buf = dt.mtl_buffer_cloned();
+ let dt_buf = dt.buffer().clone();
drop(dt);
let z = base_arrays[4].borrow();
- let z_buf = z.mtl_buffer_cloned();
+ let z_buf = z.buffer().clone();
drop(z);
let state_buf = base_arrays[5].borrow();
- let state_raw = state_buf.mtl_buffer_cloned();
+ let state_raw = state_buf.buffer().clone();
drop(state_buf);
let out = base_arrays[6].borrow();
- let out_buf = out.mtl_buffer_cloned();
+ let out_buf = out.buffer().clone();
drop(out);
let skip_weights = self.skip_connection_weight.clone();
- let skip = skip_weights.mtl_buffer_cloned();
-
- self.ssm_prefill
- .encode(
- encoder,
- SSDPrefillArguments {
- x: &x_buf,
- dt: &dt_buf,
- b: &b_buf,
- c: &c_buf,
- d: &skip,
- z: &z_buf,
- state: &state_raw,
- y: &out_buf,
- suffix_len: suffix_length,
- group_size: (self.config.num_heads / self.config.num_groups)
- as i32,
- state_size: self.config.state_dim as i32,
- x_strides: [
- self.config.num_heads * self.config.head_dim,
- self.config.head_dim,
- 1,
- ],
- dt_strides: [self.config.num_heads, 1],
- cb_strides: [
- self.config.num_groups * self.config.state_dim,
- self.config.state_dim,
- 1,
- ],
- state_strides: [
- self.config.head_dim * self.config.state_dim,
- self.config.state_dim,
- 1,
- ],
- channels: self.config.num_heads,
- head_dim: self.config.head_dim,
- },
- self.prefill_mode,
- )
- .expect("Failed to encode SSD prefill kernel");
+ let skip = skip_weights.buffer().clone();
+
+ self.ssd_prefill.encode(
+ encoder,
+ SSDPrefillArguments {
+ x: &x_buf,
+ dt: &dt_buf,
+ b: &b_buf,
+ c: &c_buf,
+ d: &skip,
+ z: &z_buf,
+ state: &state_raw,
+ y: &out_buf,
+ suffix_len: suffix_length,
+ group_size: (self.config.num_heads / self.config.num_groups)
+ as i32,
+ state_size: self.config.state_dim as i32,
+ x_strides: [
+ self.config.num_heads * self.config.head_dim,
+ self.config.head_dim,
+ 1,
+ ],
+ dt_strides: [self.config.num_heads, 1],
+ cb_strides: [
+ self.config.num_groups * self.config.state_dim,
+ self.config.state_dim,
+ 1,
+ ],
+ state_strides: [
+ self.config.head_dim * self.config.state_dim,
+ self.config.state_dim,
+ 1,
+ ],
+ channels: self.config.num_heads,
+ head_dim: self.config.head_dim,
+ },
+ self.prefill_mode,
+ )
}
fn run_decode_ssm(
@@ -430,65 +427,61 @@ impl MambaMixer {
ArrayId::AttentionOutput,
]);
let x = arrays[0].borrow();
- let x_buf = x.mtl_buffer_cloned();
+ let x_buf = x.buffer().clone();
drop(x);
let b = arrays[1].borrow();
- let b_buf = b.mtl_buffer_cloned();
+ let b_buf = b.buffer().clone();
drop(b);
let c = arrays[2].borrow();
- let c_buf = c.mtl_buffer_cloned();
+ let c_buf = c.buffer().clone();
drop(c);
let dt = arrays[3].borrow();
- let dt_buf = dt.mtl_buffer_cloned();
+ let dt_buf = dt.buffer().clone();
drop(dt);
let z = arrays[4].borrow();
- let z_buf = z.mtl_buffer_cloned();
+ let z_buf = z.buffer().clone();
drop(z);
let state_arr = arrays[5].borrow();
- let state_buf = state_arr.mtl_buffer_cloned();
+ let state_buf = state_arr.buffer().clone();
drop(state_arr);
let y = arrays[6].borrow();
- let y_buf = y.mtl_buffer_cloned();
+ let y_buf = y.buffer().clone();
let skip_weights = self.skip_connection_weight.clone();
- let skip_buf = skip_weights.mtl_buffer_cloned();
+ let skip_buf = skip_weights.buffer().clone();
- let h = self.config.num_heads;
- let g = self.config.num_groups;
- let dh = self.config.head_dim;
- let n = self.config.state_dim;
+ let h = self.config.num_heads as u32;
+ let g = self.config.num_groups as u32;
+ let dh = self.config.head_dim as u32;
+ let n = self.config.state_dim as u32;
- let x_strides = [h * dh, dh, 1usize];
- let dt_strides = [h, 1usize];
- let cb_strides = [g * n, n, 1usize];
- let state_strides = [h * dh * n, dh * n, n, 1usize];
+ let x_strides = [h * dh, dh, 1u32];
+ let dt_strides = [h, 1u32];
+ let cb_strides = [g * n, n, 1u32];
+ let state_strides = [h * dh * n, dh * n, n, 1u32];
let group_size = (h / g) as i32;
let state_size = n as i32;
- self.ssd_update
- .encode(
- encoder,
- SSDUpdateArguments {
- x: &x_buf,
- dt: &dt_buf,
- b: &b_buf,
- c: &c_buf,
- d: &skip_buf,
- z: &z_buf,
- state: &state_buf,
- y: &y_buf,
- next_state: &state_buf,
- group_size,
- state_size,
- x_strides,
- dt_strides,
- cb_strides,
- state_strides,
- b_size: suffix_length,
- h_size: h,
- dh_size: dh,
- },
- )
- .expect("Failed to encode SSD decode kernel");
+ self.ssd_update.encode(
+ &x_buf,
+ &dt_buf,
+ &b_buf,
+ &c_buf,
+ &skip_buf,
+ &z_buf,
+ &state_buf,
+ &y_buf,
+ &state_buf,
+ group_size as u32,
+ state_size as u32,
+ x_strides.as_slice(),
+ dt_strides.as_slice(),
+ cb_strides.as_slice(),
+ state_strides.as_slice(),
+ suffix_length as u32,
+ h,
+ dh,
+ encoder,
+ );
}
}
@@ -505,16 +498,17 @@ fn resolve_prefill_mode_from_env() -> SSDPrefillMode {
}
}
-impl EncodableBlock for MambaMixer {
+impl EncodableBlock<Metal> for MambaMixer {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
if self.supports_shared_encoder() {
- let encoder = command_buffer.new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
self.encode_pipeline_with_encoder(state, &encoder, parameters);
encoder.end_encoding();
@@ -532,7 +526,8 @@ impl EncodableBlock for MambaMixer {
self.in_projection.encode(state, command_buffer, parameters);
- let encoder = command_buffer.new_compute_command_encoder()
+ let encoder = command_buffer
+ .new_compute_command_encoder()
.expect("Failed to create compute command encoder");
self.run_split_inproj(state, &encoder, active_suffix_length);
self.run_conv_scan(state, &encoder, active_suffix_length);
@@ -567,9 +562,9 @@ impl EncodableBlock for MambaMixer {
}
fn resolve_subtree<'tree>(
- loader: &'tree ParameterTree<Rc<MTLContext>>,
+ loader: &'tree ParameterTree<MTLContext>,
candidates: &[&str],
-) -> ParameterTree<'tree, Rc<MTLContext>> {
+) -> ParameterTree<'tree, MTLContext> {
for candidate in candidates {
if let Ok(tree) = loader.subtree(candidate) {
return tree;
diff --git a/crates/uzu/src/backends/metal/encodable_block/mlp.rs b/crates/uzu/src/backends/metal/encodable_block/mlp.rs
index 14eafe50a..b32a00eec 100644
--- a/crates/uzu/src/backends/metal/encodable_block/mlp.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/mlp.rs
@@ -7,9 +7,9 @@ use crate::backends::metal::{
ProtocolObject, Retained,
};
-use super::{EncodableBlock, EncodingParameters};
+use super::{EncodableBlock, EncodingParameters, Metal};
use crate::{
- Array, DataType,
+ DataType,
backends::metal::{
MTLContext,
forward_pass::{ArrayId, ForwardPassState},
@@ -26,16 +26,16 @@ use crate::{
};
pub struct MlpBlock {
- up: Box<dyn EncodableBlock>,
+ up: Box<dyn EncodableBlock<Metal>>,
gate: MlpGateActMulEncodable,
- down: Box<dyn EncodableBlock>,
+ down: Box<dyn EncodableBlock<Metal>>,
}
impl MlpBlock {
pub fn new(
- up: Box<dyn EncodableBlock>,
+ up: Box<dyn EncodableBlock<Metal>>,
gate: MlpGateActMulEncodable,
- down: Box<dyn EncodableBlock>,
+ down: Box<dyn EncodableBlock<Metal>>,
) -> Self {
Self {
up,
@@ -45,16 +45,17 @@ impl MlpBlock {
}
}
-impl EncodableBlock for MlpBlock {
+impl EncodableBlock<Metal> for MlpBlock {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
params: &EncodingParameters,
) {
if self.supports_shared_encoder() {
- let encoder = command_buffer.new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
self.encode_with_shared_encoder(state, &encoder, params);
encoder.end_encoding();
} else {
@@ -65,14 +66,15 @@ impl EncodableBlock for MlpBlock {
{
let arrays =
state.arrays(&[ArrayId::MlpFusedUp, ArrayId::MlpHidden]);
- let mut fused = arrays[0].borrow_mut();
- let mut hidden = arrays[1].borrow_mut();
+ let fused = arrays[0].borrow_mut();
+ let hidden = arrays[1].borrow_mut();
let m = fused.shape()[0] as i32;
- let fused_buf = unsafe { fused.mtl_buffer() };
- let hidden_buf = unsafe { hidden.mtl_buffer() };
+ let fused_buf = fused.buffer();
+ let hidden_buf = hidden.buffer();
- let encoder = command_buffer.new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
self.gate
.encode(&encoder, fused_buf, hidden_buf, m)
.expect("Failed to encode MLP activation/mul kernel");
@@ -104,11 +106,11 @@ impl EncodableBlock for MlpBlock {
// Gate act+mul (fused_up -> hidden)
let arrays = state.arrays(&[ArrayId::MlpFusedUp, ArrayId::MlpHidden]);
- let mut fused = arrays[0].borrow_mut();
- let mut hidden = arrays[1].borrow_mut();
+ let fused = arrays[0].borrow_mut();
+ let hidden = arrays[1].borrow_mut();
let m = fused.shape()[0] as i32;
- let fused_buf = unsafe { fused.mtl_buffer() };
- let hidden_buf = unsafe { hidden.mtl_buffer() };
+ let fused_buf = fused.buffer();
+ let hidden_buf = hidden.buffer();
self.gate
.encode(encoder, fused_buf, hidden_buf, m)
.expect("Failed to encode MLP activation/mul kernel");
@@ -137,10 +139,11 @@ pub enum MlpFusedUpKernel {
pub struct MlpFusedBlock {
context: Rc<MTLContext>,
fused_up: MlpFusedUpKernel,
- down: Box<dyn EncodableBlock>,
+ down: Box<dyn EncodableBlock<Metal>>,
weights_buffer: Retained<ProtocolObject<dyn MTLBuffer>>,
scales_buffer: Option<Retained<ProtocolObject<dyn MTLBuffer>>>,
- zero_points_or_biases_buffer: Option<Retained<ProtocolObject<dyn MTLBuffer>>>,
+ zero_points_or_biases_buffer:
+ Option<Retained<ProtocolObject<dyn MTLBuffer>>>,
input_dim: usize,
hidden_dim: usize,
activation: MlpActivationType,
@@ -157,7 +160,7 @@ impl MlpFusedBlock {
input_dim: usize,
hidden_dim: usize,
activation: &Activation,
- down: Box<dyn EncodableBlock>,
+ down: Box<dyn EncodableBlock<Metal>>,
input_array_id: ArrayId,
hidden_array_id: ArrayId,
) -> Result<Self, crate::backends::metal::MTLError> {
@@ -193,7 +196,7 @@ impl MlpFusedBlock {
mode: crate::config::QuantizationMode,
quantization_type: crate::backends::metal::kernel::quant_matmul::QuantizationType,
activation: &Activation,
- down: Box<dyn EncodableBlock>,
+ down: Box<dyn EncodableBlock<Metal>>,
input_array_id: ArrayId,
hidden_array_id: ArrayId,
) -> Result<Self, crate::backends::metal::MTLError> {
@@ -240,9 +243,9 @@ impl MlpFusedBlock {
fn encode_fused_up(
&self,
encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- input: &ProtocolObject<dyn MTLBuffer>,
+ input: &Retained<ProtocolObject<dyn MTLBuffer>>,
input_offset: u64,
- output: &ProtocolObject<dyn MTLBuffer>,
+ output: &Retained<ProtocolObject<dyn MTLBuffer>>,
batch: i32,
) {
match &self.fused_up {
@@ -311,16 +314,17 @@ impl MlpFusedBlock {
}
}
-impl EncodableBlock for MlpFusedBlock {
+impl EncodableBlock<Metal> for MlpFusedBlock {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
params: &EncodingParameters,
) {
if self.supports_shared_encoder() {
- let encoder = command_buffer.new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
self.encode_with_shared_encoder(state, &encoder, params);
encoder.end_encoding();
} else {
@@ -328,14 +332,15 @@ impl EncodableBlock for MlpFusedBlock {
{
let arrays =
state.arrays(&[self.input_array_id, self.hidden_array_id]);
- let mut input = arrays[0].borrow_mut();
- let mut hidden = arrays[1].borrow_mut();
+ let input = arrays[0].borrow_mut();
+ let hidden = arrays[1].borrow_mut();
let batch = input.shape()[0] as i32;
- let input_buf = unsafe { input.mtl_buffer() };
- let hidden_buf = unsafe { hidden.mtl_buffer() };
+ let input_buf = input.buffer();
+ let hidden_buf = hidden.buffer();
- let encoder = command_buffer.new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
self.encode_fused_up(&encoder, input_buf, 0, hidden_buf, batch);
encoder.end_encoding();
}
@@ -362,11 +367,11 @@ impl EncodableBlock for MlpFusedBlock {
) {
// Fused up + activation
let arrays = state.arrays(&[self.input_array_id, self.hidden_array_id]);
- let mut input = arrays[0].borrow_mut();
- let mut hidden = arrays[1].borrow_mut();
+ let input = arrays[0].borrow_mut();
+ let hidden = arrays[1].borrow_mut();
let batch = input.shape()[0] as i32;
- let input_buf = unsafe { input.mtl_buffer() };
- let hidden_buf = unsafe { hidden.mtl_buffer() };
+ let input_buf = input.buffer();
+ let hidden_buf = hidden.buffer();
self.encode_fused_up(encoder, input_buf, 0, hidden_buf, batch);
diff --git a/crates/uzu/src/backends/metal/encodable_block/mod.rs b/crates/uzu/src/backends/metal/encodable_block/mod.rs
index 8938b4a26..26e150c67 100644
--- a/crates/uzu/src/backends/metal/encodable_block/mod.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/mod.rs
@@ -6,10 +6,9 @@
//! Encodables implement `EncodableBlock` and orchestrate one or more
//! kernels to perform operations on `ForwardPassState`.
-use super::{
- MTLCommandBuffer, MTLComputeCommandEncoder, ProtocolObject,
- forward_pass::ForwardPassState,
-};
+pub use super::Metal;
+use super::forward_pass::ForwardPassState;
+pub use crate::encodable_block::EncodableBlock;
mod activation;
mod attention;
@@ -26,7 +25,6 @@ mod normalization;
mod pooling;
mod prediction_head;
mod rope;
-mod sampling;
mod short_conv_mixer;
mod tensor_add_swap;
mod tensor_copy;
@@ -53,35 +51,6 @@ pub use normalization::{
pub use pooling::Pooling;
pub use prediction_head::ClassifierPredictionHead;
pub use rope::Rope;
-pub use sampling::Sampling;
pub(crate) use short_conv_mixer::ShortConvMixer;
pub use tensor_add_swap::TensorAddSwap;
pub use tensor_copy::TensorCopy;
-
-/// Trait for encodable blocks that operate on `ForwardPassState`.
-pub trait EncodableBlock {
- fn encode(
- &self,
- state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- parameters: &EncodingParameters,
- );
-
- /// Returns true if this block supports using a shared compute encoder.
- ///
- /// If true, `encode_with_shared_encoder` must correctly encode into the
- /// provided encoder without creating or ending encoders internally.
- fn supports_shared_encoder(&self) -> bool {
- false
- }
-
- /// Encode using a shared compute encoder. Only called if `supports_shared_encoder` returns true.
- fn encode_with_shared_encoder(
- &self,
- _state: &mut ForwardPassState,
- _encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- _parameters: &EncodingParameters,
- ) {
- panic!("encode_with_shared_encoder called on unsupported type");
- }
-}
diff --git a/crates/uzu/src/backends/metal/encodable_block/moe_block/moe_block.rs b/crates/uzu/src/backends/metal/encodable_block/moe_block/moe_block.rs
index e93eb0d23..e2bc760a1 100644
--- a/crates/uzu/src/backends/metal/encodable_block/moe_block/moe_block.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/moe_block/moe_block.rs
@@ -2,28 +2,34 @@
use std::{cell::RefCell, rc::Rc};
-use crate::backends::metal::{
- MTLBlitCommandEncoder, MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
- MTLComputeCommandEncoder, NSRange, ProtocolObject, Retained,
-};
-
use super::{
- super::{EncodableBlock, EncodingParameters},
+ super::{EncodableBlock, EncodingParameters, Metal},
SharedMoeWeights,
};
use crate::{
Activation, DataType, LinearConfig, MixtureOfExpertsConfig,
RoutingFunctionConfig,
- backends::metal::{
- KernelDataType, MTLContext, MetalArray,
- forward_pass::{ArrayId, ForwardPassState},
- kernel::moe::{
- MoeBlockBasesArguments, MoeCountsOffsetsFusedArguments,
- MoeCountsOffsetsFusedKernel, MoeExpertsTwoPassArguments,
- MoeExpertsTwoPassDecodeKernel, MoeExpertsTwoPassPrefillKernel,
- MoeFinalizeArguments, MoeFinalizeKernel, MoeGatherArguments,
- MoeGatherKernel, MoeRouterTopKArguments, MoeRouterTopKKernel,
- MoeScatterArguments, MoeScatterKernels, MoeScatterWithMapArguments,
+ backends::{
+ common::kernel::{MoeCountsOffsetsFusedKernel, MoeFinalizeKernel},
+ metal::{
+ KernelDataType, MTLBlitCommandEncoder, MTLBuffer, MTLCommandBuffer,
+ MTLCommandEncoder, MTLComputeCommandEncoder, MTLContext,
+ MetalArray, NSRange, ProtocolObject, Retained,
+ forward_pass::{ArrayId, ForwardPassState},
+ kernel::{
+ MoeGatherKernels,
+ dsl::{
+ MoeCountsOffsetsFusedMetalKernel, MoeFinalizeMetalKernel,
+ },
+ moe::{
+ MoeBlockBasesArguments, MoeExpertsTwoPassArguments,
+ MoeExpertsTwoPassDecodeKernels,
+ MoeExpertsTwoPassPrefillKernel, MoeGatherArguments,
+ MoeRouterTopKArguments, MoeRouterTopKKernel,
+ MoeScatterArguments, MoeScatterKernels,
+ MoeScatterWithMapArguments,
+ },
+ },
},
},
parameters::ParameterTree,
@@ -41,12 +47,12 @@ pub struct MoeBlock {
router_data_type: KernelDataType,
router_renorm: bool,
router_topk_kernel: MoeRouterTopKKernel,
- counts_offsets_kernel: MoeCountsOffsetsFusedKernel,
+ counts_offsets_kernel: MoeCountsOffsetsFusedMetalKernel,
scatter_kernels: MoeScatterKernels,
- gather_kernel: MoeGatherKernel,
- experts_two_pass_decode_kernel: MoeExpertsTwoPassDecodeKernel,
+ gather_kernels: MoeGatherKernels,
+ experts_two_pass_decode_kernel: MoeExpertsTwoPassDecodeKernels,
experts_two_pass_prefill_kernel: MoeExpertsTwoPassPrefillKernel,
- finalize_kernel: MoeFinalizeKernel,
+ finalize_kernel: MoeFinalizeMetalKernel,
moe_config: MixtureOfExpertsConfig,
model_dim: usize,
hidden_dim: usize,
@@ -60,7 +66,7 @@ impl MoeBlock {
moe_config: &MixtureOfExpertsConfig,
model_dim: usize,
hidden_dim: usize,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
) -> Result<Self, crate::backends::metal::MTLError> {
let activation_data_type: DataType = moe_config
.expert_config
@@ -99,23 +105,21 @@ impl MoeBlock {
LinearConfig::FullPrecision {
..
} => {
- let mut weights_arr =
- router_tree.leaf("weights").map_err(|e| {
- crate::backends::metal::MTLError::Generic(format!(
- "Router weights error: {:?}",
- e
- ))
- })?;
- let weights_buf = unsafe { weights_arr.mtl_buffer() };
-
- let mut biases_arr =
- router_tree.leaf("biases").map_err(|e| {
- crate::backends::metal::MTLError::Generic(format!(
- "Router biases error: {:?}",
- e
- ))
- })?;
- let biases_buf = unsafe { biases_arr.mtl_buffer() };
+ let weights_arr = router_tree.leaf("weights").map_err(|e| {
+ crate::backends::metal::MTLError::Generic(format!(
+ "Router weights error: {:?}",
+ e
+ ))
+ })?;
+ let weights_buf = weights_arr.buffer();
+
+ let biases_arr = router_tree.leaf("biases").map_err(|e| {
+ crate::backends::metal::MTLError::Generic(format!(
+ "Router biases error: {:?}",
+ e
+ ))
+ })?;
+ let biases_buf = biases_arr.buffer();
RouterBlock::Metal {
weights_buf: weights_buf.into(),
@@ -149,14 +153,14 @@ impl MoeBlock {
e
))
})?;
- let gather_kernel = MoeGatherKernel::new(context).map_err(|e| {
+ let gather_kernel = MoeGatherKernels::new(context).map_err(|e| {
crate::backends::metal::MTLError::Generic(format!(
"Gather kernel error: {:?}",
e
))
})?;
let experts_two_pass_decode_kernel =
- MoeExpertsTwoPassDecodeKernel::new(context).map_err(|e| {
+ MoeExpertsTwoPassDecodeKernels::new(context).map_err(|e| {
crate::backends::metal::MTLError::Generic(format!(
"Experts two-pass decode kernel error: {:?}",
e
@@ -169,12 +173,15 @@ impl MoeBlock {
e
))
})?;
- let finalize_kernel = MoeFinalizeKernel::new(context).map_err(|e| {
- crate::backends::metal::MTLError::Generic(format!(
- "Finalize kernel error: {:?}",
- e
- ))
- })?;
+ let finalize_kernel =
+ MoeFinalizeMetalKernel::new(context, data_type.into()).map_err(
+ |e| {
+ crate::backends::metal::MTLError::Generic(format!(
+ "Finalize kernel error: {:?}",
+ e
+ ))
+ },
+ )?;
let experts_tree = parameter_tree.subtree("experts").map_err(|e| {
crate::backends::metal::MTLError::Generic(format!(
@@ -198,7 +205,7 @@ impl MoeBlock {
e
))
})?;
- let w13_buf = w13_arr.mtl_buffer_cloned();
+ let w13_buf = w13_arr.buffer().clone();
let w2_arr = experts_tree
.subtree("down_projection")
@@ -215,7 +222,7 @@ impl MoeBlock {
e
))
})?;
- let w2_buf = w2_arr.mtl_buffer_cloned();
+ let w2_buf = w2_arr.buffer().clone();
let up_biases_arr = experts_tree
.subtree("up_projection")
@@ -232,7 +239,7 @@ impl MoeBlock {
e
))
})?;
- let up_biases_buf = up_biases_arr.mtl_buffer_cloned();
+ let up_biases_buf = up_biases_arr.buffer().clone();
let down_biases_arr = experts_tree
.subtree("down_projection")
@@ -249,7 +256,7 @@ impl MoeBlock {
e
))
})?;
- let down_biases_buf = down_biases_arr.mtl_buffer_cloned();
+ let down_biases_buf = down_biases_arr.buffer().clone();
let shared_weights = SharedMoeWeights {
w13_buf: Rc::new(w13_buf),
@@ -265,7 +272,7 @@ impl MoeBlock {
router_topk_kernel,
counts_offsets_kernel,
scatter_kernels,
- gather_kernel,
+ gather_kernels: gather_kernel,
experts_two_pass_decode_kernel,
experts_two_pass_prefill_kernel,
finalize_kernel,
@@ -290,11 +297,11 @@ impl MoeBlock {
}
}
-impl EncodableBlock for MoeBlock {
+impl EncodableBlock<Metal> for MoeBlock {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
let suffix_length = state.active_suffix_length();
@@ -322,9 +329,7 @@ impl EncodableBlock for MoeBlock {
]);
let clone_buffer = |array: &RefCell<MetalArray>| -> Retained<ProtocolObject<dyn MTLBuffer>> {
- let mut borrow = array.borrow_mut();
- let buffer = unsafe { borrow.mtl_buffer().to_owned().into() };
- buffer
+ array.borrow().buffer().to_owned().into()
};
let mut array_iter = arrays.iter();
@@ -353,8 +358,7 @@ impl EncodableBlock for MoeBlock {
let e = self.moe_config.mixture_size;
let k = self.moe_config.num_experts_per_token;
- let mtl_command_buffer = command_buffer.to_owned();
- let root = &*mtl_command_buffer;
+ let root = command_buffer;
let k_tile = 128;
// Clear internal MoE buffers
@@ -433,7 +437,7 @@ impl EncodableBlock for MoeBlock {
// Use the fused router+topk kernel
self.router_topk_kernel
.encode(
- &root,
+ root,
self.router_data_type,
MoeRouterTopKArguments {
input_buffer: &main_buf,
@@ -452,27 +456,29 @@ impl EncodableBlock for MoeBlock {
},
}
- self.counts_offsets_kernel
- .encode(
- &root,
- MoeCountsOffsetsFusedArguments {
- topk_ids_buffer: &topk_ids_buf,
- offsets_buffer: &offsets_buf,
- sum_k_buffer: &sumk_buf,
- partials_buffer: &partials_buf,
- t: suffix_length,
- e,
- k,
- },
- )
- .expect("MoE counts+offsets failed");
+ {
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+ self.counts_offsets_kernel.encode(
+ &topk_ids_buf,
+ &offsets_buf,
+ &sumk_buf,
+ &partials_buf,
+ suffix_length as u32,
+ e as u32,
+ k as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
+ }
let num_blocks = ((suffix_length + 255) / 256).max(1);
let num_tiles = ((e + 512 - 1) / 512).max(1);
self.scatter_kernels
.encode_block_bases(
- &root,
+ root,
MoeBlockBasesArguments {
partials_buffer: &partials_buf,
block_bases_buffer: &block_bases_buf,
@@ -486,7 +492,7 @@ impl EncodableBlock for MoeBlock {
self.scatter_kernels
.encode_scatter_with_map(
- &root,
+ root,
MoeScatterWithMapArguments {
base: MoeScatterArguments {
topk_ids_buffer: &topk_ids_buf,
@@ -508,21 +514,19 @@ impl EncodableBlock for MoeBlock {
)
.expect("MoE scatter failed");
- self.gather_kernel
- .encode(
- &root,
- self.data_type,
- MoeGatherArguments {
- x_buffer: &main_buf,
- bucketed_ids_buffer: &bucketed_ids_buf,
- x_perm_buffer: &x_perm_buf,
- sumk_buffer: &sumk_buf,
- t: suffix_length,
- k,
- d_model: self.model_dim,
- },
- )
- .expect("MoE gather failed");
+ self.gather_kernels.encode(
+ root,
+ self.data_type,
+ &MoeGatherArguments {
+ x_buffer: &main_buf,
+ bucketed_ids_buffer: &bucketed_ids_buf,
+ x_perm_buffer: &x_perm_buf,
+ sumk_buffer: &sumk_buf,
+ t: suffix_length,
+ k,
+ d_model: self.model_dim,
+ },
+ );
let gating_code = Self::gating_code_from_activation(
&self.moe_config.expert_config.activation,
@@ -541,46 +545,44 @@ impl EncodableBlock for MoeBlock {
let total_rows = suffix_length * k;
let num_tiles_k = ((self.hidden_dim + k_tile - 1) / k_tile) as u32;
- self.experts_two_pass_decode_kernel
- .encode(
- &root,
- MoeExpertsTwoPassArguments {
- x_perm_buffer: &x_perm_buf,
- expert_offsets: &offsets_buf,
- row_expert_map: &row_expert_map_buf,
- hidden_buffer: &hidden_buf,
- output_buffer: &y_partial_buf,
- w13_all: &self.shared_weights.w13_buf,
- w2_all: &self.shared_weights.w2_buf,
- up_biases: &self.shared_weights.up_biases_buf,
- down_biases: &self.shared_weights.down_biases_buf,
- tile_counts: &tile_counts_buf,
- tile_offsets: &tile_offsets_buf,
- tile_map: &tile_map_buf,
- total_tiles: &total_tiles_buf,
- dispatch_args: &dispatch_args_buf,
- total_rows,
- d_model: self.model_dim,
- d_ff: self.hidden_dim,
- e,
- num_tiles_k,
- gating_code,
- gate_clip_min,
- gate_clip_max,
- up_clip_min,
- up_clip_max,
- silu_alpha,
- data_type: self.data_type,
- },
- )
- .expect("MoE experts two-pass failed");
+ self.experts_two_pass_decode_kernel.encode(
+ root,
+ &MoeExpertsTwoPassArguments {
+ x_perm_buffer: &x_perm_buf,
+ expert_offsets: &offsets_buf,
+ row_expert_map: &row_expert_map_buf,
+ hidden_buffer: &hidden_buf,
+ output_buffer: &y_partial_buf,
+ w13_all: &self.shared_weights.w13_buf,
+ w2_all: &self.shared_weights.w2_buf,
+ up_biases: &self.shared_weights.up_biases_buf,
+ down_biases: &self.shared_weights.down_biases_buf,
+ tile_counts: &tile_counts_buf,
+ tile_offsets: &tile_offsets_buf,
+ tile_map: &tile_map_buf,
+ total_tiles: &total_tiles_buf,
+ dispatch_args: &dispatch_args_buf,
+ total_rows,
+ d_model: self.model_dim,
+ d_ff: self.hidden_dim,
+ e,
+ num_tiles_k,
+ gating_code,
+ gate_clip_min,
+ gate_clip_max,
+ up_clip_min,
+ up_clip_max,
+ silu_alpha,
+ data_type: self.data_type,
+ },
+ );
} else {
let total_rows = suffix_length * k;
let num_tiles_k = ((self.hidden_dim + k_tile - 1) / k_tile) as u32;
self.experts_two_pass_prefill_kernel
.encode(
- &root,
+ root,
MoeExpertsTwoPassArguments {
x_perm_buffer: &x_perm_buf,
expert_offsets: &offsets_buf,
@@ -613,21 +615,22 @@ impl EncodableBlock for MoeBlock {
.expect("MoE experts two-pass prefill failed");
}
- self.finalize_kernel
- .encode(
- &root,
- MoeFinalizeArguments {
- tok2row_buffer: &tok2row_buf,
- probs_buffer: &topk_probs_buf,
- y_partial_buffer: &y_partial_buf,
- y_out_buffer: &main_buf,
- t: suffix_length,
- d_model: self.model_dim,
- k,
- },
- self.data_type,
- )
- .expect("MoE finalize failed");
+ {
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+ self.finalize_kernel.encode(
+ &tok2row_buf,
+ &topk_probs_buf,
+ &y_partial_buf,
+ &main_buf,
+ suffix_length as u32,
+ self.model_dim as u32,
+ k as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
+ }
if parameters.wait_until_completed {
command_buffer.commit();
diff --git a/crates/uzu/src/backends/metal/encodable_block/normalization/layer_norm.rs b/crates/uzu/src/backends/metal/encodable_block/normalization/layer_norm.rs
index 9f081180f..d88dabdee 100644
--- a/crates/uzu/src/backends/metal/encodable_block/normalization/layer_norm.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/normalization/layer_norm.rs
@@ -1,16 +1,16 @@
//! LayerNorm encodable.
-use std::rc::Rc;
-
-use super::super::{EncodableBlock, EncodingParameters};
+use super::super::{EncodableBlock, EncodingParameters, Metal};
use crate::{
- Array, DataType,
- backends::metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- MTLContext, MTLDeviceExt, MTLError, MTLResourceOptions, ProtocolObject, Retained,
- forward_pass::{ArrayId, ForwardPassState},
- kernel::layer_norm::{
- LayerNormArguments, LayerNormError, LayerNormKernel,
+ DataType,
+ backends::{
+ common::kernel::LayerNormKernel,
+ metal::{
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
+ MTLComputeCommandEncoder, MTLContext, MTLDeviceExt, MTLError,
+ MTLResourceOptions, ProtocolObject, Retained,
+ forward_pass::{ArrayId, ForwardPassState},
+ kernel::dsl::LayerNormMetalKernel,
},
},
config::{NormalizationConfig, UpcastMode},
@@ -18,7 +18,7 @@ use crate::{
};
pub struct LayerNorm {
- kernel: LayerNormKernel,
+ kernel: LayerNormMetalKernel,
config: NormalizationConfig,
input_array_id: ArrayId,
output_array_id: ArrayId,
@@ -32,19 +32,19 @@ impl LayerNorm {
config: NormalizationConfig,
input_array_id: ArrayId,
output_array_id: ArrayId,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
- ) -> Result<Self, LayerNormError> {
+ parameter_tree: &ParameterTree<MTLContext>,
+ ) -> Result<Self, MTLError> {
// Load scales from parameter tree
let scales_param = parameter_tree.leaf("scales").map_err(|e| {
- LayerNormError::MetalError(MTLError::Library(
+ MTLError::Library(
crate::backends::metal::error::LibraryError::Custom(format!(
"Failed to load scales: {:?}",
e
)),
- ))
+ )
})?;
- let scales_data = scales_param.buffer();
+ let scales_data = scales_param.as_bytes();
let scales_buffer = context
.device
.new_buffer_with_data(
@@ -57,24 +57,13 @@ impl LayerNorm {
config.accumulation_precision.into();
let scale_data_type: DataType = config.scale_precision.into();
- let (input_type, scales_type, output_type) = match config.upcast_mode {
- UpcastMode::OnlyNormalization => {
- (intermediate_data_type, scale_data_type, scale_data_type)
- },
- UpcastMode::FullLayer => {
- (intermediate_data_type, scale_data_type, scale_data_type)
- },
- };
-
- let kernel = LayerNormKernel::new(
+ let kernel = LayerNormMetalKernel::new(
context,
- input_type,
- scales_type,
- output_type,
- accumulation_data_type,
- config.upcast_mode == UpcastMode::FullLayer,
+ intermediate_data_type.into(),
+ scale_data_type.into(),
+ scale_data_type.into(),
+ accumulation_data_type.into(),
)?;
-
Ok(Self {
kernel,
config,
@@ -85,11 +74,11 @@ impl LayerNorm {
}
}
-impl EncodableBlock for LayerNorm {
+impl EncodableBlock<Metal> for LayerNorm {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
let compute_encoder = command_buffer
@@ -122,28 +111,30 @@ impl EncodableBlock for LayerNorm {
input_array.shape().to_vec()
};
- let mut input_array = input_binding[0].borrow_mut();
- let mut output_array = output_binding[0].borrow_mut();
+ let input_array = input_binding[0].borrow_mut();
+ let output_array = output_binding[0].borrow_mut();
- let input_buffer = unsafe { input_array.mtl_buffer() };
- let output_buffer = unsafe { output_array.mtl_buffer() };
+ let input_buffer = input_array.buffer();
+ let output_buffer = output_array.buffer();
- let batch_size = input_shape[0] as i32;
- let model_dim = input_shape[1] as i32;
+ let batch_size = input_shape[0] as u32;
+ let model_dim = input_shape[1] as u32;
+ let full_layer = if self.config.upcast_mode == UpcastMode::FullLayer {
+ 1u32
+ } else {
+ 0u32
+ };
- if let Err(e) = self.kernel.encode(
+ self.kernel.encode(
+ &input_buffer,
+ &self.scales_buffer,
+ &output_buffer,
+ batch_size,
+ model_dim,
+ self.config.epsilon,
+ self.config.scale_offset.unwrap_or(0.0),
+ full_layer,
compute_encoder,
- LayerNormArguments {
- input_buffer: &input_buffer,
- scales_buffer: &self.scales_buffer,
- output_buffer: &output_buffer,
- batch_size,
- model_dim,
- epsilon: self.config.epsilon,
- scale_offset: self.config.scale_offset.unwrap_or(0.0),
- },
- ) {
- eprintln!("Failed to encode LayerNorm kernel: {:?}", e);
- }
+ )
}
}
diff --git a/crates/uzu/src/backends/metal/encodable_block/normalization/mod.rs b/crates/uzu/src/backends/metal/encodable_block/normalization/mod.rs
index 48937ebce..e892320d8 100644
--- a/crates/uzu/src/backends/metal/encodable_block/normalization/mod.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/normalization/mod.rs
@@ -4,19 +4,19 @@ mod layer_norm;
mod qk_norm;
mod rms_norm;
-use std::rc::Rc;
-
pub use layer_norm::LayerNorm;
pub use qk_norm::QKNorm;
pub use rms_norm::RMSNorm;
-use super::{EncodableBlock, EncodingParameters};
+use super::{EncodableBlock, EncodingParameters, Metal};
+use crate::backends::metal::MTLError;
use crate::{
DataType,
backends::metal::{
MTLCommandBuffer, MTLComputeCommandEncoder, MTLContext, ProtocolObject,
+ Retained,
forward_pass::{ArrayId, ForwardPassState},
- kernel::{LayerNormError, RMSNormError},
+ kernel::RMSNormError,
},
config::NormalizationConfig,
parameters::ParameterTree,
@@ -26,7 +26,7 @@ use crate::{
#[derive(Debug, thiserror::Error)]
pub enum NormalizationError {
#[error("LayerNorm error: {0}")]
- LayerNorm(#[from] LayerNormError),
+ LayerNorm(#[from] MTLError),
#[error("RMSNorm error: {0}")]
RMSNorm(#[from] RMSNormError),
}
@@ -46,7 +46,7 @@ impl Normalization {
config: NormalizationConfig,
input_array_id: ArrayId,
output_array_id: ArrayId,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
) -> Result<Self, NormalizationError> {
if config.subtract_mean {
// Use LayerNorm (subtract mean before normalization)
@@ -72,11 +72,11 @@ impl Normalization {
}
}
-impl EncodableBlock for Normalization {
+impl EncodableBlock<Metal> for Normalization {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
match self {
diff --git a/crates/uzu/src/backends/metal/encodable_block/normalization/qk_norm.rs b/crates/uzu/src/backends/metal/encodable_block/normalization/qk_norm.rs
index 939a7acac..a9694ee31 100644
--- a/crates/uzu/src/backends/metal/encodable_block/normalization/qk_norm.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/normalization/qk_norm.rs
@@ -1,13 +1,12 @@
//! QK Normalization encodable.
-use std::rc::Rc;
-
-use super::super::{EncodableBlock, EncodingParameters};
+use super::super::{EncodableBlock, EncodingParameters, Metal};
use crate::{
- Array, DataType,
+ DataType,
backends::metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- MTLContext, MTLDeviceExt, MTLError, MTLResourceOptions, ProtocolObject, Retained,
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
+ MTLComputeCommandEncoder, MTLContext, MTLDeviceExt, MTLError,
+ MTLResourceOptions, ProtocolObject, Retained,
forward_pass::{ArrayId, ForwardPassState},
kernel::rms_norm::{
QKNormArguments, QKNormTarget, RMSNormError, RMSNormKernel,
@@ -38,7 +37,7 @@ impl QKNorm {
query_config: Option<NormalizationConfig>,
key_config: Option<NormalizationConfig>,
qkv_array_id: ArrayId,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
num_q_heads: usize,
num_kv_heads: usize,
head_dim: usize,
@@ -59,7 +58,7 @@ impl QKNorm {
))
})?;
- let scales_data = scales_param.buffer();
+ let scales_data = scales_param.as_bytes();
let scales_buffer = context.device.new_buffer_with_data(
scales_data,
MTLResourceOptions::STORAGE_MODE_SHARED,
@@ -105,7 +104,7 @@ impl QKNorm {
))
})?;
- let scales_data = scales_param.buffer();
+ let scales_data = scales_param.as_bytes();
let scales_buffer = context.device.new_buffer_with_data(
scales_data,
MTLResourceOptions::STORAGE_MODE_SHARED,
@@ -155,11 +154,11 @@ impl QKNorm {
}
}
-impl EncodableBlock for QKNorm {
+impl EncodableBlock<Metal> for QKNorm {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
let compute_encoder = command_buffer
@@ -190,8 +189,8 @@ impl EncodableBlock for QKNorm {
qkv_array.shape().to_vec()
};
- let mut qkv_array = qkv_binding[0].borrow_mut();
- let qkv_buffer = unsafe { qkv_array.mtl_buffer() };
+ let qkv_array = qkv_binding[0].borrow_mut();
+ let qkv_buffer = qkv_array.buffer();
let batch_size = qkv_shape[0] as i32;
let head_dim = self.head_dim as i32;
diff --git a/crates/uzu/src/backends/metal/encodable_block/normalization/rms_norm.rs b/crates/uzu/src/backends/metal/encodable_block/normalization/rms_norm.rs
index e7b704d8d..4ec1da37c 100644
--- a/crates/uzu/src/backends/metal/encodable_block/normalization/rms_norm.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/normalization/rms_norm.rs
@@ -1,13 +1,12 @@
//! RMS Normalization encodable.
-use std::rc::Rc;
-
-use super::super::{EncodableBlock, EncodingParameters};
+use super::super::{EncodableBlock, EncodingParameters, Metal};
use crate::{
- Array, DataType,
+ DataType,
backends::metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- MTLContext, MTLDeviceExt, MTLError, MTLResourceOptions, ProtocolObject, Retained,
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
+ MTLComputeCommandEncoder, MTLContext, MTLDeviceExt, MTLError,
+ MTLResourceOptions, ProtocolObject, Retained,
forward_pass::{ArrayId, ForwardPassState},
kernel::rms_norm::{
RMSNormArguments, RMSNormError, RMSNormKernel, RMSNormKernelType,
@@ -33,7 +32,7 @@ impl RMSNorm {
config: NormalizationConfig,
input_array_id: ArrayId,
output_array_id: ArrayId,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
) -> Result<Self, RMSNormError> {
// Load scales from parameter tree
let scales_param = parameter_tree.leaf("scales").map_err(|e| {
@@ -46,7 +45,7 @@ impl RMSNorm {
})?;
// TODO: Don't create buffers dynamically, we need to use forward pass storage for thing like this
- let scales_data = scales_param.buffer();
+ let scales_data = scales_param.as_bytes();
let scales_buffer = context
.device
.new_buffer_with_data(
@@ -98,11 +97,11 @@ impl RMSNorm {
}
}
-impl EncodableBlock for RMSNorm {
+impl EncodableBlock<Metal> for RMSNorm {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
let compute_encoder = command_buffer
@@ -135,14 +134,14 @@ impl EncodableBlock for RMSNorm {
input_array.shape().to_vec()
};
- let mut input_array = input_binding[0].borrow_mut();
- let mut output_array = output_binding[0].borrow_mut();
+ let input_array = input_binding[0].borrow_mut();
+ let output_array = output_binding[0].borrow_mut();
let suffix_length = input_shape[0];
let input_elem_size = input_array.data_type().size_in_bytes();
let output_elem_size = output_array.data_type().size_in_bytes();
- let input_buffer = unsafe { input_array.mtl_buffer() };
- let output_buffer = unsafe { output_array.mtl_buffer() };
+ let input_buffer = input_array.buffer();
+ let output_buffer = output_array.buffer();
let (batch_start, batch_len) = if self.use_sampling_range {
(state.sampling_start(), state.sampling_length())
diff --git a/crates/uzu/src/backends/metal/encodable_block/pooling.rs b/crates/uzu/src/backends/metal/encodable_block/pooling.rs
index df3aa5240..9899cc8d2 100644
--- a/crates/uzu/src/backends/metal/encodable_block/pooling.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/pooling.rs
@@ -1,43 +1,81 @@
//! Pooling encodable for sequence-level aggregation.
-use super::{EncodableBlock, EncodingParameters};
+use super::{EncodableBlock, EncodingParameters, Metal};
#[cfg(feature = "tracing")]
-use crate::Array;
+use crate::backends::metal::MTLBlitCommandEncoder;
use crate::{
- backends::metal::{
- MTLCommandBuffer,
- MTLCommandEncoder, MTLComputeCommandEncoder, ProtocolObject,
- forward_pass::{ArrayId, ForwardPassState},
- kernel::PoolingKernel,
+ backends::{
+ common::kernel::{PoolingClsKernel, PoolingMeanKernel},
+ metal::{
+ KernelDataType, MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
+ MTLComputeCommandEncoder, MTLContext, MTLError, ProtocolObject,
+ Retained,
+ forward_pass::{ArrayId, ForwardPassState},
+ kernel::dsl::{PoolingClsMetalKernel, PoolingMeanMetalKernel},
+ },
},
config::PoolingType,
};
+enum PoolingKernel {
+ Cls(PoolingClsMetalKernel),
+ Mean(PoolingMeanMetalKernel),
+}
+
+impl PoolingKernel {
+ fn encode(
+ &self,
+ input: &Retained<ProtocolObject<dyn MTLBuffer>>,
+ output: &Retained<ProtocolObject<dyn MTLBuffer>>,
+ seq_len: u32,
+ hidden_dim: u32,
+ batch_size: u32,
+ encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
+ ) {
+ match self {
+ Self::Cls(k) => k.encode(
+ input, output, seq_len, hidden_dim, batch_size, encoder,
+ ),
+ Self::Mean(k) => k.encode(
+ input, output, seq_len, hidden_dim, batch_size, encoder,
+ ),
+ }
+ }
+}
+
pub struct Pooling {
pooling_kernel: PoolingKernel,
- pooling_type: PoolingType,
model_dim: usize,
}
impl Pooling {
pub fn new(
- pooling_kernel: PoolingKernel,
+ context: &MTLContext,
+ data_type: KernelDataType,
pooling_type: PoolingType,
model_dim: usize,
- ) -> Self {
- Self {
+ ) -> Result<Self, MTLError> {
+ let pooling_kernel = match pooling_type {
+ PoolingType::Cls => PoolingKernel::Cls(PoolingClsMetalKernel::new(
+ context,
+ data_type.into(),
+ )?),
+ PoolingType::Mean => PoolingKernel::Mean(
+ PoolingMeanMetalKernel::new(context, data_type.into())?,
+ ),
+ };
+ Ok(Self {
pooling_kernel,
- pooling_type,
model_dim,
- }
+ })
}
}
-impl EncodableBlock for Pooling {
+impl EncodableBlock<Metal> for Pooling {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
let encoder = command_buffer
@@ -49,18 +87,17 @@ impl EncodableBlock for Pooling {
#[cfg(feature = "tracing")]
{
let batch_size = 1;
- let data_type = {
- Array::data_type(&*state.arrays(&[ArrayId::Main])[0].borrow())
- };
+ let data_type =
+ { state.arrays(&[ArrayId::Main])[0].borrow().data_type() };
let arrays = state.arrays(&[ArrayId::ClassifierPooling]);
let pooling_array = arrays[0].borrow();
- let output_buffer = pooling_array.backend_buffer();
+ let output_buffer = pooling_array.buffer();
let traces_rc = state.traces().clone();
let traces_ref = traces_rc.borrow();
let trace_arr = traces_ref.output_pooling().borrow();
- let dst_buf = trace_arr.backend_buffer();
+ let dst_buf = trace_arr.buffer();
let blit = command_buffer
.new_blit_command_encoder()
@@ -95,33 +132,19 @@ impl EncodableBlock for Pooling {
let seq_len = state.aux_buffers_suffix_length();
let arrays = state.arrays(&[ArrayId::Main, ArrayId::ClassifierPooling]);
- let mut main_array = arrays[0].borrow_mut();
- let input_buffer = unsafe { main_array.mtl_buffer() };
-
- let mut pooling_array = arrays[1].borrow_mut();
- let output_buffer = unsafe { pooling_array.mtl_buffer().to_owned() };
-
- let result = match self.pooling_type {
- PoolingType::Cls => self.pooling_kernel.encode_cls(
- encoder,
- input_buffer,
- &output_buffer,
- batch_size as i32,
- seq_len as i32,
- self.model_dim as i32,
- ),
- PoolingType::Mean => self.pooling_kernel.encode_mean(
- encoder,
- input_buffer,
- &output_buffer,
- batch_size as i32,
- seq_len as i32,
- self.model_dim as i32,
- ),
- };
+ let main_array = arrays[0].borrow_mut();
+ let input_buffer = main_array.buffer();
- if result.is_err() {
- panic!("Failed to encode pooling kernel");
- }
+ let pooling_array = arrays[1].borrow_mut();
+ let output_buffer = pooling_array.buffer().to_owned();
+
+ self.pooling_kernel.encode(
+ input_buffer,
+ &output_buffer,
+ seq_len as u32,
+ self.model_dim as u32,
+ batch_size,
+ encoder,
+ );
}
}
diff --git a/crates/uzu/src/backends/metal/encodable_block/prediction_head.rs b/crates/uzu/src/backends/metal/encodable_block/prediction_head.rs
index fba8e8aab..b47b01a87 100644
--- a/crates/uzu/src/backends/metal/encodable_block/prediction_head.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/prediction_head.rs
@@ -1,29 +1,28 @@
//! Prediction head encodable for classification output.
-use super::{EncodableBlock, EncodingParameters};
+use super::{EncodableBlock, EncodingParameters, Metal};
#[cfg(feature = "tracing")]
-use crate::Array;
-#[cfg(feature = "tracing")]
-use crate::backends::metal::forward_pass::ArrayId;
-use crate::backends::metal::{ProtocolObject,
- MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder, forward_pass::ForwardPassState,
+use crate::backends::metal::{MTLBlitCommandEncoder, forward_pass::ArrayId};
+use crate::backends::metal::{
+ MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
+ ProtocolObject, Retained, forward_pass::ForwardPassState,
};
pub struct ClassifierPredictionHead {
- dense: Box<dyn EncodableBlock>,
- activation: Box<dyn EncodableBlock>,
- norm: Box<dyn EncodableBlock>,
- readout: Box<dyn EncodableBlock>,
+ dense: Box<dyn EncodableBlock<Metal>>,
+ activation: Box<dyn EncodableBlock<Metal>>,
+ norm: Box<dyn EncodableBlock<Metal>>,
+ readout: Box<dyn EncodableBlock<Metal>>,
#[cfg_attr(not(feature = "tracing"), allow(dead_code))]
num_labels: usize,
}
impl ClassifierPredictionHead {
pub fn new(
- dense: Box<dyn EncodableBlock>,
- activation: Box<dyn EncodableBlock>,
- norm: Box<dyn EncodableBlock>,
- readout: Box<dyn EncodableBlock>,
+ dense: Box<dyn EncodableBlock<Metal>>,
+ activation: Box<dyn EncodableBlock<Metal>>,
+ norm: Box<dyn EncodableBlock<Metal>>,
+ readout: Box<dyn EncodableBlock<Metal>>,
num_labels: usize,
) -> Self {
Self {
@@ -36,11 +35,11 @@ impl ClassifierPredictionHead {
}
}
-impl EncodableBlock for ClassifierPredictionHead {
+impl EncodableBlock<Metal> for ClassifierPredictionHead {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
if self.supports_shared_encoder() {
@@ -62,13 +61,13 @@ impl EncodableBlock for ClassifierPredictionHead {
let logits_arrays =
state.arrays(&[ArrayId::ClassifierPredictionHeadLogits]);
let logits_array_ref = logits_arrays[0].borrow();
- let linear_output_buffer = logits_array_ref.backend_buffer();
- let data_type = Array::data_type(&*logits_array_ref);
- let batch_size = Array::shape(&*logits_array_ref)[0];
+ let linear_output_buffer = logits_array_ref.buffer();
+ let data_type = logits_array_ref.data_type();
+ let batch_size = logits_array_ref.shape()[0];
let traces_ref = traces_rc.borrow();
let trace_logits = traces_ref.logits.borrow();
- let dst_trace_buf = trace_logits.backend_buffer();
+ let dst_trace_buf = trace_logits.buffer();
let copy_size_bytes =
batch_size * self.num_labels * data_type.size_in_bytes();
diff --git a/crates/uzu/src/backends/metal/encodable_block/rope.rs b/crates/uzu/src/backends/metal/encodable_block/rope.rs
index 80f5279a1..d3b6e847a 100644
--- a/crates/uzu/src/backends/metal/encodable_block/rope.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/rope.rs
@@ -1,17 +1,15 @@
//! Rope (Rotary Position Embedding) encodable.
-use crate::backends::metal::{ProtocolObject,
+use crate::backends::metal::{
MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
+ ProtocolObject, Retained,
};
-use super::{EncodableBlock, EncodingParameters};
-use crate::{
- Array,
- backends::metal::{
- KernelDataType, MTLContext,
- forward_pass::{ArrayId, ForwardPassState, RopeType},
- kernel::rope::{RopeError, RopeKernel, RopeKernelArguments},
- },
+use super::{EncodableBlock, EncodingParameters, Metal};
+use crate::backends::metal::{
+ KernelDataType, MTLContext,
+ forward_pass::{ArrayId, ForwardPassState, RopeType},
+ kernel::rope::{RopeError, RopeKernel, RopeKernelArguments},
};
pub struct Rope {
@@ -33,14 +31,15 @@ impl Rope {
}
}
-impl EncodableBlock for Rope {
+impl EncodableBlock<Metal> for Rope {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
- let compute_encoder = command_buffer.new_compute_command_encoder()
+ let compute_encoder = command_buffer
+ .new_compute_command_encoder()
.expect("Failed to create compute command encoder");
self.encode_with_shared_encoder(state, &compute_encoder, parameters);
compute_encoder.end_encoding();
@@ -85,41 +84,37 @@ impl EncodableBlock for Rope {
};
let qkv_buffer_binding = state.arrays(&[ArrayId::QKV]);
- let mut qkv = qkv_buffer_binding[0].borrow_mut();
+ let qkv = qkv_buffer_binding[0].borrow_mut();
let token_positions_binding = state.arrays(&[ArrayId::TokenPositions]);
- let mut token_positions = token_positions_binding[0].borrow_mut();
+ let token_positions = token_positions_binding[0].borrow_mut();
let query_buffer_binding = state.arrays(&[ArrayId::RotatedQueries]);
- let mut rotated_queries = query_buffer_binding[0].borrow_mut();
+ let rotated_queries = query_buffer_binding[0].borrow_mut();
let rotated_keys_binding = state.arrays(&[ArrayId::RotatedKeys]);
- let mut rotated_keys = rotated_keys_binding[0].borrow_mut();
+ let rotated_keys = rotated_keys_binding[0].borrow_mut();
let cos_buffer_binding =
state.arrays(&[ArrayId::RopeCosines(self.rope_type)]);
- let mut rope_cosines = cos_buffer_binding[0].borrow_mut();
+ let rope_cosines = cos_buffer_binding[0].borrow_mut();
let sin_buffer_binding =
state.arrays(&[ArrayId::RopeSines(self.rope_type)]);
- let mut rope_sines = sin_buffer_binding[0].borrow_mut();
+ let rope_sines = sin_buffer_binding[0].borrow_mut();
- let token_positions_offset = token_positions.buffer_offset();
+ let token_positions_offset = token_positions.offset();
self.kernel.encode(
compute_encoder,
RopeKernelArguments {
- qkv_buffer: unsafe { &qkv.mtl_buffer() },
- cosines_buffer: unsafe { &rope_cosines.mtl_buffer() },
- sines_buffer: unsafe { &rope_sines.mtl_buffer() },
- token_positions_buffer: unsafe {
- &token_positions.mtl_buffer()
- },
+ qkv_buffer: &qkv.buffer(),
+ cosines_buffer: &rope_cosines.buffer(),
+ sines_buffer: &rope_sines.buffer(),
+ token_positions_buffer: &token_positions.buffer(),
token_positions_offset,
- rotated_queries_buffer: unsafe {
- &rotated_queries.mtl_buffer()
- },
- rotated_keys_buffer: unsafe { &rotated_keys.mtl_buffer() },
+ rotated_queries_buffer: &rotated_queries.buffer(),
+ rotated_keys_buffer: &rotated_keys.buffer(),
head_dim,
num_heads,
num_groups,
diff --git a/crates/uzu/src/backends/metal/encodable_block/sampling.rs b/crates/uzu/src/backends/metal/encodable_block/sampling.rs
deleted file mode 100644
index 0650dc2b4..000000000
--- a/crates/uzu/src/backends/metal/encodable_block/sampling.rs
+++ /dev/null
@@ -1,141 +0,0 @@
-//! Sampling kernel encodable.
-
-use std::rc::Rc;
-
-use super::{EncodableBlock, EncodingParameters};
-use crate::backends::metal::{ProtocolObject,
- KernelDataType,
- MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder, MTLContext,
- forward_pass::{ArrayId, ForwardPassState},
- kernel::sampling::{ArgmaxStrategy, SamplingError, SamplingKernel},
-};
-
-pub struct Sampling {
- pub kernel: SamplingKernel,
-}
-
-impl Sampling {
- pub fn new(
- context: &Rc<MTLContext>,
- data_type: KernelDataType,
- max_batch_size: usize,
- max_vocab_size: usize,
- ) -> Result<Self, SamplingError> {
- let kernel = SamplingKernel::new(
- context,
- data_type,
- max_batch_size,
- max_vocab_size,
- )?;
- Ok(Self {
- kernel,
- })
- }
-
- pub fn new_with_strategy(
- context: &Rc<MTLContext>,
- data_type: KernelDataType,
- max_batch_size: usize,
- max_vocab_size: usize,
- argmax_strategy: ArgmaxStrategy,
- ) -> Result<Self, SamplingError> {
- let kernel = SamplingKernel::new_with_strategy(
- context,
- data_type,
- max_batch_size,
- max_vocab_size,
- argmax_strategy,
- )?;
- Ok(Self {
- kernel,
- })
- }
-}
-
-impl EncodableBlock for Sampling {
- fn encode(
- &self,
- state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- parameters: &EncodingParameters,
- ) {
- let encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- self.encode_with_shared_encoder(state, &encoder, parameters);
- encoder.end_encoding();
-
- if parameters.wait_until_completed {
- command_buffer.commit();
- command_buffer.wait_until_completed();
- }
- }
-
- fn supports_shared_encoder(&self) -> bool {
- true
- }
-
- fn encode_with_shared_encoder(
- &self,
- state: &mut ForwardPassState,
- encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- _parameters: &EncodingParameters,
- ) {
- assert!(
- state.sampling_output().is_some(),
- "Sampling output buffer must be pre-allocated"
- );
-
- let logits_binding = state.arrays(&[ArrayId::Logits]);
- let mut logits = logits_binding[0].borrow_mut();
-
- let logits_shape = {
- use crate::Array;
- logits.shape()
- };
- let batch_size = state.sampling_length();
- if batch_size == 0 {
- return;
- }
- let sampling_start = state.sampling_start();
- let vocab_size = logits_shape[1];
-
- let seeds_binding = state.arrays(&[ArrayId::TokenSeeds]);
- let mut seeds = seeds_binding[0].borrow_mut();
-
- let mut output_buffer_ref =
- state.sampling_output().unwrap().borrow_mut();
-
- let sampling_method = state.sampling_method().unwrap();
- let seeds_offset =
- seeds.buffer_offset() + sampling_start * std::mem::size_of::<u64>();
-
- let (bitmask_buffer, bitmask_offset) =
- state.token_bitmask().map_or((None, 0usize), |cell| {
- let bitmask = cell.borrow_mut();
- let bitmask_row_len = {
- use crate::Array;
- bitmask.shape()[1]
- };
- let bitmask_offset = bitmask.buffer_offset()
- + sampling_start
- * bitmask_row_len
- * std::mem::size_of::<u32>();
- (Some(bitmask.mtl_buffer_cloned()), bitmask_offset)
- });
- if let Err(e) = self.kernel.encode_with_encoder(
- unsafe { &logits.mtl_buffer() },
- unsafe { Some(&seeds.mtl_buffer()) },
- seeds_offset,
- bitmask_buffer.as_deref(),
- bitmask_offset,
- unsafe { &output_buffer_ref.mtl_buffer() },
- sampling_method,
- batch_size,
- vocab_size,
- encoder,
- ) {
- panic!("Sampling encoding failed: {:?}", e);
- }
- }
-}
diff --git a/crates/uzu/src/backends/metal/encodable_block/short_conv_mixer.rs b/crates/uzu/src/backends/metal/encodable_block/short_conv_mixer.rs
index bfb33ae3a..65c524b26 100644
--- a/crates/uzu/src/backends/metal/encodable_block/short_conv_mixer.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/short_conv_mixer.rs
@@ -1,16 +1,20 @@
use std::rc::Rc;
-use super::{EncodableBlock, EncodingParameters, transformer_layer};
+use super::{EncodableBlock, EncodingParameters, Metal, transformer_layer};
use crate::{
DataType,
- backends::metal::{
- KernelDataType, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder, MTLContext,
- MTLDeviceExt, MTLResource, MTLResourceOptions, MetalArray, ProtocolObject,
- compilation_parameters::CompilationConfig,
- forward_pass::{ArrayId, ForwardPassState},
- kernel::short_conv::{
- ShortConvDecodeArguments, ShortConvKernel, ShortConvPackArguments,
- ShortConvPrefillArguments,
+ backends::{
+ common::Context,
+ metal::{
+ KernelDataType, MTLCommandBuffer, MTLCommandEncoder,
+ MTLComputeCommandEncoder, MTLContext, MetalArray, ProtocolObject,
+ Retained,
+ compilation_parameters::CompilationConfig,
+ forward_pass::{ArrayId, ForwardPassState},
+ kernel::short_conv::{
+ ShortConvDecodeArguments, ShortConvKernel,
+ ShortConvPackArguments, ShortConvPrefillArguments,
+ },
},
},
config::{DecoderLayerType, ShortConvConfig},
@@ -21,17 +25,17 @@ pub(crate) struct ShortConvMixer {
layer_index: usize,
config: ShortConvConfig,
model_dim: usize,
- in_projection: Box<dyn EncodableBlock>,
- out_projection: Box<dyn EncodableBlock>,
+ in_projection: Box<dyn EncodableBlock<Metal>>,
+ out_projection: Box<dyn EncodableBlock<Metal>>,
short_conv_kernel: ShortConvKernel,
conv_weight: MetalArray,
conv_bias: Option<MetalArray>,
}
fn resolve_subtree<'tree>(
- tree: &'tree ParameterTree<Rc<MTLContext>>,
+ tree: &'tree ParameterTree<MTLContext>,
candidates: &[&str],
-) -> ParameterTree<'tree, Rc<MTLContext>> {
+) -> ParameterTree<'tree, MTLContext> {
for candidate in candidates {
if let Ok(subtree) = tree.subtree(candidate) {
return subtree;
@@ -49,7 +53,7 @@ impl ShortConvMixer {
_compilation_config: Rc<CompilationConfig>,
layer_index: usize,
model_dim: usize,
- decoder_layer_loader: &ParameterTree<Rc<MTLContext>>,
+ decoder_layer_loader: &ParameterTree<MTLContext>,
) -> Self {
if !matches!(layer_type, DecoderLayerType::ShortConv { .. }) {
panic!(
@@ -117,7 +121,7 @@ impl ShortConvMixer {
fn encode_pipeline(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
let active_suffix_length = state.active_suffix_length();
@@ -184,15 +188,15 @@ impl ShortConvMixer {
let conv_state = arrays[1].borrow_mut();
let out = arrays[2].borrow_mut();
- let in_proj_buf = in_proj.mtl_buffer_cloned();
- let state_buf = conv_state.mtl_buffer_cloned();
- let out_buf = out.mtl_buffer_cloned();
+ let in_proj_buf = in_proj.buffer().clone();
+ let state_buf = conv_state.buffer().clone();
+ let out_buf = out.buffer().clone();
let conv_weight = self.conv_weight.clone();
- let weight_buf = conv_weight.mtl_buffer_cloned();
+ let weight_buf = conv_weight.buffer().clone();
let bias_buf = self.conv_bias.as_ref().map(|b| {
let b = b.clone();
- b.mtl_buffer_cloned()
+ b.buffer().clone()
});
let kernel_size = self.config.kernel_size;
@@ -204,9 +208,9 @@ impl ShortConvMixer {
let element_size = data_type.size_in_bytes();
let padded_rows = state_stride + suffix_length;
let padded_size = padded_rows * self.model_dim * element_size;
- let device = in_proj_buf.device();
- let padded_buf = device
- .new_buffer(padded_size, MTLResourceOptions::STORAGE_MODE_PRIVATE)
+ let padded_buf = state
+ .mtl_context()
+ .create_buffer(padded_size)
.expect("Failed to create padded buffer");
self.short_conv_kernel
.encode_pack(
@@ -258,15 +262,15 @@ impl ShortConvMixer {
let conv_state = arrays[1].borrow_mut();
let out = arrays[2].borrow_mut();
- let in_proj_buf = in_proj.mtl_buffer_cloned();
- let state_buf = conv_state.mtl_buffer_cloned();
- let out_buf = out.mtl_buffer_cloned();
+ let in_proj_buf = in_proj.buffer().clone();
+ let state_buf = conv_state.buffer().clone();
+ let out_buf = out.buffer().clone();
let conv_weight = self.conv_weight.clone();
- let weight_buf = conv_weight.mtl_buffer_cloned();
+ let weight_buf = conv_weight.buffer().clone();
let bias_buf = self.conv_bias.as_ref().map(|b| {
let b = b.clone();
- b.mtl_buffer_cloned()
+ b.buffer().clone()
});
let kernel_size = self.config.kernel_size;
@@ -293,11 +297,11 @@ impl ShortConvMixer {
}
}
-impl EncodableBlock for ShortConvMixer {
+impl EncodableBlock<Metal> for ShortConvMixer {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
if self.supports_shared_encoder() {
diff --git a/crates/uzu/src/backends/metal/encodable_block/tensor_add_swap.rs b/crates/uzu/src/backends/metal/encodable_block/tensor_add_swap.rs
index c9588b8ec..524876baa 100644
--- a/crates/uzu/src/backends/metal/encodable_block/tensor_add_swap.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/tensor_add_swap.rs
@@ -1,21 +1,22 @@
//! Tensor add-swap encodable.
-use crate::backends::metal::{ProtocolObject,
- MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
+use crate::backends::{
+ common::kernel::TensorAddSwapKernel,
+ metal::{
+ MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
+ ProtocolObject, Retained,
+ },
};
-use super::{EncodableBlock, EncodingParameters};
-use crate::{
- Array,
- backends::metal::{
- MTLContext, MTLError,
- forward_pass::{ArrayId, ForwardPassState},
- kernel::TensorAddSwapKernel,
- },
+use super::{EncodableBlock, EncodingParameters, Metal};
+use crate::backends::metal::{
+ MTLContext, MTLError,
+ forward_pass::{ArrayId, ForwardPassState},
+ kernel::dsl::TensorAddSwapMetalKernel,
};
pub struct TensorAddSwap {
- kernel: TensorAddSwapKernel,
+ kernel: TensorAddSwapMetalKernel,
argument_arrays: Box<[ArrayId]>,
}
@@ -25,7 +26,7 @@ impl TensorAddSwap {
data_type: crate::backends::metal::KernelDataType,
argument_arrays: Box<[ArrayId]>,
) -> Result<Self, MTLError> {
- let kernel = TensorAddSwapKernel::new(context, data_type)?;
+ let kernel = TensorAddSwapMetalKernel::new(context, data_type.into())?;
Ok(Self {
kernel,
argument_arrays,
@@ -33,14 +34,15 @@ impl TensorAddSwap {
}
}
-impl EncodableBlock for TensorAddSwap {
+impl EncodableBlock<Metal> for TensorAddSwap {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
- let encoder = command_buffer.new_compute_command_encoder()
+ let encoder = command_buffer
+ .new_compute_command_encoder()
.expect("Failed to create compute command encoder");
self.encode_with_shared_encoder(state, &encoder, parameters);
encoder.end_encoding();
@@ -66,15 +68,15 @@ impl EncodableBlock for TensorAddSwap {
let length = arrays[0].borrow().num_elements();
- let mut skip_array = arrays[0].borrow_mut();
- let mut main_array = arrays[1].borrow_mut();
- let skip_mtl_buffer = unsafe { skip_array.mtl_buffer() };
- let main_mtl_buffer = unsafe { main_array.mtl_buffer() };
+ let skip_array = arrays[0].borrow_mut();
+ let main_array = arrays[1].borrow_mut();
+ let skip_mtl_buffer = skip_array.buffer();
+ let main_mtl_buffer = main_array.buffer();
- self.kernel.encode_with_encoder(
+ self.kernel.encode(
&skip_mtl_buffer,
&main_mtl_buffer,
- length,
+ length as u32,
encoder,
);
}
diff --git a/crates/uzu/src/backends/metal/encodable_block/tensor_copy.rs b/crates/uzu/src/backends/metal/encodable_block/tensor_copy.rs
index bb6c07a51..301dcfec7 100644
--- a/crates/uzu/src/backends/metal/encodable_block/tensor_copy.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/tensor_copy.rs
@@ -1,21 +1,22 @@
//! Tensor copy encodable.
-use crate::backends::metal::{ProtocolObject,
- MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
+use crate::backends::{
+ common::kernel::TensorCopyKernel,
+ metal::{
+ MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
+ ProtocolObject, Retained,
+ },
};
-use super::{EncodableBlock, EncodingParameters};
-use crate::{
- Array,
- backends::metal::{
- MTLContext, MTLError,
- forward_pass::{ArrayId, ForwardPassState},
- kernel::TensorCopyKernel,
- },
+use super::{EncodableBlock, EncodingParameters, Metal};
+use crate::backends::metal::{
+ MTLContext, MTLError,
+ forward_pass::{ArrayId, ForwardPassState},
+ kernel::dsl::TensorCopyMetalKernel,
};
pub struct TensorCopy {
- kernel: TensorCopyKernel,
+ kernel: TensorCopyMetalKernel,
argument_arrays: Box<[ArrayId]>,
}
@@ -25,7 +26,7 @@ impl TensorCopy {
data_type: crate::backends::metal::KernelDataType,
argument_arrays: Box<[ArrayId]>,
) -> Result<Self, MTLError> {
- let kernel = TensorCopyKernel::new(context, data_type)?;
+ let kernel = TensorCopyMetalKernel::new(context, data_type.into())?;
Ok(Self {
kernel,
argument_arrays,
@@ -33,14 +34,15 @@ impl TensorCopy {
}
}
-impl EncodableBlock for TensorCopy {
+impl EncodableBlock<Metal> for TensorCopy {
fn encode(
&self,
state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
parameters: &EncodingParameters,
) {
- let encoder = command_buffer.new_compute_command_encoder()
+ let encoder = command_buffer
+ .new_compute_command_encoder()
.expect("Failed to create compute command encoder");
self.encode_with_shared_encoder(state, &encoder, parameters);
encoder.end_encoding();
@@ -66,15 +68,15 @@ impl EncodableBlock for TensorCopy {
let length = arrays[0].borrow().num_elements();
- let mut source_array = arrays[0].borrow_mut();
- let mut destination_array = arrays[1].borrow_mut();
- let source_mtl_buffer = unsafe { source_array.mtl_buffer() };
- let destination_mtl_buffer = unsafe { destination_array.mtl_buffer() };
+ let source_array = arrays[0].borrow_mut();
+ let destination_array = arrays[1].borrow_mut();
+ let source_mtl_buffer = source_array.buffer();
+ let destination_mtl_buffer = destination_array.buffer();
- self.kernel.encode_with_encoder(
+ self.kernel.encode(
&source_mtl_buffer,
&destination_mtl_buffer,
- length,
+ length as u32,
encoder,
);
}
diff --git a/crates/uzu/src/backends/metal/encodable_block/transformer_layer.rs b/crates/uzu/src/backends/metal/encodable_block/transformer_layer.rs
index ab5650ed6..9c32dda6d 100644
--- a/crates/uzu/src/backends/metal/encodable_block/transformer_layer.rs
+++ b/crates/uzu/src/backends/metal/encodable_block/transformer_layer.rs
@@ -1,5 +1,6 @@
use std::rc::Rc;
+use super::Metal;
use super::{
EncodableBlock, FullPrecisionEmbeddingLookup,
FullPrecisionEmbeddingReadout, FullPrecisionLinear, MlpBlock,
@@ -23,10 +24,10 @@ pub fn linear_block<const N: usize>(
input_dimension: usize,
output_dimensions: [usize; N],
context: &MTLContext,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
input_array_id: ArrayId,
output_array_id: ArrayId,
-) -> Result<Box<dyn EncodableBlock>, MTLError> {
+) -> Result<Box<dyn EncodableBlock<Metal>>, MTLError> {
let output_dimension_sum: usize = output_dimensions.iter().sum();
match config {
LinearConfig::Quantized(quantization_config)
@@ -70,8 +71,8 @@ pub fn mlp_block(
model_dimension: usize,
hidden_dimension: usize,
context: &MTLContext,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
-) -> Result<Box<dyn EncodableBlock>, MTLError> {
+ parameter_tree: &ParameterTree<MTLContext>,
+) -> Result<Box<dyn EncodableBlock<Metal>>, MTLError> {
if let crate::config::MLPConfig::Dense(dense_config) = config {
let data_type: DataType =
dense_config.linear_config.activation_precision().into();
@@ -143,8 +144,8 @@ pub fn mlp_fused_block(
model_dimension: usize,
hidden_dimension: usize,
context: Rc<MTLContext>,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
-) -> Result<Box<dyn EncodableBlock>, MTLError> {
+ parameter_tree: &ParameterTree<MTLContext>,
+) -> Result<Box<dyn EncodableBlock<Metal>>, MTLError> {
if let crate::config::MLPConfig::Dense(dense_config) = config {
match &dense_config.linear_config {
LinearConfig::Quantized(quantization_config)
@@ -158,7 +159,7 @@ pub fn mlp_fused_block(
|error| MTLError::Generic(format!("{:?}", error)),
)?;
- let mut up_projection_weights =
+ let up_projection_weights =
up_projection_tree.leaf("weights").map_err(|error| {
MTLError::Generic(format!(
"Failed to load up weights: {:?}",
@@ -166,9 +167,9 @@ pub fn mlp_fused_block(
))
})?;
let up_projection_weights_buffer =
- unsafe { up_projection_weights.mtl_buffer().into() };
+ up_projection_weights.buffer().into();
- let mut up_projection_scales =
+ let up_projection_scales =
up_projection_tree.leaf("scales").map_err(|error| {
MTLError::Generic(format!(
"Failed to load up scales: {:?}",
@@ -176,24 +177,18 @@ pub fn mlp_fused_block(
))
})?;
let up_projection_scales_buffer =
- unsafe { up_projection_scales.mtl_buffer().into() };
+ up_projection_scales.buffer().into();
// Load zero_points or biases depending on quantization type
let (
up_projection_zero_points_or_biases_buffer,
quantization_type,
- ) = if let Ok(mut biases) = up_projection_tree.leaf("biases") {
- (
- unsafe { biases.mtl_buffer().into() },
- QuantizationType::Mlx,
- )
- } else if let Ok(mut zero_points) =
+ ) = if let Ok(biases) = up_projection_tree.leaf("biases") {
+ (biases.buffer().into(), QuantizationType::Mlx)
+ } else if let Ok(zero_points) =
up_projection_tree.leaf("zero_points")
{
- (
- unsafe { zero_points.mtl_buffer().into() },
- QuantizationType::ZeroPoint,
- )
+ (zero_points.buffer().into(), QuantizationType::ZeroPoint)
} else {
return Err(MTLError::Generic(
"Missing zero_points or biases for quantized up_projection"
@@ -243,7 +238,7 @@ pub fn mlp_fused_block(
|error| MTLError::Generic(format!("{:?}", error)),
)?;
- let mut up_projection_weights =
+ let up_projection_weights =
up_projection_tree.leaf("weights").map_err(|error| {
MTLError::Generic(format!(
"Failed to load up weights: {:?}",
@@ -251,7 +246,7 @@ pub fn mlp_fused_block(
))
})?;
let up_projection_weights_buffer =
- unsafe { up_projection_weights.mtl_buffer().into() };
+ up_projection_weights.buffer().into();
// Create down projection as separate linear
let down_projection = FullPrecisionLinear::new(
@@ -309,8 +304,8 @@ pub fn mlp_fused_block(
pub fn embed_block(
config: &DecoderConfig,
context: &MTLContext,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
-) -> Box<dyn EncodableBlock> {
+ parameter_tree: &ParameterTree<MTLContext>,
+) -> Box<dyn EncodableBlock<Metal>> {
match &config.embedding_config {
EmbeddingConfig::Tied {
common,
@@ -465,8 +460,8 @@ pub fn embed_block(
pub fn readout_block(
config: &DecoderConfig,
context: &MTLContext,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
-) -> Box<dyn EncodableBlock> {
+ parameter_tree: &ParameterTree<MTLContext>,
+) -> Box<dyn EncodableBlock<Metal>> {
match &config.embedding_config {
EmbeddingConfig::Tied {
precision,
diff --git a/crates/uzu/src/backends/metal/error.rs b/crates/uzu/src/backends/metal/error.rs
index 384128852..34164f418 100644
--- a/crates/uzu/src/backends/metal/error.rs
+++ b/crates/uzu/src/backends/metal/error.rs
@@ -52,6 +52,8 @@ pub enum ContextError {
// Device errors
#[derive(Debug, Error)]
pub enum DeviceError {
+ #[error("No default Metal device found")]
+ NoDefaultDevice,
#[error("Failed to create argument encoder")]
ArgumentEncoderCreationFailed,
#[error("Failed to create buffer")]
diff --git a/crates/uzu/src/backends/metal/forward_pass/cache_layers.rs b/crates/uzu/src/backends/metal/forward_pass/cache_layers.rs
index 7cb3cfdfd..6a5a2096e 100644
--- a/crates/uzu/src/backends/metal/forward_pass/cache_layers.rs
+++ b/crates/uzu/src/backends/metal/forward_pass/cache_layers.rs
@@ -11,10 +11,9 @@ use super::{
ssm_layer::SSMLayer,
};
use crate::{
- DeviceContext,
- array::Array,
+ array::ArrayContextExt,
backends::metal::{
- MTLCommandBuffer, ProtocolObject, kernel::KVCacheUpdate,
+ MTLCommandBuffer, ProtocolObject, Retained, kernel::KVCacheUpdate,
},
config::DecoderLayerType,
};
@@ -136,17 +135,17 @@ impl CacheLayers {
CacheLayer::Transformer(KVCacheLayer {
state: state.clone(),
- keys: RefCell::new(context.array(
+ keys: RefCell::new(context.create_array(
&shape,
model_shape.kv_cache_data_type(),
- format!(
+ &format!(
"{ARRAY_TRANSFORMER_KEYS_LABEL}_{layer_index}"
),
)),
- values: RefCell::new(context.array(
+ values: RefCell::new(context.create_array(
&shape,
model_shape.kv_cache_data_type(),
- format!(
+ &format!(
"{ARRAY_TRANSFORMER_VALUES_LABEL}_{layer_index}"
),
)),
@@ -177,17 +176,17 @@ impl CacheLayers {
let dtype = model_shape.activation_data_type();
CacheLayer::StateSpace(SSMLayer {
- conv_state: RefCell::new(context.array(
+ conv_state: RefCell::new(context.create_array(
&conv_shape,
dtype,
- format!(
+ &format!(
"{ARRAY_STATE_SPACE_CONV_STATE_LABEL}_{layer_index}"
),
)),
- ssm_state: RefCell::new(context.array(
+ ssm_state: RefCell::new(context.create_array(
&ssm_shape,
dtype,
- format!(
+ &format!(
"{ARRAY_STATE_SPACE_SSM_STATE_LABEL}_{layer_index}"
),
)),
@@ -203,10 +202,10 @@ impl CacheLayers {
let dtype = model_shape.activation_data_type();
CacheLayer::ShortConv(ShortConvLayer {
- conv_state: RefCell::new(context.array(
+ conv_state: RefCell::new(context.create_array(
&conv_shape,
dtype,
- format!(
+ &format!(
"{ARRAY_SHORT_CONV_CONV_STATE_LABEL}_{layer_index}"
),
)),
@@ -261,7 +260,6 @@ impl CacheLayers {
dst: &mut HashMap<Option<usize>, MetalArray>,
suffix_token_positions: &[usize],
suffix_length: usize,
- context: &MTLContext,
external_bias_fn: Option<&dyn Fn(usize, usize) -> bool>,
) {
for layer in self.data.iter() {
@@ -271,7 +269,6 @@ impl CacheLayers {
array,
suffix_token_positions,
suffix_length,
- context,
external_bias_fn,
);
}
@@ -284,7 +281,7 @@ impl CacheLayers {
dst: &HashMap<Option<usize>, RefCell<MetalArray>>,
suffix_token_positions: &[usize],
suffix_length: usize,
- context: &MTLContext,
+ _context: &MTLContext,
) {
for layer in self.data.iter() {
if let CacheLayer::Transformer(layer) = layer {
@@ -293,7 +290,6 @@ impl CacheLayers {
&mut cell.borrow_mut(),
suffix_token_positions,
suffix_length,
- context,
None,
);
}
@@ -305,7 +301,7 @@ impl CacheLayers {
&mut self,
accepted_suffix_indices: &[usize],
suffix_start: Option<usize>,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
kv_cache_update: &KVCacheUpdate,
) {
for layer in self.data.iter_mut() {
@@ -417,15 +413,17 @@ impl CacheLayers {
}
let new_shape = [num_groups, new_total_len, head_dim];
- let mut new_keys = context.array(
+ let mut new_keys = context.create_array(
&new_shape,
dtype,
- format!("{ARRAY_TRANSFORMER_KEYS_LABEL}_{layer_index}"),
+ &format!(
+ "{ARRAY_TRANSFORMER_KEYS_LABEL}_{layer_index}"
+ ),
);
- let mut new_values = context.array(
+ let mut new_values = context.create_array(
&new_shape,
dtype,
- format!(
+ &format!(
"{ARRAY_TRANSFORMER_VALUES_LABEL}_{layer_index}"
),
);
@@ -454,10 +452,10 @@ impl CacheLayers {
CacheLayer::StateSpace(layer) => {
let conv_shape = layer.conv_state.borrow().shape().to_vec();
let conv_dtype = layer.conv_state.borrow().data_type();
- let mut new_conv = context.array(
+ let mut new_conv = context.create_array(
&conv_shape,
conv_dtype,
- format!(
+ &format!(
"{ARRAY_STATE_SPACE_CONV_STATE_LABEL}_{layer_index}"
),
);
@@ -468,10 +466,10 @@ impl CacheLayers {
let ssm_shape = layer.ssm_state.borrow().shape().to_vec();
let ssm_dtype = layer.ssm_state.borrow().data_type();
- let mut new_ssm = context.array(
+ let mut new_ssm = context.create_array(
&ssm_shape,
ssm_dtype,
- format!(
+ &format!(
"{ARRAY_STATE_SPACE_SSM_STATE_LABEL}_{layer_index}"
),
);
@@ -488,10 +486,10 @@ impl CacheLayers {
CacheLayer::ShortConv(layer) => {
let conv_shape = layer.conv_state.borrow().shape().to_vec();
let conv_dtype = layer.conv_state.borrow().data_type();
- let mut new_conv = context.array(
+ let mut new_conv = context.create_array(
&conv_shape,
conv_dtype,
- format!(
+ &format!(
"{ARRAY_SHORT_CONV_CONV_STATE_LABEL}_{layer_index}"
),
);
diff --git a/crates/uzu/src/backends/metal/forward_pass/encoder_resolver.rs b/crates/uzu/src/backends/metal/forward_pass/encoder_resolver.rs
index 546507bc5..90854cadb 100644
--- a/crates/uzu/src/backends/metal/forward_pass/encoder_resolver.rs
+++ b/crates/uzu/src/backends/metal/forward_pass/encoder_resolver.rs
@@ -1,17 +1,18 @@
use super::ForwardPassState;
use crate::backends::metal::{
- MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder, ProtocolObject, Retained,
+ MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder, Metal,
+ ProtocolObject, Retained,
encodable_block::{EncodableBlock, EncodingParameters},
};
pub struct EncoderResolver<'a> {
- command_buffer: &'a ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &'a Retained<ProtocolObject<dyn MTLCommandBuffer>>,
encoder: Option<Retained<ProtocolObject<dyn MTLComputeCommandEncoder>>>,
}
impl<'a> EncoderResolver<'a> {
pub fn new(
- command_buffer: &'a ProtocolObject<dyn MTLCommandBuffer>
+ command_buffer: &'a Retained<ProtocolObject<dyn MTLCommandBuffer>>
) -> Self {
Self {
command_buffer,
@@ -21,7 +22,7 @@ impl<'a> EncoderResolver<'a> {
pub fn encode(
&mut self,
- block: &dyn EncodableBlock,
+ block: &dyn EncodableBlock<Metal>,
state: &mut ForwardPassState,
parameters: &EncodingParameters,
) {
diff --git a/crates/uzu/src/backends/metal/forward_pass/kv_cache_layer.rs b/crates/uzu/src/backends/metal/forward_pass/kv_cache_layer.rs
index 70408847b..9c4e5dcf9 100644
--- a/crates/uzu/src/backends/metal/forward_pass/kv_cache_layer.rs
+++ b/crates/uzu/src/backends/metal/forward_pass/kv_cache_layer.rs
@@ -2,12 +2,12 @@ use std::cell::RefCell;
use super::super::{MTLContext, MetalArray};
use crate::{
- DeviceContext,
- array::Array,
+ array::ArrayContextExt,
backends::metal::{
- MTLCommandBuffer, ProtocolObject,
+ MTLCommandBuffer, ProtocolObject, Retained,
kernel::{KVCacheUpdate, kv_cache_update::KVLayerData},
},
+ utils::attention::fill_attention_bias,
};
pub type ArrayCell = RefCell<MetalArray>;
@@ -115,12 +115,10 @@ impl KVCacheLayer {
dst: &mut MetalArray,
suffix_token_positions: &[usize],
suffix_length: usize,
- context: &MTLContext,
external_bias_fn: Option<&dyn Fn(usize, usize) -> bool>,
) {
let prefix_segment_length = self.prefix_segment_length();
-
- context.fill_attention_bias(
+ fill_attention_bias(
dst,
suffix_length,
prefix_segment_length,
@@ -128,12 +126,11 @@ impl KVCacheLayer {
if let Some(bias_fn) = external_bias_fn {
bias_fn(row_index, column_index)
} else {
- let result = self.bias_should_be_neg_inf(
+ self.bias_should_be_neg_inf(
row_index,
column_index,
suffix_token_positions,
- );
- result
+ )
}
},
);
@@ -184,7 +181,7 @@ impl KVCacheLayer {
&mut self,
accepted_suffix_indices: &[usize],
suffix_start: Option<usize>,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
kv_cache_update: &KVCacheUpdate,
) {
match &mut self.state {
@@ -259,24 +256,8 @@ impl KVCacheLayer {
return;
}
- let key_buffer = {
- let mut k = self.keys.borrow_mut();
- unsafe {
- objc2::rc::Retained::retain(
- std::ptr::from_ref(&*k.mtl_buffer()) as *mut _,
- )
- .unwrap()
- }
- };
- let value_buffer = {
- let mut v = self.values.borrow_mut();
- unsafe {
- objc2::rc::Retained::retain(
- std::ptr::from_ref(&*v.mtl_buffer()) as *mut _,
- )
- .unwrap()
- }
- };
+ let key_buffer = self.keys.borrow().buffer().clone();
+ let value_buffer = self.values.borrow().buffer().clone();
let k_shape = self.keys.borrow().shape().to_vec();
let v_shape = self.values.borrow().shape().to_vec();
@@ -288,15 +269,11 @@ impl KVCacheLayer {
value_shape: [v_shape[0], v_shape[1], v_shape[2]],
};
- let cmd_buf = unsafe {
- objc2::rc::Retained::retain(command_buffer as *const _ as *mut _)
- .unwrap()
- };
let _ = kv_cache_update.encode(
&[layer_data],
source_indices,
destination_indices,
- &cmd_buf,
+ command_buffer,
);
}
@@ -402,15 +379,15 @@ impl KVCacheLayer {
let dtype = keys.data_type();
let slice_shape = [num_groups, len, head_dim];
- let mut slice_keys = context.array(
+ let mut slice_keys = context.create_array(
&slice_shape,
dtype,
- String::from("kv_cache_layer_slice_keys"),
+ "kv_cache_layer_slice_keys",
);
- let mut slice_values = context.array(
+ let mut slice_values = context.create_array(
&slice_shape,
dtype,
- String::from("kv_cache_layer_slice_values"),
+ "kv_cache_layer_slice_values",
);
let slots: Vec<usize> = (range.start..range.end)
diff --git a/crates/uzu/src/backends/metal/forward_pass/scratch_buffers.rs b/crates/uzu/src/backends/metal/forward_pass/scratch_buffers.rs
index 170206407..ed7b8cc11 100644
--- a/crates/uzu/src/backends/metal/forward_pass/scratch_buffers.rs
+++ b/crates/uzu/src/backends/metal/forward_pass/scratch_buffers.rs
@@ -2,95 +2,110 @@ use std::{cell::RefCell, collections::HashMap};
use super::model_shape::ModelShape;
use crate::{
- DataType, DeviceContext,
+ DataType,
+ array::ArrayContextExt,
+ backends::common::Context,
config::{DecoderConfig, MLPConfig},
};
-type Array<Context> = RefCell<<Context as DeviceContext>::DeviceArray>;
+type Array<C> = RefCell<crate::array::Array<<C as Context>::Backend>>;
-pub struct ScratchBuffers<Context: DeviceContext> {
+pub struct ScratchBuffers<C: Context> {
// 1-D
- pub token_ids: Array<Context>,
- pub token_positions: Array<Context>,
- pub token_bitmask: Array<Context>,
- pub token_seeds: Array<Context>,
- pub sampling_output: Array<Context>,
+ pub token_ids: Array<C>,
+ pub token_positions: Array<C>,
+ pub token_bitmask: Array<C>,
+ pub token_seeds: Array<C>,
+ pub sampling_output: Array<C>,
// 2-D
- pub attention_window_size_to_bias: HashMap<Option<usize>, Array<Context>>,
- pub logits: Array<Context>,
- pub main: Array<Context>,
- pub shortcut: Array<Context>,
- pub qkv: Array<Context>,
- pub attention_output: Array<Context>,
- pub mlp_fused_up: Array<Context>,
- pub mlp_hidden: Array<Context>,
- pub ssm_inproj: Option<Array<Context>>,
- pub ssm_packed: Option<Array<Context>>,
- pub ssm_conv_padded: Option<Array<Context>>,
- pub short_conv_padded: Option<Array<Context>>,
- pub ssm_x: Option<Array<Context>>,
- pub ssm_b: Option<Array<Context>>,
- pub ssm_c: Option<Array<Context>>,
- pub ssm_dt: Option<Array<Context>>,
- pub ssm_z: Option<Array<Context>>,
+ pub attention_window_size_to_bias: HashMap<Option<usize>, Array<C>>,
+ pub logits: Array<C>,
+ pub main: Array<C>,
+ pub shortcut: Array<C>,
+ pub qkv: Array<C>,
+ pub attention_output: Array<C>,
+ pub mlp_fused_up: Array<C>,
+ pub mlp_hidden: Array<C>,
+ pub ssm_inproj: Option<Array<C>>,
+ pub ssm_packed: Option<Array<C>>,
+ pub ssm_conv_padded: Option<Array<C>>,
+ pub short_conv_padded: Option<Array<C>>,
+ pub ssm_x: Option<Array<C>>,
+ pub ssm_b: Option<Array<C>>,
+ pub ssm_c: Option<Array<C>>,
+ pub ssm_dt: Option<Array<C>>,
+ pub ssm_z: Option<Array<C>>,
// 3-D
- pub rotated_queries: Array<Context>,
- pub rotated_keys: Array<Context>,
- pub extracted_values: Array<Context>,
+ pub rotated_queries: Array<C>,
+ pub rotated_keys: Array<C>,
+ pub extracted_values: Array<C>,
// 2-pass attention intermediate buffers
- pub attention_partials: Array<Context>, // [num_heads * max_suffix_len * total_blocks_count * head_dim]
- pub attention_sums: Array<Context>, // [num_heads * max_suffix_len * total_blocks_count]
- pub attention_maxs: Array<Context>, // [num_heads * max_suffix_len * total_blocks_count]
+ pub attention_partials: Array<C>, // [num_heads * max_suffix_len * total_blocks_count * head_dim]
+ pub attention_sums: Array<C>, // [num_heads * max_suffix_len * total_blocks_count]
+ pub attention_maxs: Array<C>, // [num_heads * max_suffix_len * total_blocks_count]
- pub moe_topk_ids: Option<Array<Context>>,
- pub moe_topk_probs: Option<Array<Context>>,
- pub moe_offsets: Option<Array<Context>>,
- pub moe_sumk: Option<Array<Context>>,
- pub moe_bucketed_token_ids: Option<Array<Context>>,
- pub moe_bucketed_probs: Option<Array<Context>>,
- pub moe_x_perm: Option<Array<Context>>,
- pub moe_tok2row: Option<Array<Context>>,
- pub moe_y_partial: Option<Array<Context>>,
- pub moe_hidden: Option<Array<Context>>,
- pub moe_two_pass_row_expert_map: Option<Array<Context>>,
- pub moe_tile_counts: Option<Array<Context>>,
- pub moe_tile_offsets: Option<Array<Context>>,
- pub moe_tile_map: Option<Array<Context>>,
- pub moe_total_tiles: Option<Array<Context>>,
- pub moe_dispatch_args: Option<Array<Context>>,
- pub moe_scatter_partials: Option<Array<Context>>,
- pub moe_scatter_block_bases: Option<Array<Context>>,
- pub moe_block_alloc: Option<Array<Context>>,
+ pub moe_topk_ids: Option<Array<C>>,
+ pub moe_topk_probs: Option<Array<C>>,
+ pub moe_offsets: Option<Array<C>>,
+ pub moe_sumk: Option<Array<C>>,
+ pub moe_bucketed_token_ids: Option<Array<C>>,
+ pub moe_bucketed_probs: Option<Array<C>>,
+ pub moe_x_perm: Option<Array<C>>,
+ pub moe_tok2row: Option<Array<C>>,
+ pub moe_y_partial: Option<Array<C>>,
+ pub moe_hidden: Option<Array<C>>,
+ pub moe_two_pass_row_expert_map: Option<Array<C>>,
+ pub moe_tile_counts: Option<Array<C>>,
+ pub moe_tile_offsets: Option<Array<C>>,
+ pub moe_tile_map: Option<Array<C>>,
+ pub moe_total_tiles: Option<Array<C>>,
+ pub moe_dispatch_args: Option<Array<C>>,
+ pub moe_scatter_partials: Option<Array<C>>,
+ pub moe_scatter_block_bases: Option<Array<C>>,
+ pub moe_block_alloc: Option<Array<C>>,
}
-impl<Context: DeviceContext> ScratchBuffers<Context> {
+impl<C: Context> ScratchBuffers<C> {
// TODO: use device arrays instead of MTLBuffers
/// Allocate the buffers with `StorageModeShared` so that they are CPU-accessible as well.
pub fn new(
- context: &Context,
+ context: &C,
decoder_config: &DecoderConfig,
model_shape: &ModelShape,
max_prefix_len: usize,
max_suffix_len: usize,
) -> Self {
// Helper closure for allocation
- let alloc = |shape: &[usize],
- dtype: DataType,
- label: &str|
- -> Array<Context> {
- let array =
- context.array(shape, dtype, format!("scratch_buffers_{label}"));
- RefCell::new(array)
- };
+ let alloc =
+ |shape: &[usize], dtype: DataType, label: &str| -> Array<C> {
+ let array = context.create_array(
+ shape,
+ dtype,
+ &format!("scratch_buffers_{label}"),
+ );
+ RefCell::new(array)
+ };
let act_ty = model_shape.activation_data_type();
let partials_shape =
model_shape.attention_partials_shape(max_suffix_len);
let sums_maxs_shape = model_shape.attention_sums_shape(max_suffix_len);
+ let moe = match &decoder_config.layer_config.mlp_config {
+ MLPConfig::MixtureOfExperts(moe) => Some(moe),
+ _ => None,
+ };
+ let moe_max_routed =
+ moe.map(|moe| max_suffix_len * moe.num_experts_per_token);
+ let moe_mixture_size = moe.map(|moe| moe.mixture_size);
+ let moe_scatter_entries = moe.map(|moe| {
+ let num_blocks = ((max_suffix_len + 255) / 256).max(1);
+ let num_tiles = ((moe.mixture_size + 512 - 1) / 512).max(1);
+ num_blocks * num_tiles * 512
+ });
Self {
// 1-D
@@ -145,7 +160,7 @@ impl<Context: DeviceContext> ScratchBuffers<Context> {
)
})
})
- .collect::<HashMap<Option<usize>, Array<Context>>>()
+ .collect::<HashMap<Option<usize>, Array<C>>>()
},
logits: alloc(
&model_shape.logits_shape(max_suffix_len),
@@ -230,184 +245,92 @@ impl<Context: DeviceContext> ScratchBuffers<Context> {
attention_sums: alloc(&sums_maxs_shape, act_ty, "attention_sums"),
attention_maxs: alloc(&sums_maxs_shape, act_ty, "attention_maxs"),
- moe_topk_ids: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let shape = model_shape.moe_topk_ids_shape(
- max_suffix_len,
- moe.num_experts_per_token,
- );
- Some(alloc(&shape, DataType::U32, "moe_topk_ids"))
- },
- _ => None,
- },
- moe_topk_probs: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let shape = model_shape.moe_topk_probs_shape(
- max_suffix_len,
- moe.num_experts_per_token,
- );
- Some(alloc(&shape, act_ty, "moe_topk_probs"))
- },
- _ => None,
- },
- moe_offsets: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let shape = model_shape.moe_offsets_shape(moe.mixture_size);
- Some(alloc(&shape, DataType::U32, "moe_offsets"))
- },
- _ => None,
- },
- moe_sumk: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(_) => {
- let shape = model_shape.moe_sumk_shape();
- Some(alloc(&shape, DataType::U32, "moe_sumk"))
- },
- _ => None,
- },
- moe_bucketed_token_ids: match &decoder_config
- .layer_config
- .mlp_config
- {
- MLPConfig::MixtureOfExperts(moe) => {
- let max_routed = max_suffix_len * moe.num_experts_per_token;
- let shape =
- model_shape.moe_bucketed_token_ids_shape(max_routed);
- Some(alloc(&shape, DataType::U32, "moe_bucketed_token_ids"))
- },
- _ => None,
- },
- moe_bucketed_probs: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let max_routed = max_suffix_len * moe.num_experts_per_token;
- let shape =
- model_shape.moe_bucketed_probs_shape(max_routed);
- Some(alloc(&shape, act_ty, "moe_bucketed_probs"))
- },
- _ => None,
- },
- moe_x_perm: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let max_routed = max_suffix_len * moe.num_experts_per_token;
- let shape = model_shape.moe_x_perm_shape(max_routed);
- Some(alloc(&shape, DataType::F16, "moe_x_perm"))
- },
- _ => None,
- },
- moe_tok2row: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let shape = model_shape.moe_tok2row_shape(
- max_suffix_len,
- moe.num_experts_per_token,
- );
- Some(alloc(&shape, DataType::I32, "moe_tok2row"))
- },
- _ => None,
- },
- moe_y_partial: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let max_routed = max_suffix_len * moe.num_experts_per_token;
- let shape = model_shape.moe_y_partial_shape(max_routed);
- Some(alloc(&shape, DataType::F16, "moe_y_partial"))
- },
- _ => None,
- },
- moe_hidden: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let max_routed = max_suffix_len * moe.num_experts_per_token;
- let shape = model_shape.moe_hidden_shape(max_routed);
- Some(alloc(&shape, DataType::F32, "moe_hidden"))
- },
- _ => None,
- },
- moe_two_pass_row_expert_map: match &decoder_config
- .layer_config
- .mlp_config
- {
- MLPConfig::MixtureOfExperts(moe) => {
- let max_routed = max_suffix_len * moe.num_experts_per_token;
- Some(alloc(
- &[max_routed],
- DataType::U32,
- "moe_two_pass_row_expert_map",
- ))
- },
- _ => None,
- },
- moe_tile_counts: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let shape = model_shape.moe_counts_shape(moe.mixture_size);
- Some(alloc(&shape, DataType::U32, "moe_tile_counts"))
- },
- _ => None,
- },
- moe_tile_offsets: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let shape = model_shape.moe_offsets_shape(moe.mixture_size);
- Some(alloc(&shape, DataType::U32, "moe_tile_offsets"))
- },
- _ => None,
- },
- moe_tile_map: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let max_routed = max_suffix_len * moe.num_experts_per_token;
- let shape = model_shape.moe_tile_map_shape(max_routed);
- Some(alloc(&shape, DataType::U32, "moe_tile_map"))
- },
- _ => None,
- },
- moe_total_tiles: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(_) => {
- let shape = model_shape.moe_total_tiles_shape();
- Some(alloc(&shape, DataType::U32, "moe_total_tiles"))
- },
- _ => None,
- },
- moe_dispatch_args: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(_) => {
- let shape = model_shape.moe_dispatch_args_shape();
- Some(alloc(&shape, DataType::U32, "moe_dispatch_args"))
- },
- _ => None,
- },
- moe_scatter_partials: match &decoder_config.layer_config.mlp_config
- {
- MLPConfig::MixtureOfExperts(moe) => {
- let num_blocks = ((max_suffix_len + 255) / 256).max(1);
- let num_tiles = ((moe.mixture_size + 512 - 1) / 512).max(1);
- let entries = num_blocks * num_tiles * 512;
- Some(alloc(
- &[entries],
- DataType::U32,
- "moe_scatter_partials",
- ))
- },
- _ => None,
- },
- moe_scatter_block_bases: match &decoder_config
- .layer_config
- .mlp_config
- {
- MLPConfig::MixtureOfExperts(moe) => {
- let num_blocks = ((max_suffix_len + 255) / 256).max(1);
- let num_tiles = ((moe.mixture_size + 512 - 1) / 512).max(1);
- let entries = num_blocks * num_tiles * 512;
- Some(alloc(
- &[entries],
- DataType::U32,
- "moe_scatter_block_bases",
- ))
- },
- _ => None,
- },
- moe_block_alloc: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let num_blocks = ((max_suffix_len + 255) / 256).max(1);
- let num_tiles = ((moe.mixture_size + 512 - 1) / 512).max(1);
- let entries = num_blocks * num_tiles * 512;
- Some(alloc(&[entries], DataType::U32, "moe_block_alloc"))
- },
- _ => None,
- },
+ moe_topk_ids: moe.map(|moe| {
+ let shape = model_shape.moe_topk_ids_shape(
+ max_suffix_len,
+ moe.num_experts_per_token,
+ );
+ alloc(&shape, DataType::U32, "moe_topk_ids")
+ }),
+ moe_topk_probs: moe.map(|moe| {
+ let shape = model_shape.moe_topk_probs_shape(
+ max_suffix_len,
+ moe.num_experts_per_token,
+ );
+ alloc(&shape, act_ty, "moe_topk_probs")
+ }),
+ moe_offsets: moe_mixture_size.map(|mixture_size| {
+ let shape = model_shape.moe_offsets_shape(mixture_size);
+ alloc(&shape, DataType::U32, "moe_offsets")
+ }),
+ moe_sumk: moe.map(|_| {
+ let shape = model_shape.moe_sumk_shape();
+ alloc(&shape, DataType::U32, "moe_sumk")
+ }),
+ moe_bucketed_token_ids: moe_max_routed.map(|max_routed| {
+ let shape =
+ model_shape.moe_bucketed_token_ids_shape(max_routed);
+ alloc(&shape, DataType::U32, "moe_bucketed_token_ids")
+ }),
+ moe_bucketed_probs: moe_max_routed.map(|max_routed| {
+ let shape = model_shape.moe_bucketed_probs_shape(max_routed);
+ alloc(&shape, act_ty, "moe_bucketed_probs")
+ }),
+ moe_x_perm: moe_max_routed.map(|max_routed| {
+ let shape = model_shape.moe_x_perm_shape(max_routed);
+ alloc(&shape, DataType::F16, "moe_x_perm")
+ }),
+ moe_tok2row: moe.map(|moe| {
+ let shape = model_shape.moe_tok2row_shape(
+ max_suffix_len,
+ moe.num_experts_per_token,
+ );
+ alloc(&shape, DataType::I32, "moe_tok2row")
+ }),
+ moe_y_partial: moe_max_routed.map(|max_routed| {
+ let shape = model_shape.moe_y_partial_shape(max_routed);
+ alloc(&shape, DataType::F16, "moe_y_partial")
+ }),
+ moe_hidden: moe_max_routed.map(|max_routed| {
+ let shape = model_shape.moe_hidden_shape(max_routed);
+ alloc(&shape, DataType::F32, "moe_hidden")
+ }),
+ moe_two_pass_row_expert_map: moe_max_routed.map(|max_routed| {
+ alloc(
+ &[max_routed],
+ DataType::U32,
+ "moe_two_pass_row_expert_map",
+ )
+ }),
+ moe_tile_counts: moe_mixture_size.map(|mixture_size| {
+ let shape = model_shape.moe_counts_shape(mixture_size);
+ alloc(&shape, DataType::U32, "moe_tile_counts")
+ }),
+ moe_tile_offsets: moe_mixture_size.map(|mixture_size| {
+ let shape = model_shape.moe_offsets_shape(mixture_size);
+ alloc(&shape, DataType::U32, "moe_tile_offsets")
+ }),
+ moe_tile_map: moe_max_routed.map(|max_routed| {
+ let shape = model_shape.moe_tile_map_shape(max_routed);
+ alloc(&shape, DataType::U32, "moe_tile_map")
+ }),
+ moe_total_tiles: moe.map(|_| {
+ let shape = model_shape.moe_total_tiles_shape();
+ alloc(&shape, DataType::U32, "moe_total_tiles")
+ }),
+ moe_dispatch_args: moe.map(|_| {
+ let shape = model_shape.moe_dispatch_args_shape();
+ alloc(&shape, DataType::U32, "moe_dispatch_args")
+ }),
+ moe_scatter_partials: moe_scatter_entries.map(|entries| {
+ alloc(&[entries], DataType::U32, "moe_scatter_partials")
+ }),
+ moe_scatter_block_bases: moe_scatter_entries.map(|entries| {
+ alloc(&[entries], DataType::U32, "moe_scatter_block_bases")
+ }),
+ moe_block_alloc: moe_scatter_entries.map(|entries| {
+ alloc(&[entries], DataType::U32, "moe_block_alloc")
+ }),
}
}
}
diff --git a/crates/uzu/src/backends/metal/forward_pass/short_conv_layer.rs b/crates/uzu/src/backends/metal/forward_pass/short_conv_layer.rs
index 0d4d5ac84..0933ec813 100644
--- a/crates/uzu/src/backends/metal/forward_pass/short_conv_layer.rs
+++ b/crates/uzu/src/backends/metal/forward_pass/short_conv_layer.rs
@@ -1,7 +1,6 @@
use bytemuck::fill_zeroes;
use super::kv_cache_layer::ArrayCell;
-use crate::device::array::Array;
#[derive(Debug)]
pub struct ShortConvLayer {
@@ -11,6 +10,6 @@ pub struct ShortConvLayer {
impl ShortConvLayer {
pub fn zero(&self) {
let mut conv = self.conv_state.borrow_mut();
- fill_zeroes(conv.buffer_mut());
+ fill_zeroes(conv.as_bytes_mut());
}
}
diff --git a/crates/uzu/src/backends/metal/forward_pass/ssm_layer.rs b/crates/uzu/src/backends/metal/forward_pass/ssm_layer.rs
index 1f633d314..43ede5bfc 100644
--- a/crates/uzu/src/backends/metal/forward_pass/ssm_layer.rs
+++ b/crates/uzu/src/backends/metal/forward_pass/ssm_layer.rs
@@ -1,7 +1,6 @@
use bytemuck::fill_zeroes;
use super::kv_cache_layer::ArrayCell;
-use crate::device::array::Array;
#[derive(Debug)]
pub struct SSMLayer {
@@ -13,11 +12,11 @@ impl SSMLayer {
pub fn zero(&self) {
{
let mut conv = self.conv_state.borrow_mut();
- fill_zeroes(conv.buffer_mut());
+ fill_zeroes(conv.as_bytes_mut());
}
{
let mut ssm = self.ssm_state.borrow_mut();
- fill_zeroes(ssm.buffer_mut());
+ fill_zeroes(ssm.as_bytes_mut());
}
}
}
diff --git a/crates/uzu/src/backends/metal/forward_pass/state/common_aux_buffers.rs b/crates/uzu/src/backends/metal/forward_pass/state/common_aux_buffers.rs
index 7a71e93c1..e5d516905 100644
--- a/crates/uzu/src/backends/metal/forward_pass/state/common_aux_buffers.rs
+++ b/crates/uzu/src/backends/metal/forward_pass/state/common_aux_buffers.rs
@@ -1,8 +1,8 @@
-use std::{cell::RefCell, rc::Rc};
+use std::cell::RefCell;
use super::super::{ModelShape, ScratchBuffers};
use crate::{
- Array,
+ array::ArrayCellExt,
backends::metal::{MTLContext, MetalArray},
};
@@ -26,80 +26,44 @@ pub struct CommonAuxBuffers {
impl CommonAuxBuffers {
pub fn new(
- scratch: &ScratchBuffers<Rc<MTLContext>>,
+ scratch: &ScratchBuffers<MTLContext>,
model_shape: &ModelShape,
suffix_length: usize,
) -> Self {
- let act_dtype = model_shape.activation_data_type();
- unsafe {
- Self {
- suffix_length,
- main: RefCell::new(MetalArray::new(
- scratch.main.borrow_mut().mtl_buffer().into(),
- &model_shape.main_shape(suffix_length),
- act_dtype,
- )),
- shortcut: RefCell::new(MetalArray::new(
- scratch.shortcut.borrow_mut().mtl_buffer().into(),
- &model_shape.main_shape(suffix_length),
- act_dtype,
- )),
- qkv: RefCell::new(MetalArray::new(
- scratch.qkv.borrow_mut().mtl_buffer().into(),
- &model_shape.qkv_shape(suffix_length),
- act_dtype,
- )),
- attention_output: RefCell::new(MetalArray::new(
- scratch.attention_output.borrow().mtl_buffer_cloned(),
- &model_shape.attention_output_shape(suffix_length),
- act_dtype,
- )),
- mlp_fused_up: RefCell::new(MetalArray::new(
- scratch.mlp_fused_up.borrow().mtl_buffer_cloned(),
- &model_shape.mlp_fused_up_shape(suffix_length),
- act_dtype,
- )),
- mlp_hidden: RefCell::new(MetalArray::new(
- scratch.mlp_hidden.borrow().mtl_buffer_cloned(),
- &model_shape.mlp_hidden_shape(suffix_length),
- act_dtype,
- )),
- rotated_queries: RefCell::new(MetalArray::new(
- scratch.rotated_queries.borrow().mtl_buffer_cloned(),
- &model_shape.rotated_queries_shape(suffix_length),
- act_dtype,
- )),
- rotated_keys: RefCell::new(MetalArray::new(
- scratch.rotated_keys.borrow().mtl_buffer_cloned(),
- &model_shape.rotated_keys_shape(suffix_length),
- act_dtype,
- )),
- extracted_values: RefCell::new(MetalArray::new(
- scratch.extracted_values.borrow().mtl_buffer_cloned(),
- &model_shape.extracted_values_shape(suffix_length),
- act_dtype,
- )),
- attention_partials: RefCell::new(MetalArray::new(
- scratch
- .attention_partials
- .borrow_mut()
- .backend_buffer()
- .to_owned()
- .into(),
- &model_shape.attention_partials_shape(suffix_length),
- act_dtype,
- )),
- attention_sums: RefCell::new(MetalArray::new(
- scratch.attention_sums.borrow().mtl_buffer_cloned(),
- &model_shape.attention_sums_shape(suffix_length),
- act_dtype,
- )),
- attention_maxs: RefCell::new(MetalArray::new(
- scratch.attention_maxs.borrow().mtl_buffer_cloned(),
- &model_shape.attention_maxs_shape(suffix_length),
- act_dtype,
- )),
- }
+ Self {
+ suffix_length,
+ main: scratch.main.view(&model_shape.main_shape(suffix_length)),
+ shortcut: scratch
+ .shortcut
+ .view(&model_shape.main_shape(suffix_length)),
+ qkv: scratch.qkv.view(&model_shape.qkv_shape(suffix_length)),
+ attention_output: scratch
+ .attention_output
+ .view(&model_shape.attention_output_shape(suffix_length)),
+ mlp_fused_up: scratch
+ .mlp_fused_up
+ .view(&model_shape.mlp_fused_up_shape(suffix_length)),
+ mlp_hidden: scratch
+ .mlp_hidden
+ .view(&model_shape.mlp_hidden_shape(suffix_length)),
+ rotated_queries: scratch
+ .rotated_queries
+ .view(&model_shape.rotated_queries_shape(suffix_length)),
+ rotated_keys: scratch
+ .rotated_keys
+ .view(&model_shape.rotated_keys_shape(suffix_length)),
+ extracted_values: scratch
+ .extracted_values
+ .view(&model_shape.extracted_values_shape(suffix_length)),
+ attention_partials: scratch
+ .attention_partials
+ .view(&model_shape.attention_partials_shape(suffix_length)),
+ attention_sums: scratch
+ .attention_sums
+ .view(&model_shape.attention_sums_shape(suffix_length)),
+ attention_maxs: scratch
+ .attention_maxs
+ .view(&model_shape.attention_maxs_shape(suffix_length)),
}
}
diff --git a/crates/uzu/src/backends/metal/forward_pass/state/language_model_generator_aux_buffers.rs b/crates/uzu/src/backends/metal/forward_pass/state/language_model_generator_aux_buffers.rs
index 7cfec8f88..46dc4186c 100644
--- a/crates/uzu/src/backends/metal/forward_pass/state/language_model_generator_aux_buffers.rs
+++ b/crates/uzu/src/backends/metal/forward_pass/state/language_model_generator_aux_buffers.rs
@@ -1,8 +1,9 @@
-use std::{cell::RefCell, rc::Rc};
+use std::cell::RefCell;
use super::super::{ModelShape, ScratchBuffers};
use crate::{
- DataType, DecoderConfig,
+ DecoderConfig,
+ array::ArrayCellExt,
backends::metal::{MTLContext, MetalArray},
config::MLPConfig,
};
@@ -43,422 +44,149 @@ pub struct LanguageModelGeneratorAuxBuffers {
impl LanguageModelGeneratorAuxBuffers {
pub fn new(
- scratch: &ScratchBuffers<Rc<MTLContext>>,
+ scratch: &ScratchBuffers<MTLContext>,
decoder_config: &DecoderConfig,
model_shape: &ModelShape,
suffix_length: usize,
) -> Self {
- let act_dtype = model_shape.activation_data_type();
+ let moe = match &decoder_config.layer_config.mlp_config {
+ MLPConfig::MixtureOfExperts(moe) => Some(moe),
+ _ => None,
+ };
+ let num_experts_per_token = moe.map(|moe| moe.num_experts_per_token);
+ let mixture_size = moe.map(|moe| moe.mixture_size);
+ let max_routed = num_experts_per_token.map(|k| suffix_length * k);
+ let num_blocks = suffix_length.div_ceil(256).max(1);
+ let scatter_entries =
+ mixture_size.map(|e| num_blocks * e.div_ceil(512).max(1) * 512);
+ let block_alloc_entries =
+ mixture_size.map(|e| num_blocks * e.div_ceil(512).max(1));
- unsafe {
- Self {
- ssm_inproj: match (
- scratch.ssm_inproj.as_ref(),
- model_shape.ssm_inproj_shape(suffix_length),
- ) {
- (Some(buf), Some(shape)) => {
- Some(RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &shape,
- act_dtype,
- )))
- },
- _ => None,
- },
- ssm_packed: match (
- scratch.ssm_packed.as_ref(),
- model_shape.ssm_packed_shape(suffix_length),
- ) {
- (Some(buf), Some(shape)) => {
- Some(RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &shape,
- act_dtype,
- )))
- },
- _ => None,
- },
- ssm_conv_padded: match (
- scratch.ssm_conv_padded.as_ref(),
- model_shape.ssm_conv_padded_shape(suffix_length),
- ) {
- (Some(buf), Some(shape)) => {
- Some(RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &shape,
- act_dtype,
- )))
- },
- _ => None,
- },
- short_conv_padded: match (
- scratch.short_conv_padded.as_ref(),
- model_shape.short_conv_padded_shape(suffix_length),
- ) {
- (Some(buf), Some(shape)) => {
- Some(RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &shape,
- act_dtype,
- )))
- },
- _ => None,
- },
- ssm_x: match (
- scratch.ssm_x.as_ref(),
- model_shape.ssm_x_shape(suffix_length),
- ) {
- (Some(buf), Some(shape)) => {
- Some(RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &shape,
- act_dtype,
- )))
- },
- _ => None,
- },
- ssm_b: match (
- scratch.ssm_b.as_ref(),
- model_shape.ssm_bc_shape(suffix_length),
- ) {
- (Some(buf), Some(shape)) => {
- Some(RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &shape,
- act_dtype,
- )))
- },
- _ => None,
- },
- ssm_c: match (
- scratch.ssm_c.as_ref(),
- model_shape.ssm_bc_shape(suffix_length),
- ) {
- (Some(buf), Some(shape)) => {
- Some(RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &shape,
- act_dtype,
- )))
- },
- _ => None,
- },
- ssm_dt: match (
- scratch.ssm_dt.as_ref(),
- model_shape.ssm_dt_shape(suffix_length),
- ) {
- (Some(buf), Some(shape)) => {
- Some(RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &shape,
- act_dtype,
- )))
- },
- _ => None,
- },
- ssm_z: match (
- scratch.ssm_z.as_ref(),
- model_shape.ssm_z_shape(suffix_length),
- ) {
- (Some(buf), Some(shape)) => {
- Some(RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &shape,
- act_dtype,
- )))
- },
- _ => None,
- },
- moe_topk_ids: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- scratch.moe_topk_ids.as_ref().map(|buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &model_shape.moe_topk_ids_shape(
- suffix_length,
- moe.num_experts_per_token,
- ),
- DataType::U32,
- ))
- })
- },
- _ => None,
- },
- moe_topk_probs: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- scratch.moe_topk_probs.as_ref().map(|buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &model_shape.moe_topk_probs_shape(
- suffix_length,
- moe.num_experts_per_token,
- ),
- act_dtype,
- ))
- })
- },
- _ => None,
- },
- moe_offsets: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- scratch.moe_offsets.as_ref().map(|buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &model_shape
- .moe_offsets_shape(moe.mixture_size),
- DataType::U32,
- ))
- })
- },
- _ => None,
- },
- moe_sumk: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(_) => {
- scratch.moe_sumk.as_ref().map(|buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &model_shape.moe_sumk_shape(),
- DataType::U32,
- ))
- })
- },
- _ => None,
- },
- moe_bucketed_token_ids: match &decoder_config
- .layer_config
- .mlp_config
- {
- MLPConfig::MixtureOfExperts(moe) => {
- let max_routed =
- suffix_length * moe.num_experts_per_token;
- scratch.moe_bucketed_token_ids.as_ref().map(|buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &model_shape
- .moe_bucketed_token_ids_shape(max_routed),
- DataType::U32,
- ))
- })
- },
- _ => None,
- },
- moe_bucketed_probs: match &decoder_config
- .layer_config
- .mlp_config
- {
- MLPConfig::MixtureOfExperts(moe) => {
- let max_routed =
- suffix_length * moe.num_experts_per_token;
- scratch.moe_bucketed_probs.as_ref().map(|buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &model_shape
- .moe_bucketed_probs_shape(max_routed),
- act_dtype,
- ))
- })
- },
- _ => None,
- },
- moe_x_perm: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let max_routed =
- suffix_length * moe.num_experts_per_token;
- scratch.moe_x_perm.as_ref().map(|buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &model_shape.moe_x_perm_shape(max_routed),
- DataType::F16,
- ))
- })
- },
- _ => None,
- },
- moe_tok2row: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- scratch.moe_tok2row.as_ref().map(|buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &model_shape.moe_tok2row_shape(
- suffix_length,
- moe.num_experts_per_token,
- ),
- DataType::I32,
- ))
- })
- },
- _ => None,
- },
- moe_y_partial: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let max_routed =
- suffix_length * moe.num_experts_per_token;
- scratch.moe_y_partial.as_ref().map(|buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &model_shape.moe_y_partial_shape(max_routed),
- DataType::F16,
- ))
- })
- },
- _ => None,
- },
- moe_hidden: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let max_routed =
- suffix_length * moe.num_experts_per_token;
- scratch.moe_hidden.as_ref().map(|buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &model_shape.moe_hidden_shape(max_routed),
- DataType::F32,
- ))
- })
- },
- _ => None,
- },
- moe_two_pass_row_expert_map: match &decoder_config
- .layer_config
- .mlp_config
- {
- MLPConfig::MixtureOfExperts(moe) => {
- let max_routed =
- suffix_length * moe.num_experts_per_token;
- scratch.moe_two_pass_row_expert_map.as_ref().map(
- |buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &[max_routed],
- DataType::U32,
- ))
- },
- )
- },
- _ => None,
- },
- moe_tile_counts: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- scratch.moe_tile_counts.as_ref().map(|buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &model_shape.moe_counts_shape(moe.mixture_size),
- DataType::U32,
- ))
- })
- },
- _ => None,
- },
- moe_tile_offsets: match &decoder_config.layer_config.mlp_config
- {
- MLPConfig::MixtureOfExperts(moe) => {
- scratch.moe_tile_offsets.as_ref().map(|buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &model_shape
- .moe_offsets_shape(moe.mixture_size),
- DataType::U32,
- ))
- })
- },
- _ => None,
- },
- moe_tile_map: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- let max_routed =
- suffix_length * moe.num_experts_per_token;
- scratch.moe_tile_map.as_ref().map(|buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &model_shape.moe_tile_map_shape(max_routed),
- DataType::U32,
- ))
- })
- },
- _ => None,
- },
- moe_total_tiles: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(_) => {
- scratch.moe_total_tiles.as_ref().map(|buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &model_shape.moe_total_tiles_shape(),
- DataType::U32,
- ))
- })
- },
- _ => None,
- },
- moe_dispatch_args: match &decoder_config.layer_config.mlp_config
- {
- MLPConfig::MixtureOfExperts(_) => {
- scratch.moe_dispatch_args.as_ref().map(|buf| {
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &model_shape.moe_dispatch_args_shape(),
- DataType::U32,
- ))
- })
- },
- _ => None,
- },
- moe_scatter_partials: match &decoder_config
- .layer_config
- .mlp_config
- {
- MLPConfig::MixtureOfExperts(moe) => {
- scratch.moe_scatter_partials.as_ref().map(|buf| {
- let num_blocks =
- ((suffix_length + 255) / 256).max(1);
- let num_tiles =
- ((moe.mixture_size + 512 - 1) / 512).max(1);
- let entries = num_blocks * num_tiles * 512;
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &[entries],
- DataType::U32,
- ))
- })
- },
- _ => None,
- },
- moe_scatter_block_bases: match &decoder_config
- .layer_config
- .mlp_config
- {
- MLPConfig::MixtureOfExperts(moe) => {
- scratch.moe_scatter_block_bases.as_ref().map(|buf| {
- let num_blocks =
- ((suffix_length + 255) / 256).max(1);
- let num_tiles =
- ((moe.mixture_size + 512 - 1) / 512).max(1);
- let entries = num_blocks * num_tiles * 512;
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &[entries],
- DataType::U32,
- ))
- })
- },
- _ => None,
- },
- moe_block_alloc: match &decoder_config.layer_config.mlp_config {
- MLPConfig::MixtureOfExperts(moe) => {
- scratch.moe_block_alloc.as_ref().map(|buf| {
- let num_blocks =
- ((suffix_length + 255) / 256).max(1);
- let num_tiles =
- ((moe.mixture_size + 512 - 1) / 512).max(1);
- let entries = num_blocks * num_tiles;
- RefCell::new(MetalArray::new(
- buf.borrow().mtl_buffer_cloned(),
- &[entries],
- DataType::U32,
- ))
- })
- },
- _ => None,
- },
- }
+ Self {
+ ssm_inproj: scratch
+ .ssm_inproj
+ .as_ref()
+ .zip(model_shape.ssm_inproj_shape(suffix_length))
+ .map(|(buf, shape)| buf.view(&shape)),
+ ssm_packed: scratch
+ .ssm_packed
+ .as_ref()
+ .zip(model_shape.ssm_packed_shape(suffix_length))
+ .map(|(buf, shape)| buf.view(&shape)),
+ ssm_conv_padded: scratch
+ .ssm_conv_padded
+ .as_ref()
+ .zip(model_shape.ssm_conv_padded_shape(suffix_length))
+ .map(|(buf, shape)| buf.view(&shape)),
+ short_conv_padded: scratch
+ .short_conv_padded
+ .as_ref()
+ .zip(model_shape.short_conv_padded_shape(suffix_length))
+ .map(|(buf, shape)| buf.view(&shape)),
+ ssm_x: scratch
+ .ssm_x
+ .as_ref()
+ .zip(model_shape.ssm_x_shape(suffix_length))
+ .map(|(buf, shape)| buf.view(&shape)),
+ ssm_b: scratch
+ .ssm_b
+ .as_ref()
+ .zip(model_shape.ssm_bc_shape(suffix_length))
+ .map(|(buf, shape)| buf.view(&shape)),
+ ssm_c: scratch
+ .ssm_c
+ .as_ref()
+ .zip(model_shape.ssm_bc_shape(suffix_length))
+ .map(|(buf, shape)| buf.view(&shape)),
+ ssm_dt: scratch
+ .ssm_dt
+ .as_ref()
+ .zip(model_shape.ssm_dt_shape(suffix_length))
+ .map(|(buf, shape)| buf.view(&shape)),
+ ssm_z: scratch
+ .ssm_z
+ .as_ref()
+ .zip(model_shape.ssm_z_shape(suffix_length))
+ .map(|(buf, shape)| buf.view(&shape)),
+ moe_topk_ids: num_experts_per_token
+ .zip(scratch.moe_topk_ids.as_ref())
+ .map(|(k, buf)| {
+ buf.view(&model_shape.moe_topk_ids_shape(suffix_length, k))
+ }),
+ moe_topk_probs: num_experts_per_token
+ .zip(scratch.moe_topk_probs.as_ref())
+ .map(|(k, buf)| {
+ buf.view(
+ &model_shape.moe_topk_probs_shape(suffix_length, k),
+ )
+ }),
+ moe_offsets: mixture_size
+ .zip(scratch.moe_offsets.as_ref())
+ .map(|(e, buf)| buf.view(&model_shape.moe_offsets_shape(e))),
+ moe_sumk: moe
+ .and(scratch.moe_sumk.as_ref())
+ .map(|buf| buf.view(&model_shape.moe_sumk_shape())),
+ moe_bucketed_token_ids: max_routed
+ .zip(scratch.moe_bucketed_token_ids.as_ref())
+ .map(|(max_routed, buf)| {
+ buf.view(
+ &model_shape.moe_bucketed_token_ids_shape(max_routed),
+ )
+ }),
+ moe_bucketed_probs: max_routed
+ .zip(scratch.moe_bucketed_probs.as_ref())
+ .map(|(max_routed, buf)| {
+ buf.view(&model_shape.moe_bucketed_probs_shape(max_routed))
+ }),
+ moe_x_perm: max_routed.zip(scratch.moe_x_perm.as_ref()).map(
+ |(max_routed, buf)| {
+ buf.view(&model_shape.moe_x_perm_shape(max_routed))
+ },
+ ),
+ moe_tok2row: num_experts_per_token
+ .zip(scratch.moe_tok2row.as_ref())
+ .map(|(k, buf)| {
+ buf.view(&model_shape.moe_tok2row_shape(suffix_length, k))
+ }),
+ moe_y_partial: max_routed.zip(scratch.moe_y_partial.as_ref()).map(
+ |(max_routed, buf)| {
+ buf.view(&model_shape.moe_y_partial_shape(max_routed))
+ },
+ ),
+ moe_hidden: max_routed.zip(scratch.moe_hidden.as_ref()).map(
+ |(max_routed, buf)| {
+ buf.view(&model_shape.moe_hidden_shape(max_routed))
+ },
+ ),
+ moe_two_pass_row_expert_map: max_routed
+ .zip(scratch.moe_two_pass_row_expert_map.as_ref())
+ .map(|(max_routed, buf)| buf.view(&[max_routed])),
+ moe_tile_counts: mixture_size
+ .zip(scratch.moe_tile_counts.as_ref())
+ .map(|(e, buf)| buf.view(&model_shape.moe_counts_shape(e))),
+ moe_tile_offsets: mixture_size
+ .zip(scratch.moe_tile_offsets.as_ref())
+ .map(|(e, buf)| buf.view(&model_shape.moe_offsets_shape(e))),
+ moe_tile_map: max_routed.zip(scratch.moe_tile_map.as_ref()).map(
+ |(max_routed, buf)| {
+ buf.view(&model_shape.moe_tile_map_shape(max_routed))
+ },
+ ),
+ moe_total_tiles: moe
+ .and(scratch.moe_total_tiles.as_ref())
+ .map(|buf| buf.view(&model_shape.moe_total_tiles_shape())),
+ moe_dispatch_args: moe
+ .and(scratch.moe_dispatch_args.as_ref())
+ .map(|buf| buf.view(&model_shape.moe_dispatch_args_shape())),
+ moe_scatter_partials: scatter_entries
+ .zip(scratch.moe_scatter_partials.as_ref())
+ .map(|(entries, buf)| buf.view(&[entries])),
+ moe_scatter_block_bases: scatter_entries
+ .zip(scratch.moe_scatter_block_bases.as_ref())
+ .map(|(entries, buf)| buf.view(&[entries])),
+ moe_block_alloc: block_alloc_entries
+ .zip(scratch.moe_block_alloc.as_ref())
+ .map(|(entries, buf)| buf.view(&[entries])),
}
}
}
diff --git a/crates/uzu/src/backends/metal/forward_pass/state/mod.rs b/crates/uzu/src/backends/metal/forward_pass/state/mod.rs
index 970c00c01..91e55dd34 100644
--- a/crates/uzu/src/backends/metal/forward_pass/state/mod.rs
+++ b/crates/uzu/src/backends/metal/forward_pass/state/mod.rs
@@ -24,13 +24,18 @@ pub use shared_buffers::{MoeExpertWeights, SharedBuffers};
use super::traces::ActivationTrace;
use super::{ModelShape, ScratchBuffers, cache_layers::CacheLayers};
use crate::{
- Array, DataType, DecoderConfig, DeviceContext,
- backends::metal::{
- MTLBlitCommandEncoder, MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
- MTLContext, MTLDeviceExt, MTLResourceOptions, MetalArray,
- ProtocolObject, Retained,
+ DataType, DecoderConfig,
+ array::ArrayCellExt,
+ backends::{
+ common::Context,
+ metal::{
+ MTLBlitCommandEncoder, MTLBuffer, MTLCommandBuffer,
+ MTLCommandEncoder, MTLContext, MetalArray, ProtocolObject,
+ Retained,
+ },
},
session::parameter::SamplingMethod,
+ utils::attention::fill_attention_bias,
};
pub type ArrayCell = RefCell<MetalArray>;
@@ -53,42 +58,28 @@ impl ForwardPassState {
// ========================================================================
fn init_token_ids(
- context: &MTLContext,
- scratch: &ScratchBuffers<Rc<MTLContext>>,
+ scratch: &ScratchBuffers<MTLContext>,
token_ids: &[u64],
) -> ArrayCell {
let suffix_length = token_ids.len();
- let mut token_ids_array = unsafe {
- MetalArray::new(
- scratch.token_ids.borrow().mtl_buffer_cloned(),
- &[suffix_length],
- DataType::U64,
- )
- };
- context.copy_from_view(&mut token_ids_array, token_ids.into());
- RefCell::new(token_ids_array)
+ let token_ids_array = scratch.token_ids.view(&[suffix_length]);
+ token_ids_array.borrow_mut().copy_from_view(token_ids.into());
+ token_ids_array
}
fn init_token_positions(
- context: &MTLContext,
- scratch: &ScratchBuffers<Rc<MTLContext>>,
+ scratch: &ScratchBuffers<MTLContext>,
token_positions: &[usize],
) -> ArrayCell {
let suffix_length = token_positions.len();
- let mut token_positions_array = unsafe {
- MetalArray::new(
- scratch.token_positions.borrow().mtl_buffer_cloned(),
- &[suffix_length],
- DataType::I32,
- )
- };
+ let token_positions_array =
+ scratch.token_positions.view(&[suffix_length]);
let token_positions_i32: Box<[i32]> =
token_positions.iter().map(|p| *p as i32).collect();
- context.copy_from_view(
- &mut token_positions_array,
- token_positions_i32.as_ref().into(),
- );
- RefCell::new(token_positions_array)
+ token_positions_array
+ .borrow_mut()
+ .copy_from_view(token_positions_i32.as_ref().into());
+ token_positions_array
}
// ========================================================================
@@ -100,7 +91,7 @@ impl ForwardPassState {
context: Rc<MTLContext>,
decoder_config: &DecoderConfig,
model_shape: &ModelShape,
- scratch: &ScratchBuffers<Rc<MTLContext>>,
+ scratch: &ScratchBuffers<MTLContext>,
cache_layers: Rc<RefCell<CacheLayers>>,
shared_buffers: Rc<RefCell<SharedBuffers>>,
token_ids: &[u64],
@@ -133,99 +124,66 @@ impl ForwardPassState {
// Token IDs - optionally skip copy for async path
let token_ids_cell = if skip_token_ids_copy {
- RefCell::new(unsafe {
- MetalArray::new(
- scratch.token_ids.borrow().mtl_buffer_cloned(),
- &[suffix_length],
- DataType::U64,
- )
- })
+ scratch.token_ids.view(&[suffix_length])
} else {
- Self::init_token_ids(&context, scratch, token_ids)
+ Self::init_token_ids(scratch, token_ids)
};
// Token positions - use async buffer if provided
let token_positions_cell =
if let Some((async_buf, offset)) = async_positions {
let array = unsafe {
- MetalArray::new_with_offset(
+ MetalArray::from_parts(
async_buf.clone(),
+ offset * std::mem::size_of::<i32>(),
&[suffix_length],
DataType::I32,
- offset * std::mem::size_of::<i32>(),
)
};
RefCell::new(array)
} else {
- Self::init_token_positions(&context, scratch, token_positions)
+ Self::init_token_positions(scratch, token_positions)
};
// Token bitmask
let token_bitmask_cell = token_bitmask.map(|bitmask| {
let bitmask_shape = model_shape.bitmask_shape(suffix_length);
- let mut bitmask_array = unsafe {
- MetalArray::new(
- scratch.token_bitmask.borrow().mtl_buffer_cloned(),
- &bitmask_shape,
- DataType::U32,
- )
- };
- if let Ok(dst) = bitmask_array.as_slice_mut::<u32>() {
- dst.fill(0);
- }
- context.copy_from_view(&mut bitmask_array, bitmask.into());
- RefCell::new(bitmask_array)
+ let bitmask_array = scratch.token_bitmask.view(&bitmask_shape);
+ bitmask_array.borrow_mut().as_slice_mut::<u32>().fill(0);
+ bitmask_array.borrow_mut().copy_from_view(bitmask.into());
+ bitmask_array
});
// Token seeds - use async buffer if provided
let token_seeds_cell = if let Some((async_buf, offset)) = async_seeds {
let array = unsafe {
- MetalArray::new_with_offset(
+ MetalArray::from_parts(
async_buf.clone(),
+ offset * std::mem::size_of::<u64>(),
&[suffix_length],
DataType::U64,
- offset * std::mem::size_of::<u64>(),
)
};
RefCell::new(array)
} else {
- let mut token_seeds_array = unsafe {
- MetalArray::new(
- scratch.token_seeds.borrow().mtl_buffer_cloned(),
- &[suffix_length],
- DataType::U64,
- )
- };
- context.copy_from_view(&mut token_seeds_array, token_seeds.into());
- RefCell::new(token_seeds_array)
+ let token_seeds_array = scratch.token_seeds.view(&[suffix_length]);
+ token_seeds_array.borrow_mut().copy_from_view(token_seeds.into());
+ token_seeds_array
};
// Logits
- let logits_cell = RefCell::new(unsafe {
- MetalArray::new(
- scratch.logits.borrow().mtl_buffer_cloned(),
- &model_shape.logits_shape(suffix_length),
- model_shape.activation_data_type(),
- )
- });
+ let logits_cell =
+ scratch.logits.view(&model_shape.logits_shape(suffix_length));
// Sampling output
- let sampling_output = Some(RefCell::new(unsafe {
- MetalArray::new(
- scratch.sampling_output.borrow().mtl_buffer_cloned(),
- &[suffix_length],
- DataType::U32,
- )
- }));
+ let sampling_output =
+ Some(scratch.sampling_output.view(&[suffix_length]));
// Attention bias (causal + sliding window)
- let act_dtype = model_shape.activation_data_type();
let attention_bias = Self::init_llm_attention_bias(
- &context,
scratch,
&cache_layers,
suffix_length,
- act_dtype,
token_positions,
external_bias_fn,
should_fill_attention_bias,
@@ -266,7 +224,6 @@ impl ForwardPassState {
is_prefilling,
},
);
-
Self {
context,
token_ids: token_ids_cell,
@@ -281,11 +238,9 @@ impl ForwardPassState {
}
fn init_llm_attention_bias(
- context: &MTLContext,
- scratch: &ScratchBuffers<Rc<MTLContext>>,
+ scratch: &ScratchBuffers<MTLContext>,
cache_layers: &Rc<RefCell<CacheLayers>>,
suffix_length: usize,
- act_dtype: DataType,
token_positions: &[usize],
external_bias_fn: Option<&dyn Fn(usize, usize) -> bool>,
should_fill_attention_bias: bool,
@@ -300,13 +255,7 @@ impl ForwardPassState {
window_size.unwrap_or(cache_ref.max_prefix_length());
let attention_bias_shape =
[suffix_length, suffix_length + prefix_length];
- let array = unsafe {
- MetalArray::new(
- buffer.borrow().mtl_buffer_cloned(),
- &attention_bias_shape,
- act_dtype,
- )
- };
+ let array = buffer.borrow().view(&attention_bias_shape);
(*window_size, array)
})
.collect();
@@ -320,7 +269,6 @@ impl ForwardPassState {
&mut attention_bias_map,
token_positions,
suffix_length,
- context,
external_bias_fn,
);
}
@@ -339,7 +287,7 @@ impl ForwardPassState {
pub fn new_classifier(
context: Rc<MTLContext>,
model_shape: &ModelShape,
- scratch: &ScratchBuffers<Rc<MTLContext>>,
+ scratch: &ScratchBuffers<MTLContext>,
shared_buffers: Rc<RefCell<SharedBuffers>>,
token_ids: &[u64],
token_positions: &[usize],
@@ -349,17 +297,14 @@ impl ForwardPassState {
let suffix_length = token_ids.len();
assert_eq!(suffix_length, token_positions.len());
- let token_ids_cell = Self::init_token_ids(&context, scratch, token_ids);
+ let token_ids_cell = Self::init_token_ids(scratch, token_ids);
let token_positions_cell =
- Self::init_token_positions(&context, scratch, token_positions);
+ Self::init_token_positions(scratch, token_positions);
// Attention bias (bidirectional or causal)
- let act_dtype = model_shape.activation_data_type();
let attention_bias = Self::init_classifier_attention_bias(
- &context,
scratch,
suffix_length,
- act_dtype,
bidirectional_attention,
);
@@ -393,10 +338,8 @@ impl ForwardPassState {
}
fn init_classifier_attention_bias(
- context: &MTLContext,
- scratch: &ScratchBuffers<Rc<MTLContext>>,
+ scratch: &ScratchBuffers<MTLContext>,
suffix_length: usize,
- act_dtype: DataType,
bidirectional_attention: bool,
) -> HashMap<Option<usize>, ArrayCell> {
let mut attention_bias_map: HashMap<Option<usize>, MetalArray> =
@@ -405,13 +348,7 @@ impl ForwardPassState {
.iter()
.map(|(window_size, buffer)| {
let attention_bias_shape = [suffix_length, suffix_length];
- let array = unsafe {
- MetalArray::new(
- buffer.borrow().mtl_buffer_cloned(),
- &attention_bias_shape,
- act_dtype,
- )
- };
+ let array = buffer.borrow().view(&attention_bias_shape);
(*window_size, array)
})
.collect();
@@ -420,7 +357,7 @@ impl ForwardPassState {
if bidirectional_attention {
if let Some(window_size) = window {
let half_window = (window_size / 2) as isize;
- context.fill_attention_bias(
+ fill_attention_bias(
bias_array,
suffix_length,
0,
@@ -430,7 +367,7 @@ impl ForwardPassState {
},
);
} else {
- context.fill_attention_bias(
+ fill_attention_bias(
bias_array,
suffix_length,
0,
@@ -438,7 +375,7 @@ impl ForwardPassState {
);
}
} else {
- context.fill_attention_bias(
+ fill_attention_bias(
bias_array,
suffix_length,
0,
@@ -467,15 +404,12 @@ impl ForwardPassState {
let create_buffer = |size: usize| -> ArrayCell {
let buffer_size = size * data_type.size_in_bytes();
let buffer = context
- .device
- .new_buffer(
- buffer_size,
- MTLResourceOptions::STORAGE_MODE_SHARED,
- )
+ .create_buffer(buffer_size)
.expect("Failed to create buffer");
RefCell::new(unsafe {
- MetalArray::new(
+ MetalArray::from_parts(
buffer,
+ 0,
&[batch_size, size / batch_size],
data_type,
)
@@ -490,15 +424,12 @@ impl ForwardPassState {
let buffer_size =
batch_size * num_labels * data_type.size_in_bytes();
let buffer = context
- .device
- .new_buffer(
- buffer_size,
- MTLResourceOptions::STORAGE_MODE_SHARED,
- )
+ .create_buffer(buffer_size)
.expect("Failed to create buffer");
RefCell::new(unsafe {
- MetalArray::new(
+ MetalArray::from_parts(
buffer,
+ 0,
&[batch_size, num_labels],
data_type,
)
@@ -528,13 +459,11 @@ impl ForwardPassState {
let size: usize = dims.iter().product();
let buffer_size = size * data_type.size_in_bytes();
let buffer = context
- .device
- .new_buffer(
- buffer_size,
- MTLResourceOptions::STORAGE_MODE_SHARED,
- )
+ .create_buffer(buffer_size)
.expect("Failed to create buffer");
- RefCell::new(unsafe { MetalArray::new(buffer, dims, data_type) })
+ RefCell::new(unsafe {
+ MetalArray::from_parts(buffer, 0, dims, data_type)
+ })
};
ClassifierModeState {
@@ -990,7 +919,7 @@ impl ForwardPassState {
pub fn encode_copy_array(
&self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
source_array_id: ArrayId,
destination_array: RefCell<MetalArray>,
) {
@@ -998,14 +927,11 @@ impl ForwardPassState {
let src_borrow = source_ref.borrow();
let dst_borrow = destination_array.borrow();
- let src_buf = src_borrow.mtl_buffer_cloned();
- let dst_buf = dst_borrow.mtl_buffer_cloned();
+ let src_buf = src_borrow.buffer().clone();
+ let dst_buf = dst_borrow.buffer().clone();
- let copy_size_bytes = dst_borrow.size_in_bytes();
- debug_assert_eq!(
- dst_borrow.size_in_bytes(),
- src_borrow.size_in_bytes()
- );
+ let copy_size_bytes = dst_borrow.size();
+ debug_assert_eq!(dst_borrow.size(), src_borrow.size());
let blit_encoder = command_buffer
.new_blit_command_encoder()
diff --git a/crates/uzu/src/backends/metal/forward_pass/state/rope_buffers.rs b/crates/uzu/src/backends/metal/forward_pass/state/rope_buffers.rs
index 88af4784e..efe1ce8a9 100644
--- a/crates/uzu/src/backends/metal/forward_pass/state/rope_buffers.rs
+++ b/crates/uzu/src/backends/metal/forward_pass/state/rope_buffers.rs
@@ -1,8 +1,8 @@
-use std::{cell::RefCell, rc::Rc};
+use std::cell::RefCell;
use super::super::ModelShape;
use crate::{
- DeviceContext,
+ array::ArrayContextExt,
backends::metal::{MTLContext, MetalArray},
parameters::ParameterTree,
};
@@ -21,32 +21,30 @@ impl RopeBuffers {
context: &MTLContext,
model_shape: &ModelShape,
) -> Self {
- unsafe {
- let rotated_queries_shape = model_shape.rotated_queries_shape(1);
- let head_dim = rotated_queries_shape[2];
- let rope_max_sequence_length = model_shape.context_length();
-
- Self {
- cosines: RefCell::new(context.array_uninitialized(
- &[rope_max_sequence_length, head_dim],
- model_shape.activation_data_type(),
- String::from("rope_buffers_cosines"),
- )),
- sines: RefCell::new(context.array_uninitialized(
- &[rope_max_sequence_length, head_dim],
- model_shape.activation_data_type(),
- String::from("rope_buffers_sines"),
- )),
- }
+ let rotated_queries_shape = model_shape.rotated_queries_shape(1);
+ let head_dim = rotated_queries_shape[2];
+ let rope_max_sequence_length = model_shape.context_length();
+
+ Self {
+ cosines: RefCell::new(context.create_array_uninitialized(
+ &[rope_max_sequence_length, head_dim],
+ model_shape.activation_data_type(),
+ "rope_buffers_cosines",
+ )),
+ sines: RefCell::new(context.create_array_uninitialized(
+ &[rope_max_sequence_length, head_dim],
+ model_shape.activation_data_type(),
+ "rope_buffers_sines",
+ )),
}
}
pub fn update_data(
&mut self,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
- rope_name: String,
+ parameter_tree: &ParameterTree<MTLContext>,
+ rope_name: &str,
) {
- let Ok(rope_tree) = parameter_tree.subtree(rope_name.as_str()) else {
+ let Ok(rope_tree) = parameter_tree.subtree(rope_name) else {
return;
};
diff --git a/crates/uzu/src/backends/metal/forward_pass/state/shared_buffers.rs b/crates/uzu/src/backends/metal/forward_pass/state/shared_buffers.rs
index cf6446d79..9327de281 100644
--- a/crates/uzu/src/backends/metal/forward_pass/state/shared_buffers.rs
+++ b/crates/uzu/src/backends/metal/forward_pass/state/shared_buffers.rs
@@ -1,10 +1,11 @@
-use std::{cell::RefCell, rc::Rc};
+use std::cell::RefCell;
use half::{bf16, f16};
use super::{super::ModelShape, RopeBuffers};
use crate::{
- Array, DataType, DeviceContext,
+ DataType,
+ array::ArrayContextExt,
backends::metal::{MTLContext, MetalArray},
parameters::ParameterTree,
};
@@ -58,11 +59,11 @@ impl SharedBuffers {
let num_heads = decoder_config.num_heads;
Some(
(0..decoder_config.num_layers)
- .map(|_| unsafe {
- RefCell::new(context.array_uninitialized(
+ .map(|_| {
+ RefCell::new(context.create_array_uninitialized(
&[num_heads],
DataType::F32,
- String::from("shared_buffers_attention_sinks"),
+ "shared_buffers_attention_sinks",
))
})
.collect(),
@@ -84,19 +85,17 @@ impl SharedBuffers {
pub fn update_data(
&mut self,
- parameter_tree: &ParameterTree<Rc<MTLContext>>,
+ parameter_tree: &ParameterTree<MTLContext>,
) {
let transformer_tree = parameter_tree
.subtree("transformer")
.expect("transformer subtree not found");
if let Some(global_rope) = &mut self.global_rope {
- global_rope
- .update_data(&transformer_tree, String::from("global_rope"));
+ global_rope.update_data(&transformer_tree, "global_rope");
}
if let Some(local_rope) = &mut self.local_rope {
- local_rope
- .update_data(&transformer_tree, String::from("local_rope"));
+ local_rope.update_data(&transformer_tree, "local_rope");
}
if let Some(sinks_vec) = &mut self.attention_sinks {
@@ -107,15 +106,15 @@ impl SharedBuffers {
let attn_tree = layer_tree.subtree("mixer").unwrap();
let sinks_arr = attn_tree.leaf("sinks").unwrap();
let mut dst = sink_cell.borrow_mut();
- let dst_slice = dst.as_slice_mut::<f32>().unwrap();
+ let dst_slice = dst.as_slice_mut::<f32>();
match sinks_arr.data_type() {
DataType::F32 => {
- let src = sinks_arr.as_slice::<f32>().unwrap();
+ let src = sinks_arr.as_slice::<f32>();
dst_slice.copy_from_slice(src);
},
DataType::BF16 => {
- let src = sinks_arr.as_slice::<bf16>().unwrap();
+ let src = sinks_arr.as_slice::<bf16>();
for (dst_val, src_val) in
dst_slice.iter_mut().zip(src.iter())
{
@@ -123,7 +122,7 @@ impl SharedBuffers {
}
},
DataType::F16 => {
- let src = sinks_arr.as_slice::<f16>().unwrap();
+ let src = sinks_arr.as_slice::<f16>();
for (dst_val, src_val) in
dst_slice.iter_mut().zip(src.iter())
{
diff --git a/crates/uzu/src/backends/metal/forward_pass/traces.rs b/crates/uzu/src/backends/metal/forward_pass/traces.rs
index 123c504a0..0bd3a7ddf 100644
--- a/crates/uzu/src/backends/metal/forward_pass/traces.rs
+++ b/crates/uzu/src/backends/metal/forward_pass/traces.rs
@@ -1,12 +1,38 @@
use std::{cell::RefCell, rc::Rc};
use crate::{
- DeviceContext,
+ DataType,
+ array::ArrayContextExt,
backends::metal::{MTLContext, MetalArray, ModelShape},
};
type ArrayCell = RefCell<MetalArray>;
+fn create_trace_array(
+ context: &MTLContext,
+ shape: &[usize],
+ data_type: DataType,
+ label: &str,
+) -> ArrayCell {
+ RefCell::new(context.create_array_uninitialized(shape, data_type, label))
+}
+
+fn create_layer_results(
+ context: &MTLContext,
+ model_shape: &ModelShape,
+ suffix_length: usize,
+) -> Vec<Rc<RefCell<LayerActivationTrace>>> {
+ (0..model_shape.num_layers)
+ .map(|_| {
+ Rc::new(RefCell::new(LayerActivationTrace::new(
+ context,
+ model_shape,
+ suffix_length,
+ )))
+ })
+ .collect()
+}
+
pub struct LayerActivationTrace {
pub inputs: ArrayCell,
pub pre_attention_norm: ArrayCell,
@@ -25,54 +51,31 @@ impl LayerActivationTrace {
model_shape: &ModelShape,
suffix_length: usize,
) -> Self {
- unsafe {
- Self {
- inputs: RefCell::new(context.array_uninitialized(
- &model_shape.main_shape(suffix_length),
- model_shape.activation_data_type(),
- String::from("layer_activation_trace_inputs"),
- )),
- pre_attention_norm: RefCell::new(context.array_uninitialized(
- &model_shape.main_shape(suffix_length),
- model_shape.activation_data_type(),
- String::from("layer_activation_trace_pre_attention_norm"),
- )),
- attention: RefCell::new(context.array_uninitialized(
- &model_shape.main_shape(suffix_length),
- model_shape.activation_data_type(),
- String::from("layer_activation_trace_attention"),
- )),
- post_attention_norm: RefCell::new(context.array_uninitialized(
- &model_shape.main_shape(suffix_length),
- model_shape.activation_data_type(),
- String::from("layer_activation_trace_post_attention_norm"),
- )),
- mlp_inputs: RefCell::new(context.array_uninitialized(
- &model_shape.main_shape(suffix_length),
- model_shape.activation_data_type(),
- String::from("layer_activation_trace_mlp_inputs"),
- )),
- pre_mlp_norm: RefCell::new(context.array_uninitialized(
- &model_shape.main_shape(suffix_length),
- model_shape.activation_data_type(),
- String::from("layer_activation_trace_pre_mlp_norm"),
- )),
- mlp: RefCell::new(context.array_uninitialized(
- &model_shape.main_shape(suffix_length),
- model_shape.activation_data_type(),
- String::from("layer_activation_trace_mlp"),
- )),
- post_mlp_norm: RefCell::new(context.array_uninitialized(
- &model_shape.main_shape(suffix_length),
- model_shape.activation_data_type(),
- String::from("layer_activation_trace_post_mlp_norm"),
- )),
- outputs: RefCell::new(context.array_uninitialized(
- &model_shape.main_shape(suffix_length),
- model_shape.activation_data_type(),
- String::from("layer_activation_trace_outputs"),
- )),
- }
+ let main_shape = model_shape.main_shape(suffix_length);
+ let activation_data_type = model_shape.activation_data_type();
+ let main = |label| {
+ create_trace_array(
+ context,
+ &main_shape,
+ activation_data_type,
+ label,
+ )
+ };
+
+ Self {
+ inputs: main("layer_activation_trace_inputs"),
+ pre_attention_norm: main(
+ "layer_activation_trace_pre_attention_norm",
+ ),
+ attention: main("layer_activation_trace_attention"),
+ post_attention_norm: main(
+ "layer_activation_trace_post_attention_norm",
+ ),
+ mlp_inputs: main("layer_activation_trace_mlp_inputs"),
+ pre_mlp_norm: main("layer_activation_trace_pre_mlp_norm"),
+ mlp: main("layer_activation_trace_mlp"),
+ post_mlp_norm: main("layer_activation_trace_post_mlp_norm"),
+ outputs: main("layer_activation_trace_outputs"),
}
}
}
@@ -91,31 +94,27 @@ impl ActivationTrace {
model_shape: &ModelShape,
suffix_length: usize,
) -> Self {
- let layer_results = (0..model_shape.num_layers)
- .map(|_| {
- Rc::new(RefCell::new(LayerActivationTrace::new(
- context,
- model_shape,
- suffix_length,
- )))
- })
- .collect();
- unsafe {
- Self {
- embedding_norm: None,
- layer_results,
- output_norm: RefCell::new(context.array_uninitialized(
- &model_shape.main_shape(suffix_length),
- model_shape.activation_data_type(),
- String::from("activation_trace_output_norm"),
- )),
- output_pooling: None,
- logits: RefCell::new(context.array_uninitialized(
- &model_shape.logits_shape(suffix_length),
- model_shape.activation_data_type(),
- String::from("activation_trace_logits"),
- )),
- }
+ let activation_data_type = model_shape.activation_data_type();
+ let main_shape = model_shape.main_shape(suffix_length);
+ let layer_results =
+ create_layer_results(context, model_shape, suffix_length);
+
+ Self {
+ embedding_norm: None,
+ layer_results,
+ output_norm: create_trace_array(
+ context,
+ &main_shape,
+ activation_data_type,
+ "activation_trace_output_norm",
+ ),
+ output_pooling: None,
+ logits: create_trace_array(
+ context,
+ &model_shape.logits_shape(suffix_length),
+ activation_data_type,
+ "activation_trace_logits",
+ ),
}
}
@@ -125,44 +124,38 @@ impl ActivationTrace {
suffix_length: usize,
num_labels: usize,
) -> Self {
- let layer_results = (0..model_shape.num_layers)
- .map(|_| {
- Rc::new(RefCell::new(LayerActivationTrace::new(
- context,
- model_shape,
- suffix_length,
- )))
- })
- .collect();
+ let activation_data_type = model_shape.activation_data_type();
+ let main_shape = model_shape.main_shape(suffix_length);
+ let layer_results =
+ create_layer_results(context, model_shape, suffix_length);
let model_dim = model_shape.main_shape(1)[1];
- unsafe {
- Self {
- embedding_norm: Some(RefCell::new(
- context.array_uninitialized(
- &model_shape.main_shape(suffix_length),
- model_shape.activation_data_type(),
- String::from("activation_trace_embedding_norm"),
- ),
- )),
- layer_results,
- output_norm: RefCell::new(context.array_uninitialized(
- &model_shape.main_shape(suffix_length),
- model_shape.activation_data_type(),
- String::from("activation_trace_output_norm"),
- )),
- output_pooling: Some(RefCell::new(
- context.array_uninitialized(
- &[1, model_dim],
- model_shape.activation_data_type(),
- String::from("activation_trace_output_pooling"),
- ),
- )),
- logits: RefCell::new(context.array_uninitialized(
- &[1, num_labels],
- model_shape.activation_data_type(),
- String::from("activation_trace_logits"),
- )),
- }
+
+ Self {
+ embedding_norm: Some(create_trace_array(
+ context,
+ &main_shape,
+ activation_data_type,
+ "activation_trace_embedding_norm",
+ )),
+ layer_results,
+ output_norm: create_trace_array(
+ context,
+ &main_shape,
+ activation_data_type,
+ "activation_trace_output_norm",
+ ),
+ output_pooling: Some(create_trace_array(
+ context,
+ &[1, model_dim],
+ activation_data_type,
+ "activation_trace_output_pooling",
+ )),
+ logits: create_trace_array(
+ context,
+ &[1, num_labels],
+ activation_data_type,
+ "activation_trace_logits",
+ ),
}
}
}
diff --git a/crates/uzu/src/backends/metal/generated/attention.h b/crates/uzu/src/backends/metal/generated/attention.h
new file mode 100644
index 000000000..e3167fe4c
--- /dev/null
+++ b/crates/uzu/src/backends/metal/generated/attention.h
@@ -0,0 +1,64 @@
+// Auto-generated from gpu_types/attention.rs - do not edit manually
+#pragma once
+
+#ifndef UZU_ATTENTION_H
+#define UZU_ATTENTION_H
+
+#ifdef __METAL_VERSION__
+#include <metal_stdlib>
+using namespace metal;
+
+namespace uzu {
+namespace attention {
+#else
+#include <stdint.h>
+#endif
+
+/**Parameters for the GEMM attention kernel. */
+/**All strides are in **elements**, not bytes. */
+/**Q/K/V/O are treated as 3D tensors with strides for: */
+/**- `[0]` batch */
+/**- `[1]` head (or kv-head for K/V) */
+/**- `[2]` sequence (row stride for [seq, head_dim]) */
+typedef struct {
+ int64_t q_strides[3];
+ int64_t k_strides[3];
+ int64_t v_strides[3];
+ int64_t o_strides[3];
+ int32_t gqa_factor;
+ float scale;
+ /**Query length (suffix length) */
+ int32_t q_len;
+ /**Key length (prefix + suffix) */
+ int32_t k_len;
+ /**Absolute offset of the first query token in the full key sequence. */
+ /**For LLM decode/prefill this is the segment prefix length. */
+ int32_t q_off;
+ /**Query tiling: q_len / BQ */
+ int32_t nq_aligned;
+ /**Query tiling: q_len % BQ */
+ int32_t q_rem;
+ /**Key tiling: ceil(k_len / BK) */
+ int32_t nk;
+ /**Key tiling: k_len / BK */
+ int32_t nk_aligned;
+ /**Key tiling: k_len % BK */
+ int32_t k_rem;
+} AttnParams;
+
+/**Parameters describing the additive attention mask. */
+/**Mask is treated as a 2D matrix [qL, kL] with: */
+/**- per-batch base offset (`m_strides[0]`) */
+/**- per-head base offset (`m_strides[1]`) */
+/**- row stride in elements (`m_strides[2]`) */
+/**Column stride is assumed to be 1. */
+typedef struct {
+ int64_t m_strides[3];
+} AttnMaskParams;
+
+#ifdef __METAL_VERSION__
+} // namespace attention
+} // namespace uzu
+#endif
+
+#endif // UZU_ATTENTION_H
diff --git a/crates/uzu/src/backends/metal/generated/matmul.h b/crates/uzu/src/backends/metal/generated/matmul.h
new file mode 100644
index 000000000..0fe96df40
--- /dev/null
+++ b/crates/uzu/src/backends/metal/generated/matmul.h
@@ -0,0 +1,82 @@
+// Auto-generated from gpu_types/matmul.rs - do not edit manually
+#pragma once
+
+#ifndef UZU_MATMUL_H
+#define UZU_MATMUL_H
+
+#ifdef __METAL_VERSION__
+#include <metal_stdlib>
+using namespace metal;
+
+namespace uzu {
+namespace matmul {
+#else
+#include <stdint.h>
+#endif
+
+/**Main GEMM parameters passed to Metal kernels as constant buffer. */
+typedef struct {
+ int32_t M;
+ int32_t N;
+ int32_t K;
+ int32_t lda;
+ int32_t ldb;
+ int32_t ldd;
+ int32_t tiles_n;
+ int32_t tiles_m;
+ int64_t batch_stride_a;
+ int64_t batch_stride_b;
+ int64_t batch_stride_d;
+ int32_t swizzle_log;
+ int32_t gemm_k_iterations_aligned;
+ int32_t batch_ndim;
+} GEMMParams;
+
+/**Split-K GEMM parameters. */
+typedef struct {
+ int32_t M;
+ int32_t N;
+ int32_t K;
+ int32_t lda;
+ int32_t ldb;
+ int32_t ldc;
+ int32_t tiles_n;
+ int32_t tiles_m;
+ int32_t split_k_partitions;
+ int32_t split_k_partition_stride;
+ int32_t split_k_partition_size;
+ int32_t gemm_k_iterations_aligned;
+} GEMMSpiltKParams;
+
+/**Split-K MLP Fused GEMM parameters. */
+typedef struct {
+ int32_t M;
+ int32_t N;
+ int32_t K;
+ int32_t lda;
+ int32_t ldb;
+ int32_t ldc;
+ int32_t tiles_n;
+ int32_t tiles_m;
+ int32_t split_k_partitions;
+ int32_t split_k_partition_stride;
+ int32_t split_k_partition_size;
+ int32_t gemm_k_iterations_aligned;
+ int32_t hidden_dim;
+} GEMMSpiltKMlpFusedParams;
+
+/**AddMM parameters (alpha * A @ B + beta * C). */
+typedef struct {
+ int32_t ldc;
+ int32_t fdc;
+ int64_t batch_stride_c;
+ float alpha;
+ float beta;
+} GEMMAddMMParams;
+
+#ifdef __METAL_VERSION__
+} // namespace matmul
+} // namespace uzu
+#endif
+
+#endif // UZU_MATMUL_H
diff --git a/crates/uzu/src/backends/metal/kernel/activation/activation.h b/crates/uzu/src/backends/metal/kernel/activation/activation.h
new file mode 100644
index 000000000..491eb88c9
--- /dev/null
+++ b/crates/uzu/src/backends/metal/kernel/activation/activation.h
@@ -0,0 +1,29 @@
+#include <metal_stdlib>
+
+enum ActivationType : uint {
+ ACT_SILU = 0,
+ ACT_GELU = 1,
+};
+
+template <typename T>
+inline T silu(T x) {
+ float xf = float(x);
+ float yf = xf / (1.0f + exp(-xf));
+ return T(yf);
+}
+
+template <typename T>
+inline T gelu_approx(T x) {
+ constexpr float k0 = 0.044715f;
+ constexpr float k1 = 0.7978845608f; // sqrt(2/pi)
+ float xf = float(x);
+ float yf = 0.5f * xf * (1.0f + metal::precise::tanh(k1 * (xf + k0 * xf * xf * xf)));
+ return T(yf);
+}
+
+template <typename T>
+inline T activate(T x, uint act) {
+ if (act == ACT_SILU)
+ return silu(x);
+ return gelu_approx(x);
+}
diff --git a/crates/uzu/src/backends/metal/kernel/activation/activation.metal b/crates/uzu/src/backends/metal/kernel/activation/activation.metal
index b8ff92950..b83cb6510 100644
--- a/crates/uzu/src/backends/metal/kernel/activation/activation.metal
+++ b/crates/uzu/src/backends/metal/kernel/activation/activation.metal
@@ -1,69 +1,16 @@
#include <metal_stdlib>
-using namespace metal;
+#include "../definitions.metal"
-enum ActivationType : ushort {
- ACT_SILU = 0,
- ACT_GELU = 1,
-};
+#include "activation.h"
template <typename T>
-inline T silu(T x) {
- float xf = float(x);
- float yf = xf / (1.0f + exp(-xf));
- return T(yf);
-}
-
-template <typename T>
-inline T gelu_approx(T x) {
- constexpr float k0 = 0.044715f;
- constexpr float k1 = 0.7978845608f; // sqrt(2/pi)
- float xf = float(x);
- float yf = 0.5f * xf * (1.0f + tanh(k1 * (xf + k0 * xf * xf * xf)));
- return T(yf);
-}
-
-template <typename T>
-inline T activate(T x, ushort act) {
- if (act == ACT_SILU)
- return silu(x);
- return gelu_approx(x);
-}
-
-template <typename T>
-[[kernel, max_total_threads_per_threadgroup(256)]]
-void activation(
- const device T* input [[buffer(0)]],
- device T* output [[buffer(1)]],
- const constant int& N [[buffer(2)]],
- const constant ushort& act_type [[buffer(3)]],
- uint tid [[thread_position_in_grid]]
+VARIANTS(T, half, float, bfloat)
+KERNEL(Activation) (
+ const device T* input,
+ device T* output,
+ const constant uint& n,
+ const constant uint& act_type,
+ uint tid AXIS(n, 256)
) {
- if (int(tid) >= N)
- return;
output[tid] = activate(input[tid], act_type);
}
-
-// Explicit instantiations
-template [[host_name("activation_f16")]] [[kernel]] void activation<half>(
- const device half* input [[buffer(0)]],
- device half* output [[buffer(1)]],
- const constant int& N [[buffer(2)]],
- const constant ushort& act_type [[buffer(3)]],
- uint tid [[thread_position_in_grid]]
-);
-
-template [[host_name("activation_f32")]] [[kernel]] void activation<float>(
- const device float* input [[buffer(0)]],
- device float* output [[buffer(1)]],
- const constant int& N [[buffer(2)]],
- const constant ushort& act_type [[buffer(3)]],
- uint tid [[thread_position_in_grid]]
-);
-
-template [[host_name("activation_bf16")]] [[kernel]] void activation<bfloat>(
- const device bfloat* input [[buffer(0)]],
- device bfloat* output [[buffer(1)]],
- const constant int& N [[buffer(2)]],
- const constant ushort& act_type [[buffer(3)]],
- uint tid [[thread_position_in_grid]]
-);
diff --git a/crates/uzu/src/backends/metal/kernel/activation/mod.rs b/crates/uzu/src/backends/metal/kernel/activation/mod.rs
deleted file mode 100644
index aa30aec0d..000000000
--- a/crates/uzu/src/backends/metal/kernel/activation/mod.rs
+++ /dev/null
@@ -1,76 +0,0 @@
-use crate::{
- DataType,
- backends::metal::{
- ComputeEncoderSetValue, MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState,
- MTLContext, MTLError, MTLSize, ProtocolObject, Retained,
- },
- config::Activation,
-};
-
-pub struct ActivationKernel {
- pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-impl ActivationKernel {
- fn kernel_name_for_type(
- data_type: DataType
- ) -> Result<&'static str, MTLError> {
- match data_type {
- DataType::F16 => Ok("activation_f16"),
- DataType::F32 => Ok("activation_f32"),
- DataType::BF16 => Ok("activation_bf16"),
- other => Err(MTLError::Generic(format!(
- "Unsupported dtype for activation: {:?}",
- other
- ))),
- }
- }
-
- pub fn new(
- context: &MTLContext,
- data_type: DataType,
- ) -> Result<Self, MTLError> {
- let fn_name = Self::kernel_name_for_type(data_type)?;
- let pipeline = context.compute_pipeline_state(fn_name, None)?;
- Ok(Self {
- pipeline,
- })
- }
-
- fn act_code(act: &Activation) -> u16 {
- match act {
- Activation::SiLU {
- ..
- } => 0,
- Activation::Gelu => 1,
- Activation::Identity => {
- panic!("Identity activation is not supported for kernel")
- },
- }
- }
-
- pub fn encode(
- &self,
- encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- activation: &Activation,
- input_buffer: &ProtocolObject<dyn MTLBuffer>,
- output_buffer: &ProtocolObject<dyn MTLBuffer>,
- n: usize,
- ) -> Result<(), MTLError> {
- let act_code = Self::act_code(activation);
- let threads_per_tg = 256u64;
- let threadgroups = (n as u64 + threads_per_tg - 1) / threads_per_tg;
-
- encoder.set_compute_pipeline_state(&self.pipeline);
- encoder.set_buffer(Some(input_buffer), 0, 0);
- encoder.set_buffer(Some(output_buffer), 0, 1);
- encoder.set_value(&(n as i32), 2);
- encoder.set_value(&act_code, 3);
-
- encoder.dispatch_threadgroups(
- MTLSize::new(threadgroups as usize, 1, 1),
- MTLSize::new(threads_per_tg as usize, 1, 1),
- );
- Ok(())
- }
-}
diff --git a/crates/uzu/src/backends/metal/kernel/attention/gemm_attention.metal b/crates/uzu/src/backends/metal/kernel/attention/gemm_attention.metal
index 07470e915..480b71e48 100644
--- a/crates/uzu/src/backends/metal/kernel/attention/gemm_attention.metal
+++ b/crates/uzu/src/backends/metal/kernel/attention/gemm_attention.metal
@@ -2,8 +2,8 @@
#include "../matmul/common/loader.h"
#include "../matmul/common/mma.h"
-
-#include "gemm_types.h"
+#include "../definitions.metal"
+#include "attention.h"
using namespace metal;
using namespace uzu::matmul;
@@ -390,7 +390,7 @@ template <
for (short id = 0; id < TD; id++) {
UZU_PRAGMA_UNROLL
for (short ik = 0; ik < TK; ik++) {
- if constexpr (BD == 128) {
+ IF_CONSTEXPR(BD == 128) {
simdgroup_barrier(mem_flags::mem_none);
}
@@ -401,7 +401,7 @@ template <
&Vs[Vs_offset + kk * LDV_tgp + dd]
);
- if constexpr (BD == 128) {
+ IF_CONSTEXPR(BD == 128) {
simdgroup_barrier(mem_flags::mem_none);
}
diff --git a/crates/uzu/src/backends/metal/kernel/attention/gemm_types.h b/crates/uzu/src/backends/metal/kernel/attention/gemm_types.h
deleted file mode 100644
index fada4ef99..000000000
--- a/crates/uzu/src/backends/metal/kernel/attention/gemm_types.h
+++ /dev/null
@@ -1,77 +0,0 @@
-// Shared types for Gemm attention kernel - used by both Metal shaders and Rust
-// (via bindgen). This header must be C-compatible for bindgen to parse it.
-//
-// NOTE: Only wrap namespaces for Metal builds. For C/bindgen builds, keep
-// everything in the global namespace.
-//
-// SPDX-License-Identifier: MIT
-
-#pragma once
-
-// Metal and C have different type systems
-// __METAL_VERSION__ is defined when compiling with the Metal shader compiler
-#ifdef __METAL_VERSION__
-#include <metal_stdlib>
-using namespace metal;
-
-namespace uzu {
-namespace attention {
-
-#else
-#include <stdint.h>
-#endif
-
-///////////////////////////////////////////////////////////////////////////////
-// Gemm Attention Parameters
-///////////////////////////////////////////////////////////////////////////////
-
-/// Parameters for the Gemm attention kernel.
-///
-/// All strides are in **elements**, not bytes.
-/// Q/K/V/O are treated as 3D tensors with strides for:
-/// - [0] batch
-/// - [1] head (or kv-head for K/V)
-/// - [2] sequence (row stride for [seq, head_dim])
-struct AttnParams {
- int64_t q_strides[3];
- int64_t k_strides[3];
- int64_t v_strides[3];
- int64_t o_strides[3];
-
- int gqa_factor;
- float scale;
-
- // Query length (suffix length) and key length (prefix + suffix)
- int q_len;
- int k_len;
-
- // Absolute offset of the first query token in the full key sequence.
- // For LLM decode/prefill this is the segment prefix length.
- int q_off;
-
- // Query tiling metadata (BQ tiles)
- int nq_aligned; // q_len / BQ
- int q_rem; // q_len % BQ
-
- // Key tiling metadata (BK tiles)
- int nk; // ceil(k_len / BK)
- int nk_aligned; // k_len / BK
- int k_rem; // k_len % BK
-};
-
-/// Parameters describing the additive attention mask.
-///
-/// Mask is treated as a 2D matrix [qL, kL] with:
-/// - per-batch base offset (M_strides[0])
-/// - per-head base offset (M_strides[1])
-/// - row stride in elements (M_strides[2])
-/// Column stride is assumed to be 1.
-struct AttnMaskParams {
- int64_t m_strides[3];
-};
-
-// Close Metal namespace
-#ifdef __METAL_VERSION__
-} // namespace attention
-} // namespace uzu
-#endif
diff --git a/crates/uzu/src/backends/metal/kernel/attention/gemm_types.rs b/crates/uzu/src/backends/metal/kernel/attention/gemm_types.rs
deleted file mode 100644
index 49ac92d5a..000000000
--- a/crates/uzu/src/backends/metal/kernel/attention/gemm_types.rs
+++ /dev/null
@@ -1,69 +0,0 @@
-/* automatically generated by rust-bindgen 0.72.1 */
-
-#[doc = " Parameters for the Gemm attention kernel.\n\n All strides are in **elements**, not bytes.\n Q/K/V/O are treated as 3D tensors with strides for:\n - [0] batch\n - [1] head (or kv-head for K/V)\n - [2] sequence (row stride for [seq, head_dim])"]
-#[repr(C)]
-#[derive(Debug, Default, Copy, Clone)]
-pub struct AttnParams {
- pub q_strides: [i64; 3usize],
- pub k_strides: [i64; 3usize],
- pub v_strides: [i64; 3usize],
- pub o_strides: [i64; 3usize],
- pub gqa_factor: ::core::ffi::c_int,
- pub scale: f32,
- pub q_len: ::core::ffi::c_int,
- pub k_len: ::core::ffi::c_int,
- pub q_off: ::core::ffi::c_int,
- pub nq_aligned: ::core::ffi::c_int,
- pub q_rem: ::core::ffi::c_int,
- pub nk: ::core::ffi::c_int,
- pub nk_aligned: ::core::ffi::c_int,
- pub k_rem: ::core::ffi::c_int,
-}
-#[allow(clippy::unnecessary_operation, clippy::identity_op)]
-const _: () = {
- ["Size of AttnParams"][::core::mem::size_of::<AttnParams>() - 136usize];
- ["Alignment of AttnParams"][::core::mem::align_of::<AttnParams>() - 8usize];
- ["Offset of field: AttnParams::q_strides"]
- [::core::mem::offset_of!(AttnParams, q_strides) - 0usize];
- ["Offset of field: AttnParams::k_strides"]
- [::core::mem::offset_of!(AttnParams, k_strides) - 24usize];
- ["Offset of field: AttnParams::v_strides"]
- [::core::mem::offset_of!(AttnParams, v_strides) - 48usize];
- ["Offset of field: AttnParams::o_strides"]
- [::core::mem::offset_of!(AttnParams, o_strides) - 72usize];
- ["Offset of field: AttnParams::gqa_factor"]
- [::core::mem::offset_of!(AttnParams, gqa_factor) - 96usize];
- ["Offset of field: AttnParams::scale"]
- [::core::mem::offset_of!(AttnParams, scale) - 100usize];
- ["Offset of field: AttnParams::q_len"]
- [::core::mem::offset_of!(AttnParams, q_len) - 104usize];
- ["Offset of field: AttnParams::k_len"]
- [::core::mem::offset_of!(AttnParams, k_len) - 108usize];
- ["Offset of field: AttnParams::q_off"]
- [::core::mem::offset_of!(AttnParams, q_off) - 112usize];
- ["Offset of field: AttnParams::nq_aligned"]
- [::core::mem::offset_of!(AttnParams, nq_aligned) - 116usize];
- ["Offset of field: AttnParams::q_rem"]
- [::core::mem::offset_of!(AttnParams, q_rem) - 120usize];
- ["Offset of field: AttnParams::nk"]
- [::core::mem::offset_of!(AttnParams, nk) - 124usize];
- ["Offset of field: AttnParams::nk_aligned"]
- [::core::mem::offset_of!(AttnParams, nk_aligned) - 128usize];
- ["Offset of field: AttnParams::k_rem"]
- [::core::mem::offset_of!(AttnParams, k_rem) - 132usize];
-};
-#[doc = " Parameters describing the additive attention mask.\n\n Mask is treated as a 2D matrix [qL, kL] with:\n - per-batch base offset (M_strides[0])\n - per-head base offset (M_strides[1])\n - row stride in elements (M_strides[2])\n Column stride is assumed to be 1."]
-#[repr(C)]
-#[derive(Debug, Default, Copy, Clone)]
-pub struct AttnMaskParams {
- pub m_strides: [i64; 3usize],
-}
-#[allow(clippy::unnecessary_operation, clippy::identity_op)]
-const _: () = {
- ["Size of AttnMaskParams"]
- [::core::mem::size_of::<AttnMaskParams>() - 24usize];
- ["Alignment of AttnMaskParams"]
- [::core::mem::align_of::<AttnMaskParams>() - 8usize];
- ["Offset of field: AttnMaskParams::m_strides"]
- [::core::mem::offset_of!(AttnMaskParams, m_strides) - 0usize];
-};
diff --git a/crates/uzu/src/backends/metal/kernel/attention/mod.rs b/crates/uzu/src/backends/metal/kernel/attention/mod.rs
index 6715c0000..7d4cd65c1 100644
--- a/crates/uzu/src/backends/metal/kernel/attention/mod.rs
+++ b/crates/uzu/src/backends/metal/kernel/attention/mod.rs
@@ -4,12 +4,12 @@ use objc2::rc::Retained;
use thiserror::Error;
use crate::backends::metal::{
- KernelDataType, MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext,
- MTLDataType, MTLError, MTLFunctionConstantValues, MTLSize, ProtocolObject,
+ KernelDataType, MTLBuffer, MTLComputeCommandEncoder,
+ MTLComputePipelineState, MTLContext, MTLDataType, MTLError,
+ MTLFunctionConstantValues, MTLSize, ProtocolObject,
};
-mod gemm_types;
-use gemm_types::{AttnMaskParams, AttnParams};
+use crate::backends::common::gpu_types::{AttnMaskParams, AttnParams};
#[derive(Debug, Clone, Copy)]
pub enum AttentionKernelVariant {
@@ -20,10 +20,18 @@ pub enum AttentionKernelVariant {
type PipelineKey = (usize, bool, bool, bool); // (head_dim, has_sinks, is_causal, has_mask)
pub struct AttentionKernelPipelines {
- single_pass: HashMap<PipelineKey, Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
- two_pass_1: HashMap<PipelineKey, Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
- two_pass_2: HashMap<usize, Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
- kv_cache_update: Option<Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
+ single_pass: HashMap<
+ PipelineKey,
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ >,
+ two_pass_1: HashMap<
+ PipelineKey,
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ >,
+ two_pass_2:
+ HashMap<usize, Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
+ kv_cache_update:
+ Option<Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
}
pub struct AttentionKernel {
@@ -190,7 +198,10 @@ impl AttentionKernel {
for &head_dim in &supported_head_dims {
if let Ok(pipeline) = context.compute_pipeline_state(
- &format!("attention_single_pass_{}_{}", data_suffix, head_dim),
+ &format!(
+ "attention_single_pass_{}_{}",
+ data_suffix, head_dim
+ ),
Some(&function_constants),
) {
single_pass.insert(
@@ -205,7 +216,10 @@ impl AttentionKernel {
}
if let Ok(pipeline) = context.compute_pipeline_state(
- &format!("attention_2pass_1_{}_{}", data_suffix, head_dim),
+ &format!(
+ "attention_2pass_1_{}_{}",
+ data_suffix, head_dim
+ ),
Some(&function_constants),
) {
two_pass_1.insert(
@@ -220,10 +234,15 @@ impl AttentionKernel {
}
if !two_pass_2.contains_key(&head_dim) {
- if let Ok(pipeline) = context.compute_pipeline_state(
- &format!("attention_2pass_2_{}_{}", data_suffix, head_dim),
- Some(&function_constants),
- ) {
+ if let Ok(pipeline) = context
+ .compute_pipeline_state(
+ &format!(
+ "attention_2pass_2_{}_{}",
+ data_suffix, head_dim
+ ),
+ Some(&function_constants),
+ )
+ {
two_pass_2.insert(head_dim, pipeline);
}
}
@@ -233,7 +252,10 @@ impl AttentionKernel {
}
let kv_cache_update = context
- .compute_pipeline_state(&format!("update_kv_cache_{}", data_suffix), None)
+ .compute_pipeline_state(
+ &format!("update_kv_cache_{}", data_suffix),
+ None,
+ )
.ok();
Ok(Self {
@@ -862,7 +884,11 @@ impl AttentionKernel {
);
let pipeline = context
- .compute_pipeline_state_cached(&cache_key, &function_name, Some(&fcv))
+ .compute_pipeline_state_cached(
+ &cache_key,
+ &function_name,
+ Some(&fcv),
+ )
.map_err(AttentionError::MetalError)?;
MTLComputeCommandEncoder::set_compute_pipeline_state(
diff --git a/crates/uzu/src/backends/metal/kernel/definitions.metal b/crates/uzu/src/backends/metal/kernel/definitions.metal
index 5ee29bb0d..047eec575 100644
--- a/crates/uzu/src/backends/metal/kernel/definitions.metal
+++ b/crates/uzu/src/backends/metal/kernel/definitions.metal
@@ -460,6 +460,13 @@ static T threadgroup_cooperative_reduce_min(
return result;
}
+// warning: constexpr if is a C++17 extension [-Wc++17-extensions]
+#if defined(__cpp_if_constexpr)
+ #define IF_CONSTEXPR(cond) if constexpr (cond)
+#else
+ #define IF_CONSTEXPR(cond) if (cond)
+#endif
+
// MARK: - DSL Annotation Helpers
#ifdef DSL_ANALYZE
@@ -471,11 +478,11 @@ static T threadgroup_cooperative_reduce_min(
#define DSL_STR(X) #X
#define DSL_XSTR(X) DSL_STR(X)
-#define SPECIALIZE(TYPENAME, ...) \
- template <typename TYPENAME> \
- DSL_META("dsl.specialize", #TYPENAME, #__VA_ARGS__)
+#define VARIANTS(TYPENAME, ...) DSL_META("dsl.variants", #TYPENAME, #__VA_ARGS__)
#define KERNEL(NAME) DSL_META("dsl.kernel") void NAME
+#define SPECIALIZE DSL_META("dsl.specialize")
+
#define AXIS(TDS, TPG) DSL_META("dsl.axis", DSL_XSTR(TDS), DSL_XSTR(TPG))
#define GROUPS(EXPR) DSL_META("dsl.groups", DSL_XSTR(EXPR))
#define THREADS(EXPR) DSL_META("dsl.threads", DSL_XSTR(EXPR))
diff --git a/crates/uzu/src/backends/metal/kernel/embedding/full_precision_embedding.metal b/crates/uzu/src/backends/metal/kernel/embedding/full_precision_embedding.metal
index ee7667836..56672f4e0 100644
--- a/crates/uzu/src/backends/metal/kernel/embedding/full_precision_embedding.metal
+++ b/crates/uzu/src/backends/metal/kernel/embedding/full_precision_embedding.metal
@@ -1,66 +1,29 @@
#include <metal_stdlib>
-using namespace metal;
+#include "../definitions.metal"
+
+#define BLOCK_SIZE 256
template <typename T>
-[[kernel, max_total_threads_per_threadgroup(256)]]
-void full_precision_embedding_lookup(
- const device uint64_t* token_ids [[buffer(0)]], // [batch_size]
- const device T* weights [[buffer(1)]], // [vocab_size, model_dim]
- device T* output [[buffer(2)]], // [batch_size, model_dim]
- constant uint32_t& batch_size [[buffer(3)]],
- constant uint32_t& vocab_size [[buffer(4)]],
- constant uint32_t& model_dim [[buffer(5)]],
- constant float& input_scale [[buffer(6)]],
- uint thread_position_in_grid [[thread_position_in_grid]]
+VARIANTS(T, float, half, bfloat)
+KERNEL(FullPrecisionEmbeddingLookup) (
+ const device uint64_t* token_ids, // [batch_size]
+ const device T* weights, // [vocab_size, model_dim]
+ device T* output, // [batch_size, model_dim]
+ constant uint32_t& batch_size,
+ constant uint32_t& vocab_size,
+ constant uint32_t& model_dim,
+ constant float& input_scale,
+ uint dim_idx AXIS(model_dim, BLOCK_SIZE),
+ uint batch_idx AXIS(batch_size, 1)
) {
- const uint batch_idx = thread_position_in_grid / model_dim;
- const uint dim_idx = thread_position_in_grid % model_dim;
-
- if (batch_idx >= batch_size)
- return;
-
const uint64_t token_id = token_ids[batch_idx];
+ const uint output_idx = batch_idx * model_dim + dim_idx;
+
if (token_id >= vocab_size) {
- output[thread_position_in_grid] = T(0);
+ output[output_idx] = T(0);
return;
}
T value = weights[token_id * model_dim + dim_idx];
- output[thread_position_in_grid] = value * T(input_scale);
+ output[output_idx] = value * T(input_scale);
}
-
-template [[host_name("full_precision_embedding_lookup_f32")]] [[kernel]] void
-full_precision_embedding_lookup<float>(
- const device uint64_t* token_ids [[buffer(0)]],
- const device float* weights [[buffer(1)]],
- device float* output [[buffer(2)]],
- constant uint32_t& batch_size [[buffer(3)]],
- constant uint32_t& vocab_size [[buffer(4)]],
- constant uint32_t& model_dim [[buffer(5)]],
- constant float& input_scale [[buffer(6)]],
- uint thread_position_in_grid [[thread_position_in_grid]]
-);
-
-template [[host_name("full_precision_embedding_lookup_f16")]] [[kernel]] void
-full_precision_embedding_lookup<half>(
- const device uint64_t* token_ids [[buffer(0)]],
- const device half* weights [[buffer(1)]],
- device half* output [[buffer(2)]],
- constant uint32_t& batch_size [[buffer(3)]],
- constant uint32_t& vocab_size [[buffer(4)]],
- constant uint32_t& model_dim [[buffer(5)]],
- constant float& input_scale [[buffer(6)]],
- uint thread_position_in_grid [[thread_position_in_grid]]
-);
-
-template [[host_name("full_precision_embedding_lookup_bf16")]] [[kernel]] void
-full_precision_embedding_lookup<bfloat>(
- const device uint64_t* token_ids [[buffer(0)]],
- const device bfloat* weights [[buffer(1)]],
- device bfloat* output [[buffer(2)]],
- constant uint32_t& batch_size [[buffer(3)]],
- constant uint32_t& vocab_size [[buffer(4)]],
- constant uint32_t& model_dim [[buffer(5)]],
- constant float& input_scale [[buffer(6)]],
- uint thread_position_in_grid [[thread_position_in_grid]]
-);
diff --git a/crates/uzu/src/backends/metal/kernel/embedding/mod.rs b/crates/uzu/src/backends/metal/kernel/embedding/mod.rs
deleted file mode 100644
index 2aa2b2302..000000000
--- a/crates/uzu/src/backends/metal/kernel/embedding/mod.rs
+++ /dev/null
@@ -1,180 +0,0 @@
-use crate::{
- DataType,
- backends::metal::{
- ComputeEncoderSetValue, MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState,
- MTLContext, MTLSize, ProtocolObject, Retained, encodable_block::EmbeddingError,
- },
- config::QuantizationMode,
-};
-
-// ---- Full Precision Embedding Lookup Kernel ----
-
-pub struct FullPrecisionEmbeddingLookupKernel {
- pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-#[derive(Debug)]
-pub struct FullPrecisionEmbeddingLookupArguments<'a> {
- pub token_ids_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [batch_size] as U64
- pub weights_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [vocab_size, model_dim]
- pub output_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [batch_size, model_dim]
- pub batch_size: u32,
- pub vocab_size: u32,
- pub model_dim: u32,
- pub input_scale: f32,
-}
-
-impl FullPrecisionEmbeddingLookupKernel {
- pub fn new(
- mtl_context: &MTLContext,
- data_type: DataType,
- ) -> Result<Self, EmbeddingError> {
- let dtype_suffix = match data_type {
- DataType::F32 => "f32",
- DataType::F16 => "f16",
- DataType::BF16 => "bf16",
- other => {
- return Err(EmbeddingError::UnsupportedDataType(other));
- },
- };
- let kernel_name = format!("full_precision_embedding_lookup_{}", dtype_suffix);
-
- let pipeline = mtl_context
- .compute_pipeline_state(&kernel_name, None)
- .map_err(EmbeddingError::MetalError)?;
-
- Ok(Self { pipeline })
- }
-
- pub fn encode(
- &self,
- encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- args: FullPrecisionEmbeddingLookupArguments,
- ) -> Result<(), EmbeddingError> {
- encoder.set_compute_pipeline_state(&self.pipeline);
-
- MTLComputeCommandEncoder::set_buffer(
- encoder,
- Some(args.token_ids_buffer),
- 0,
- 0,
- );
- MTLComputeCommandEncoder::set_buffer(
- encoder,
- Some(args.weights_buffer),
- 0,
- 1,
- );
- MTLComputeCommandEncoder::set_buffer(
- encoder,
- Some(args.output_buffer),
- 0,
- 2,
- );
-
- encoder.set_value(&args.batch_size, 3);
- encoder.set_value(&args.vocab_size, 4);
- encoder.set_value(&args.model_dim, 5);
- encoder.set_value(&args.input_scale, 6);
-
- let total_threads = (args.batch_size * args.model_dim) as u64;
- let threads_per_threadgroup = 256u64;
- let threadgroups = (total_threads + threads_per_threadgroup - 1)
- / threads_per_threadgroup;
-
- MTLComputeCommandEncoder::dispatch_threadgroups(
- encoder,
- MTLSize::new(threadgroups as usize, 1, 1),
- MTLSize::new(threads_per_threadgroup as usize, 1, 1),
- );
-
- Ok(())
- }
-}
-
-// ---- Quantized Embedding Lookup Kernel ----
-
-pub struct QuantizedEmbeddingLookupKernel {
- pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-#[derive(Debug)]
-pub struct QuantizedEmbeddingLookupArguments<'a> {
- pub token_ids_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [batch_size] as U64
- pub weights_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [vocab_size, model_dim/packing_divisor] as U8/I8
- pub scales_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [vocab_size, num_groups]
- pub biases_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [vocab_size, num_groups]
- pub output_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [batch_size, model_dim]
- pub batch_size: u32,
- pub vocab_size: u32,
- pub model_dim: u32,
- pub group_size: u32,
- pub input_scale: f32,
-}
-
-impl QuantizedEmbeddingLookupKernel {
- pub fn new(
- mtl_context: &MTLContext,
- data_type: DataType,
- mode: QuantizationMode,
- ) -> Result<Self, EmbeddingError> {
- let dtype_suffix = match data_type {
- DataType::F32 => "f32",
- DataType::F16 => "f16",
- DataType::BF16 => "bf16",
- other => {
- return Err(EmbeddingError::UnsupportedDataType(other));
- },
- };
- let mode_suffix = match mode {
- QuantizationMode::UInt4 => "uint4",
- QuantizationMode::Int8 => "int8",
- QuantizationMode::UInt8 => "uint8",
- };
- let kernel_name = format!(
- "quantized_embedding_lookup_{}_{}",
- dtype_suffix, mode_suffix
- );
-
- let pipeline = mtl_context
- .compute_pipeline_state(&kernel_name, None)
- .map_err(EmbeddingError::MetalError)?;
-
- Ok(Self { pipeline })
- }
-
- pub fn encode(
- &self,
- encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- args: QuantizedEmbeddingLookupArguments,
- ) -> Result<(), EmbeddingError> {
- encoder.set_compute_pipeline_state(&self.pipeline);
-
- // Set buffers
- encoder.set_buffer(Some(args.token_ids_buffer), 0, 0);
- encoder.set_buffer(Some(args.weights_buffer), 0, 1);
- encoder.set_buffer(Some(args.scales_buffer), 0, 2);
- encoder.set_buffer(Some(args.biases_buffer), 0, 3);
- encoder.set_buffer(Some(args.output_buffer), 0, 4);
-
- // Set constants
- encoder.set_value(&args.batch_size, 5);
- encoder.set_value(&args.vocab_size, 6);
- encoder.set_value(&args.model_dim, 7);
- encoder.set_value(&args.group_size, 8);
- encoder.set_value(&args.input_scale, 9);
-
- // Dispatch one thread per output element
- let total_threads = (args.batch_size * args.model_dim) as u64;
- let threads_per_threadgroup = 256u64;
- let threadgroups = (total_threads + threads_per_threadgroup - 1)
- / threads_per_threadgroup;
-
- encoder.dispatch_threadgroups(
- MTLSize::new(threadgroups as usize, 1, 1),
- MTLSize::new(threads_per_threadgroup as usize, 1, 1),
- );
-
- Ok(())
- }
-}
diff --git a/crates/uzu/src/backends/metal/kernel/embedding/quant_embedding.metal b/crates/uzu/src/backends/metal/kernel/embedding/quant_embedding.metal
index 354bc6809..6152a10e6 100644
--- a/crates/uzu/src/backends/metal/kernel/embedding/quant_embedding.metal
+++ b/crates/uzu/src/backends/metal/kernel/embedding/quant_embedding.metal
@@ -1,28 +1,30 @@
#include <metal_stdlib>
-using namespace metal;
-
-template <typename T, uint PACKING_DIVISOR, bool SIGNED_STORAGE>
-[[kernel, max_total_threads_per_threadgroup(256)]]
-void quantized_embedding_lookup(
- const device uint64_t* token_ids [[buffer(0)]], // [batch_size]
- const device uint8_t* weights
- [[buffer(1)]], // [vocab_size, model_dim/packing_divisor] packed
- const device T* scales [[buffer(2)]], // [vocab_size, num_groups]
- const device T* biases [[buffer(3)]], // [vocab_size, num_groups]
- device T* output [[buffer(4)]], // [batch_size, model_dim]
- constant uint32_t& batch_size [[buffer(5)]],
- constant uint32_t& vocab_size [[buffer(6)]],
- constant uint32_t& model_dim [[buffer(7)]],
- constant uint32_t& group_size [[buffer(8)]],
- constant float& input_scale [[buffer(9)]],
- uint thread_position_in_grid [[thread_position_in_grid]]
+#include "../definitions.metal"
+
+enum QuantizationMode : uint {
+ QUANT_UINT4 = 0,
+ QUANT_INT8 = 1,
+ QUANT_UINT8 = 2
+};
+
+template <typename T>
+VARIANTS(T, float, half, bfloat)
+KERNEL(QuantizedEmbeddingLookup) (
+ const device uint64_t* token_ids, // [batch_size]
+ const device uint8_t* weights, // [vocab_size, model_dim/packing_divisor] packed
+ const device T* scales, // [vocab_size, num_groups]
+ const device T* biases, // [vocab_size, num_groups]
+ device T* output, // [batch_size, model_dim]
+ constant uint32_t& batch_size,
+ constant uint32_t& vocab_size,
+ constant uint32_t& model_dim,
+ constant uint32_t& group_size,
+ constant float& input_scale,
+ const constant uint& quant_mode,
+ const uint dim_idx AXIS(model_dim, 16),
+ const uint batch_idx AXIS(batch_size, 16)
) {
- const uint batch_idx = thread_position_in_grid / model_dim;
- const uint dim_idx = thread_position_in_grid % model_dim;
-
- if (batch_idx >= batch_size)
- return;
-
+ const uint thread_position_in_grid = batch_idx * model_dim + dim_idx;
const uint64_t token_id = token_ids[batch_idx];
if (token_id >= vocab_size) {
output[thread_position_in_grid] = T(0);
@@ -35,162 +37,28 @@ void quantized_embedding_lookup(
const T scale = scales[token_id * num_groups + group_idx];
const T bias = biases[token_id * num_groups + group_idx];
- const uint weights_stride = model_dim / PACKING_DIVISOR;
- const device uint8_t* weights_u8 = weights;
- const device int8_t* weights_i8 =
- reinterpret_cast<const device int8_t*>(weights);
+ const uint packing_divisor = quant_mode == QUANT_UINT4 ? 2 : 1;
+ const uint weights_stride = model_dim / packing_divisor;
int quantized_value = 0;
- if (PACKING_DIVISOR == 2) {
+ if (quant_mode == QUANT_UINT4) {
const uint byte_idx = token_id * weights_stride + (dim_idx / 2);
- const uint8_t packed = weights_u8[byte_idx];
+ const uint8_t packed = weights[byte_idx];
if ((dim_idx & 1) == 0) {
quantized_value = int(packed & 0x0F);
} else {
quantized_value = int((packed >> 4) & 0x0F);
}
+ } else if (quant_mode == QUANT_INT8) {
+ const uint elem_idx = token_id * weights_stride + dim_idx;
+ const device int8_t* weights_i8 = reinterpret_cast<const device int8_t*>(weights);
+ quantized_value = int(weights_i8[elem_idx]);
} else {
const uint elem_idx = token_id * weights_stride + dim_idx;
- quantized_value =
- SIGNED_STORAGE ? int(weights_i8[elem_idx]) : int(weights_u8[elem_idx]);
+ quantized_value = int(weights[elem_idx]);
}
float out_f = float(scale) * float(quantized_value) + float(bias);
out_f *= input_scale;
output[thread_position_in_grid] = T(out_f);
-}
-
-template [[host_name("quantized_embedding_lookup_f32_uint4")]] [[kernel]] void
-quantized_embedding_lookup<float, 2, false>(
- const device uint64_t* token_ids [[buffer(0)]],
- const device uint8_t* weights [[buffer(1)]],
- const device float* scales [[buffer(2)]],
- const device float* biases [[buffer(3)]],
- device float* output [[buffer(4)]],
- constant uint32_t& batch_size [[buffer(5)]],
- constant uint32_t& vocab_size [[buffer(6)]],
- constant uint32_t& model_dim [[buffer(7)]],
- constant uint32_t& group_size [[buffer(8)]],
- constant float& input_scale [[buffer(9)]],
- uint thread_position_in_grid [[thread_position_in_grid]]
-);
-
-template [[host_name("quantized_embedding_lookup_f16_uint4")]] [[kernel]] void
-quantized_embedding_lookup<half, 2, false>(
- const device uint64_t* token_ids [[buffer(0)]],
- const device uint8_t* weights [[buffer(1)]],
- const device half* scales [[buffer(2)]],
- const device half* biases [[buffer(3)]],
- device half* output [[buffer(4)]],
- constant uint32_t& batch_size [[buffer(5)]],
- constant uint32_t& vocab_size [[buffer(6)]],
- constant uint32_t& model_dim [[buffer(7)]],
- constant uint32_t& group_size [[buffer(8)]],
- constant float& input_scale [[buffer(9)]],
- uint thread_position_in_grid [[thread_position_in_grid]]
-);
-
-template [[host_name("quantized_embedding_lookup_bf16_uint4")]] [[kernel]] void
-quantized_embedding_lookup<bfloat, 2, false>(
- const device uint64_t* token_ids [[buffer(0)]],
- const device uint8_t* weights [[buffer(1)]],
- const device bfloat* scales [[buffer(2)]],
- const device bfloat* biases [[buffer(3)]],
- device bfloat* output [[buffer(4)]],
- constant uint32_t& batch_size [[buffer(5)]],
- constant uint32_t& vocab_size [[buffer(6)]],
- constant uint32_t& model_dim [[buffer(7)]],
- constant uint32_t& group_size [[buffer(8)]],
- constant float& input_scale [[buffer(9)]],
- uint thread_position_in_grid [[thread_position_in_grid]]
-);
-
-template [[host_name("quantized_embedding_lookup_f32_int8")]] [[kernel]] void
-quantized_embedding_lookup<float, 1, true>(
- const device uint64_t* token_ids [[buffer(0)]],
- const device uint8_t* weights [[buffer(1)]],
- const device float* scales [[buffer(2)]],
- const device float* biases [[buffer(3)]],
- device float* output [[buffer(4)]],
- constant uint32_t& batch_size [[buffer(5)]],
- constant uint32_t& vocab_size [[buffer(6)]],
- constant uint32_t& model_dim [[buffer(7)]],
- constant uint32_t& group_size [[buffer(8)]],
- constant float& input_scale [[buffer(9)]],
- uint thread_position_in_grid [[thread_position_in_grid]]
-);
-
-template [[host_name("quantized_embedding_lookup_f16_int8")]] [[kernel]] void
-quantized_embedding_lookup<half, 1, true>(
- const device uint64_t* token_ids [[buffer(0)]],
- const device uint8_t* weights [[buffer(1)]],
- const device half* scales [[buffer(2)]],
- const device half* biases [[buffer(3)]],
- device half* output [[buffer(4)]],
- constant uint32_t& batch_size [[buffer(5)]],
- constant uint32_t& vocab_size [[buffer(6)]],
- constant uint32_t& model_dim [[buffer(7)]],
- constant uint32_t& group_size [[buffer(8)]],
- constant float& input_scale [[buffer(9)]],
- uint thread_position_in_grid [[thread_position_in_grid]]
-);
-
-template [[host_name("quantized_embedding_lookup_bf16_int8")]] [[kernel]] void
-quantized_embedding_lookup<bfloat, 1, true>(
- const device uint64_t* token_ids [[buffer(0)]],
- const device uint8_t* weights [[buffer(1)]],
- const device bfloat* scales [[buffer(2)]],
- const device bfloat* biases [[buffer(3)]],
- device bfloat* output [[buffer(4)]],
- constant uint32_t& batch_size [[buffer(5)]],
- constant uint32_t& vocab_size [[buffer(6)]],
- constant uint32_t& model_dim [[buffer(7)]],
- constant uint32_t& group_size [[buffer(8)]],
- constant float& input_scale [[buffer(9)]],
- uint thread_position_in_grid [[thread_position_in_grid]]
-);
-
-template [[host_name("quantized_embedding_lookup_f32_uint8")]] [[kernel]] void
-quantized_embedding_lookup<float, 1, false>(
- const device uint64_t* token_ids [[buffer(0)]],
- const device uint8_t* weights [[buffer(1)]],
- const device float* scales [[buffer(2)]],
- const device float* biases [[buffer(3)]],
- device float* output [[buffer(4)]],
- constant uint32_t& batch_size [[buffer(5)]],
- constant uint32_t& vocab_size [[buffer(6)]],
- constant uint32_t& model_dim [[buffer(7)]],
- constant uint32_t& group_size [[buffer(8)]],
- constant float& input_scale [[buffer(9)]],
- uint thread_position_in_grid [[thread_position_in_grid]]
-);
-
-template [[host_name("quantized_embedding_lookup_f16_uint8")]] [[kernel]] void
-quantized_embedding_lookup<half, 1, false>(
- const device uint64_t* token_ids [[buffer(0)]],
- const device uint8_t* weights [[buffer(1)]],
- const device half* scales [[buffer(2)]],
- const device half* biases [[buffer(3)]],
- device half* output [[buffer(4)]],
- constant uint32_t& batch_size [[buffer(5)]],
- constant uint32_t& vocab_size [[buffer(6)]],
- constant uint32_t& model_dim [[buffer(7)]],
- constant uint32_t& group_size [[buffer(8)]],
- constant float& input_scale [[buffer(9)]],
- uint thread_position_in_grid [[thread_position_in_grid]]
-);
-
-template [[host_name("quantized_embedding_lookup_bf16_uint8")]] [[kernel]] void
-quantized_embedding_lookup<bfloat, 1, false>(
- const device uint64_t* token_ids [[buffer(0)]],
- const device uint8_t* weights [[buffer(1)]],
- const device bfloat* scales [[buffer(2)]],
- const device bfloat* biases [[buffer(3)]],
- device bfloat* output [[buffer(4)]],
- constant uint32_t& batch_size [[buffer(5)]],
- constant uint32_t& vocab_size [[buffer(6)]],
- constant uint32_t& model_dim [[buffer(7)]],
- constant uint32_t& group_size [[buffer(8)]],
- constant float& input_scale [[buffer(9)]],
- uint thread_position_in_grid [[thread_position_in_grid]]
-);
+}
\ No newline at end of file
diff --git a/crates/uzu/src/backends/metal/kernel/kv_cache_update/mod.rs b/crates/uzu/src/backends/metal/kernel/kv_cache_update/mod.rs
index faa44b9e9..a01d915f6 100644
--- a/crates/uzu/src/backends/metal/kernel/kv_cache_update/mod.rs
+++ b/crates/uzu/src/backends/metal/kernel/kv_cache_update/mod.rs
@@ -6,10 +6,13 @@ use super::{
super::MTLError, KernelDataType, MTLContext,
metal_extensions::ComputeEncoderDispatch,
};
-use crate::backends::metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandEncoderExt,
- MTLComputeCommandEncoder, MTLComputePipelineState, MTLDeviceExt,
- MTLResourceOptions, MTLSize, ProtocolObject, Retained,
+use crate::backends::{
+ common::Context,
+ metal::{
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandEncoderExt,
+ MTLComputeCommandEncoder, MTLComputePipelineState, MTLSize,
+ ProtocolObject, Retained,
+ },
};
#[derive(Clone, Copy, Debug)]
@@ -65,12 +68,8 @@ impl KVCacheUpdate {
.map_err(|e| KVCacheUpdateError::MetalError(e))?;
let indices_buffer = context
- .device
- .new_buffer(
- max_sequence_length * size_of::<Swap>(),
- MTLResourceOptions::STORAGE_MODE_SHARED,
- )
- .ok_or(KVCacheUpdateError::BufferCreationFailed)?;
+ .create_buffer(max_sequence_length * size_of::<Swap>())
+ .map_err(|_| KVCacheUpdateError::BufferCreationFailed)?;
Ok(Self {
pipeline_state,
diff --git a/crates/uzu/src/backends/metal/kernel/layer_norm/layer_norm.metal b/crates/uzu/src/backends/metal/kernel/layer_norm/layer_norm.metal
index 593d76575..2e84bebdc 100644
--- a/crates/uzu/src/backends/metal/kernel/layer_norm/layer_norm.metal
+++ b/crates/uzu/src/backends/metal/kernel/layer_norm/layer_norm.metal
@@ -1,8 +1,6 @@
#include <metal_stdlib>
#include "../definitions.metal"
-using namespace metal;
-
#define BLOCK_SIZE 1024
#define SIMD_SIZE 32
#define GRAIN_SIZE 4
@@ -112,54 +110,36 @@ void layer_norm_core(
}
}
-// X-macro table defining all supported type combinations (same as RMSNorm)
-#define FOREACH_LAYER_NORM_COMBO(_) \
- /* bfloat combinations most commonly used */ \
- _(bfloat, bfloat, bfloat, float, false, bf16_bf16_bf16_f32_norm) \
- _(bfloat, bfloat, bfloat, float, true, bf16_bf16_bf16_f32_full) \
- _(bfloat, float, bfloat, float, false, bf16_f32_bf16_f32_norm) \
- _(bfloat, float, bfloat, float, true, bf16_f32_bf16_f32_full) \
- /* float combinations */ \
- _(float, float, float, float, false, f32_f32_f32_f32_norm) \
- _(float, float, float, float, true, f32_f32_f32_f32_full) \
- /* half combinations */ \
- _(half, half, half, float, false, f16_f16_f16_f32_norm) \
- _(half, half, half, float, true, f16_f16_f16_f32_full)
-
-// Generate LayerNorm kernels
-#define DEFINE_LAYER_NORM_KERNEL(IN, SC, OUT, ACC, FULL_LAYER, SUF) \
- [[max_total_threads_per_threadgroup(1024)]] \
- kernel void layer_norm_##SUF( \
- const device IN* input [[buffer(0)]], \
- const device SC* scales [[buffer(1)]], \
- device OUT* output [[buffer(2)]], \
- constant uint& batch_size [[buffer(3)]], \
- constant uint& model_dim [[buffer(4)]], \
- constant float& epsilon [[buffer(5)]], \
- constant float& scale_offset [[buffer(6)]], \
- uint batch_idx [[threadgroup_position_in_grid]], \
- uint thread_in_row [[thread_position_in_threadgroup]] \
- ) { \
- if (batch_idx >= batch_size) \
- return; \
- \
- threadgroup ACC shared_mean[SIMD_SIZE]; \
- threadgroup ACC shared_variance[SIMD_SIZE]; \
- const uint input_offset = batch_idx * model_dim; \
- \
- layer_norm_core<IN, SC, OUT, ACC>( \
- input + input_offset, \
- scales, \
- output + input_offset, \
- model_dim, \
- epsilon, \
- scale_offset, \
- shared_mean, \
- shared_variance, \
- thread_in_row, \
- FULL_LAYER \
- ); \
- }
-
-FOREACH_LAYER_NORM_COMBO(DEFINE_LAYER_NORM_KERNEL)
-#undef DEFINE_LAYER_NORM_KERNEL
+template <typename IN, typename SC, typename OUT, typename ACC>
+VARIANTS(IN, float, half, bfloat)
+VARIANTS(SC, float, half, bfloat)
+VARIANTS(OUT, float, half, bfloat)
+VARIANTS(ACC, float)
+KERNEL(LayerNorm) (
+ const device IN* input,
+ const device SC* scales,
+ device OUT* output,
+ constant uint& batch_size,
+ constant uint& model_dim,
+ constant float& epsilon,
+ constant float& scale_offset,
+ constant uint& full_layer,
+ threadgroup ACC shared_mean[SIMD_SIZE],
+ threadgroup ACC shared_variance[SIMD_SIZE],
+ uint batch_idx GROUPS(batch_size),
+ uint thread_in_row THREADS(BLOCK_SIZE)
+) {
+ const uint input_offset = batch_idx * model_dim;
+ layer_norm_core<IN, SC, OUT, ACC>(
+ input + input_offset,
+ scales,
+ output + input_offset,
+ model_dim,
+ epsilon,
+ scale_offset,
+ shared_mean,
+ shared_variance,
+ thread_in_row,
+ full_layer
+ );
+}
diff --git a/crates/uzu/src/backends/metal/kernel/layer_norm/mod.rs b/crates/uzu/src/backends/metal/kernel/layer_norm/mod.rs
deleted file mode 100644
index 26ccbec10..000000000
--- a/crates/uzu/src/backends/metal/kernel/layer_norm/mod.rs
+++ /dev/null
@@ -1,137 +0,0 @@
-use crate::{
- DataType,
- backends::metal::{
- ComputeEncoderSetValue, MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState,
- MTLContext, MTLError, MTLSize, ProtocolObject, Retained,
- },
-};
-
-#[derive(Debug)]
-pub struct LayerNormArguments<'a> {
- pub input_buffer: &'a ProtocolObject<dyn MTLBuffer>,
- pub scales_buffer: &'a ProtocolObject<dyn MTLBuffer>,
- pub output_buffer: &'a ProtocolObject<dyn MTLBuffer>,
- pub batch_size: i32,
- pub model_dim: i32,
- pub epsilon: f32,
- pub scale_offset: f32,
-}
-
-#[derive(Debug, thiserror::Error)]
-pub enum LayerNormError {
- #[error("Metal error: {0}")]
- MetalError(#[from] MTLError),
- #[error(
- "Unsupported data type combination: input={input:?}, scales={scales:?}, output={output:?}, accumulation={accumulation:?}"
- )]
- UnsupportedDataType {
- input: DataType,
- scales: DataType,
- output: DataType,
- accumulation: DataType,
- },
-}
-
-pub struct LayerNormKernel {
- pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-impl LayerNormKernel {
- pub fn new(
- context: &MTLContext,
- input_data_type: DataType,
- scales_data_type: DataType,
- output_data_type: DataType,
- accumulation_data_type: DataType,
- full_layer: bool,
- ) -> Result<Self, LayerNormError> {
- let kernel_name = Self::kernel_name(
- input_data_type,
- scales_data_type,
- output_data_type,
- accumulation_data_type,
- full_layer,
- )?;
-
- let pipeline = context
- .compute_pipeline_state(&kernel_name, None)
- .map_err(LayerNormError::MetalError)?;
-
- Ok(Self {
- pipeline,
- })
- }
-
- fn kernel_name(
- input: DataType,
- scales: DataType,
- output: DataType,
- accumulation: DataType,
- full_layer: bool,
- ) -> Result<String, LayerNormError> {
- let input_str = Self::type_to_string(input)?;
- let scales_str = Self::type_to_string(scales)?;
- let output_str = Self::type_to_string(output)?;
- let accum_str = Self::type_to_string(accumulation)?;
- let mode_str = if full_layer {
- "full"
- } else {
- "norm"
- };
-
- Ok(format!(
- "layer_norm_{}_{}_{}_{}_{}",
- input_str, scales_str, output_str, accum_str, mode_str
- ))
- }
-
- fn type_to_string(
- data_type: DataType
- ) -> Result<&'static str, LayerNormError> {
- match data_type {
- DataType::F32 => Ok("f32"),
- DataType::F16 => Ok("f16"),
- DataType::BF16 => Ok("bf16"),
- _ => Err(LayerNormError::UnsupportedDataType {
- input: data_type,
- scales: data_type,
- output: data_type,
- accumulation: data_type,
- }),
- }
- }
-
- pub fn encode(
- &self,
- compute_encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- args: LayerNormArguments,
- ) -> Result<(), LayerNormError> {
- compute_encoder.set_compute_pipeline_state(&self.pipeline);
- compute_encoder.set_buffer(Some(args.input_buffer), 0, 0);
- compute_encoder.set_buffer(Some(args.scales_buffer), 0, 1);
- compute_encoder.set_buffer(Some(args.output_buffer), 0, 2);
- compute_encoder.set_value(&args.batch_size, 3);
- compute_encoder.set_value(&args.model_dim, 4);
- compute_encoder.set_value(&args.epsilon, 5);
- compute_encoder.set_value(&args.scale_offset, 6);
-
- let threadgroups_per_grid = MTLSize {
- width: args.batch_size as usize,
- height: 1,
- depth: 1,
- };
-
- let threads_per_threadgroup = MTLSize {
- width: 1024,
- height: 1,
- depth: 1,
- };
-
- compute_encoder.dispatch_threadgroups(
- threadgroups_per_grid,
- threads_per_threadgroup,
- );
-
- Ok(())
- }
-}
diff --git a/crates/uzu/src/backends/metal/kernel/mask_update/mask_update.metal b/crates/uzu/src/backends/metal/kernel/mask_update/mask_update.metal
index c00b250a7..6fe56237e 100644
--- a/crates/uzu/src/backends/metal/kernel/mask_update/mask_update.metal
+++ b/crates/uzu/src/backends/metal/kernel/mask_update/mask_update.metal
@@ -1,34 +1,17 @@
#include <metal_stdlib>
-using namespace metal;
+#include "../definitions.metal"
template <typename T>
-[[max_total_threads_per_threadgroup(1)]]
-kernel void update_attention_mask(
- device T* mask [[buffer(0)]],
- constant int& unmask_col [[buffer(1)]],
- constant int& mask_col [[buffer(2)]],
- uint tid [[thread_position_in_grid]]
+VARIANTS(T, float, half, bfloat)
+KERNEL(MaskUpdate) (
+ device T* mask,
+ constant int& unmask_col,
+ constant int& mask_col
) {
- if (tid == 0) {
- if (unmask_col >= 0) {
- mask[unmask_col] = T(0);
- }
- if (mask_col >= 0) {
- mask[mask_col] = -numeric_limits<T>::infinity();
- }
+ if (unmask_col >= 0) {
+ mask[unmask_col] = T(0);
+ }
+ if (mask_col >= 0) {
+ mask[mask_col] = -numeric_limits<T>::infinity();
}
}
-
-#define instantiate_mask_update(name, type) \
- template [[host_name("update_attention_mask_" #name)]] \
- kernel void update_attention_mask<type>( \
- device type * mask [[buffer(0)]], \
- constant int& unmask_col [[buffer(1)]], \
- constant int& mask_col [[buffer(2)]], \
- uint tid [[thread_position_in_grid]] \
- );
-
-instantiate_mask_update(float, float) instantiate_mask_update(
- half,
- half
-) instantiate_mask_update(bfloat, bfloat)
diff --git a/crates/uzu/src/backends/metal/kernel/mask_update/mod.rs b/crates/uzu/src/backends/metal/kernel/mask_update/mod.rs
deleted file mode 100644
index 1ec53b760..000000000
--- a/crates/uzu/src/backends/metal/kernel/mask_update/mod.rs
+++ /dev/null
@@ -1,55 +0,0 @@
-use std::ptr::NonNull;
-
-use crate::backends::metal::{
- MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext, MTLSize,
- ProtocolObject, Retained, kernel::KernelDataType,
-};
-
-pub struct MaskUpdateKernel {
- pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-impl MaskUpdateKernel {
- pub fn new(
- context: &MTLContext,
- data_type: KernelDataType,
- ) -> Result<Self, crate::backends::metal::error::MTLError> {
- let function_name = format!(
- "update_attention_mask_{}",
- data_type.function_name_suffix()
- );
- let pipeline = context.compute_pipeline_state(&function_name, None)?;
-
- Ok(Self {
- pipeline,
- })
- }
-
- pub fn encode(
- &self,
- encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- mask_buffer: &ProtocolObject<dyn MTLBuffer>,
- unmask_col: i32,
- mask_col: i32,
- ) {
- encoder.set_compute_pipeline_state(&self.pipeline);
- encoder.set_buffer(Some(mask_buffer), 0, 0);
- unsafe {
- encoder.set_bytes(
- NonNull::new_unchecked(
- &unmask_col as *const i32 as *mut std::ffi::c_void,
- ),
- std::mem::size_of::<i32>(),
- 1,
- );
- encoder.set_bytes(
- NonNull::new_unchecked(
- &mask_col as *const i32 as *mut std::ffi::c_void,
- ),
- std::mem::size_of::<i32>(),
- 2,
- );
- }
- encoder.dispatch_threads(MTLSize::new(1, 1, 1), MTLSize::new(1, 1, 1));
- }
-}
diff --git a/crates/uzu/src/backends/metal/kernel/matmul/common/gemm.h b/crates/uzu/src/backends/metal/kernel/matmul/common/gemm.h
index 4c825a159..39a4fc783 100644
--- a/crates/uzu/src/backends/metal/kernel/matmul/common/gemm.h
+++ b/crates/uzu/src/backends/metal/kernel/matmul/common/gemm.h
@@ -2,7 +2,7 @@
#include "loader.h"
#include "mma.h"
-#include "shared_types.h"
+#include "matmul.h"
using namespace metal;
diff --git a/crates/uzu/src/backends/metal/kernel/matmul/common/matmul_arguments.rs b/crates/uzu/src/backends/metal/kernel/matmul/common/matmul_arguments.rs
index 70f939ac6..8cda26600 100644
--- a/crates/uzu/src/backends/metal/kernel/matmul/common/matmul_arguments.rs
+++ b/crates/uzu/src/backends/metal/kernel/matmul/common/matmul_arguments.rs
@@ -1,14 +1,14 @@
-use crate::backends::metal::{MTLBuffer, ProtocolObject};
+use crate::backends::metal::{MTLBuffer, ProtocolObject, Retained};
#[derive(Debug, Clone)]
pub struct MatmulArguments<'a> {
- pub a: &'a ProtocolObject<dyn MTLBuffer>,
+ pub a: &'a Retained<ProtocolObject<dyn MTLBuffer>>,
/// Byte offset into `a` (used for slicing the batch dimension).
pub a_offset: u64,
- pub b: &'a ProtocolObject<dyn MTLBuffer>,
- pub c: Option<&'a ProtocolObject<dyn MTLBuffer>>,
- pub d: &'a ProtocolObject<dyn MTLBuffer>,
- pub bias: Option<&'a ProtocolObject<dyn MTLBuffer>>,
+ pub b: &'a Retained<ProtocolObject<dyn MTLBuffer>>,
+ pub c: Option<&'a Retained<ProtocolObject<dyn MTLBuffer>>>,
+ pub d: &'a Retained<ProtocolObject<dyn MTLBuffer>>,
+ pub bias: Option<&'a Retained<ProtocolObject<dyn MTLBuffer>>>,
/// M dimension - batch/number of tokens (rows of A, rows of D)
pub batch: i32,
/// K dimension - input_dim/reduction dimension (cols of A, rows of B)
diff --git a/crates/uzu/src/backends/metal/kernel/matmul/common/mod.rs b/crates/uzu/src/backends/metal/kernel/matmul/common/mod.rs
index 0ca76a5d9..d05d3f17f 100644
--- a/crates/uzu/src/backends/metal/kernel/matmul/common/mod.rs
+++ b/crates/uzu/src/backends/metal/kernel/matmul/common/mod.rs
@@ -1,10 +1,8 @@
mod matmul_arguments;
-#[allow(dead_code, clippy::unnecessary_operation, clippy::identity_op)]
-mod shared_types;
mod transpose_configuration;
-pub use matmul_arguments::MatmulArguments;
-pub use shared_types::{
+pub use crate::backends::common::gpu_types::{
GEMMAddMMParams, GEMMParams, GEMMSpiltKMlpFusedParams, GEMMSpiltKParams,
};
+pub use matmul_arguments::MatmulArguments;
pub(crate) use transpose_configuration::transpose_configuration;
diff --git a/crates/uzu/src/backends/metal/kernel/matmul/common/shared_types.h b/crates/uzu/src/backends/metal/kernel/matmul/common/shared_types.h
deleted file mode 100644
index ad116a541..000000000
--- a/crates/uzu/src/backends/metal/kernel/matmul/common/shared_types.h
+++ /dev/null
@@ -1,108 +0,0 @@
-// Shared types for GEMM kernel - used by both Metal shaders and Rust (via
-// bindgen). This header must be C-compatible for bindgen to parse it.
-// Keep an include guard to avoid clang's "pragma once in main file" warning.
-#ifndef UZU_MATMUL_SHARED_TYPES_H
-#define UZU_MATMUL_SHARED_TYPES_H
-
-// Metal and C have different type systems
-// __METAL_VERSION__ is defined when compiling with the Metal shader compiler
-#ifdef __METAL_VERSION__
-// Metal shading language - int64_t comes from metal_stdlib
-#include <metal_stdlib>
-using namespace metal;
-
-namespace uzu {
-namespace matmul {
-
-#else
-// C/C++ for bindgen - use stdint types
-#include <stdint.h>
-
-#endif
-
-///////////////////////////////////////////////////////////////////////////////
-// GEMM Parameters (mirrors steel/gemm/params.h, but C-compatible)
-///////////////////////////////////////////////////////////////////////////////
-
-// Main GEMM parameters
-struct GEMMParams {
- int M;
- int N;
- int K;
-
- int lda;
- int ldb;
- int ldd;
-
- int tiles_n;
- int tiles_m;
-
- int64_t batch_stride_a;
- int64_t batch_stride_b;
- int64_t batch_stride_d;
-
- int swizzle_log;
- int gemm_k_iterations_aligned;
-
- int batch_ndim;
-};
-
-// Split-K GEMM parameters
-struct GEMMSpiltKParams {
- int M;
- int N;
- int K;
-
- int lda;
- int ldb;
- int ldc;
-
- int tiles_n;
- int tiles_m;
-
- int split_k_partitions;
- int split_k_partition_stride;
- int split_k_partition_size;
-
- int gemm_k_iterations_aligned;
-};
-
-// Split-K MLP Fused GEMM parameters
-struct GEMMSpiltKMlpFusedParams {
- int M;
- int N;
- int K;
-
- int lda;
- int ldb;
- int ldc;
-
- int tiles_n;
- int tiles_m;
-
- int split_k_partitions;
- int split_k_partition_stride;
- int split_k_partition_size;
-
- int gemm_k_iterations_aligned;
- int hidden_dim;
-};
-
-// AddMM parameters (alpha * A @ B + beta * C)
-struct GEMMAddMMParams {
- int ldc;
- int fdc;
-
- int64_t batch_stride_c;
-
- float alpha;
- float beta;
-};
-
-// Close Metal namespace
-#ifdef __METAL_VERSION__
-} // namespace matmul
-} // namespace uzu
-#endif
-
-#endif // UZU_MATMUL_SHARED_TYPES_H
diff --git a/crates/uzu/src/backends/metal/kernel/matmul/common/shared_types.rs b/crates/uzu/src/backends/metal/kernel/matmul/common/shared_types.rs
deleted file mode 100644
index 3e4c612d2..000000000
--- a/crates/uzu/src/backends/metal/kernel/matmul/common/shared_types.rs
+++ /dev/null
@@ -1,204 +0,0 @@
-/* automatically generated by rust-bindgen 0.72.1 */
-
-#![allow(non_snake_case)]
-
-#[repr(C)]
-#[derive(Debug, Default, Copy, Clone)]
-pub struct GEMMParams {
- pub M: ::core::ffi::c_int,
- pub N: ::core::ffi::c_int,
- pub K: ::core::ffi::c_int,
- pub lda: ::core::ffi::c_int,
- pub ldb: ::core::ffi::c_int,
- pub ldd: ::core::ffi::c_int,
- pub tiles_n: ::core::ffi::c_int,
- pub tiles_m: ::core::ffi::c_int,
- pub batch_stride_a: i64,
- pub batch_stride_b: i64,
- pub batch_stride_d: i64,
- pub swizzle_log: ::core::ffi::c_int,
- pub gemm_k_iterations_aligned: ::core::ffi::c_int,
- pub batch_ndim: ::core::ffi::c_int,
-}
-#[allow(clippy::unnecessary_operation, clippy::identity_op)]
-const _: () = {
- ["Size of GEMMParams"][::core::mem::size_of::<GEMMParams>() - 72usize];
- ["Alignment of GEMMParams"][::core::mem::align_of::<GEMMParams>() - 8usize];
- ["Offset of field: GEMMParams::M"]
- [::core::mem::offset_of!(GEMMParams, M) - 0usize];
- ["Offset of field: GEMMParams::N"]
- [::core::mem::offset_of!(GEMMParams, N) - 4usize];
- ["Offset of field: GEMMParams::K"]
- [::core::mem::offset_of!(GEMMParams, K) - 8usize];
- ["Offset of field: GEMMParams::lda"]
- [::core::mem::offset_of!(GEMMParams, lda) - 12usize];
- ["Offset of field: GEMMParams::ldb"]
- [::core::mem::offset_of!(GEMMParams, ldb) - 16usize];
- ["Offset of field: GEMMParams::ldd"]
- [::core::mem::offset_of!(GEMMParams, ldd) - 20usize];
- ["Offset of field: GEMMParams::tiles_n"]
- [::core::mem::offset_of!(GEMMParams, tiles_n) - 24usize];
- ["Offset of field: GEMMParams::tiles_m"]
- [::core::mem::offset_of!(GEMMParams, tiles_m) - 28usize];
- ["Offset of field: GEMMParams::batch_stride_a"]
- [::core::mem::offset_of!(GEMMParams, batch_stride_a) - 32usize];
- ["Offset of field: GEMMParams::batch_stride_b"]
- [::core::mem::offset_of!(GEMMParams, batch_stride_b) - 40usize];
- ["Offset of field: GEMMParams::batch_stride_d"]
- [::core::mem::offset_of!(GEMMParams, batch_stride_d) - 48usize];
- ["Offset of field: GEMMParams::swizzle_log"]
- [::core::mem::offset_of!(GEMMParams, swizzle_log) - 56usize];
- ["Offset of field: GEMMParams::gemm_k_iterations_aligned"][::core::mem::offset_of!(
- GEMMParams,
- gemm_k_iterations_aligned
- ) - 60usize];
- ["Offset of field: GEMMParams::batch_ndim"]
- [::core::mem::offset_of!(GEMMParams, batch_ndim) - 64usize];
-};
-#[repr(C)]
-#[derive(Debug, Default, Copy, Clone)]
-pub struct GEMMSpiltKParams {
- pub M: ::core::ffi::c_int,
- pub N: ::core::ffi::c_int,
- pub K: ::core::ffi::c_int,
- pub lda: ::core::ffi::c_int,
- pub ldb: ::core::ffi::c_int,
- pub ldc: ::core::ffi::c_int,
- pub tiles_n: ::core::ffi::c_int,
- pub tiles_m: ::core::ffi::c_int,
- pub split_k_partitions: ::core::ffi::c_int,
- pub split_k_partition_stride: ::core::ffi::c_int,
- pub split_k_partition_size: ::core::ffi::c_int,
- pub gemm_k_iterations_aligned: ::core::ffi::c_int,
-}
-#[allow(clippy::unnecessary_operation, clippy::identity_op)]
-const _: () = {
- ["Size of GEMMSpiltKParams"]
- [::core::mem::size_of::<GEMMSpiltKParams>() - 48usize];
- ["Alignment of GEMMSpiltKParams"]
- [::core::mem::align_of::<GEMMSpiltKParams>() - 4usize];
- ["Offset of field: GEMMSpiltKParams::M"]
- [::core::mem::offset_of!(GEMMSpiltKParams, M) - 0usize];
- ["Offset of field: GEMMSpiltKParams::N"]
- [::core::mem::offset_of!(GEMMSpiltKParams, N) - 4usize];
- ["Offset of field: GEMMSpiltKParams::K"]
- [::core::mem::offset_of!(GEMMSpiltKParams, K) - 8usize];
- ["Offset of field: GEMMSpiltKParams::lda"]
- [::core::mem::offset_of!(GEMMSpiltKParams, lda) - 12usize];
- ["Offset of field: GEMMSpiltKParams::ldb"]
- [::core::mem::offset_of!(GEMMSpiltKParams, ldb) - 16usize];
- ["Offset of field: GEMMSpiltKParams::ldc"]
- [::core::mem::offset_of!(GEMMSpiltKParams, ldc) - 20usize];
- ["Offset of field: GEMMSpiltKParams::tiles_n"]
- [::core::mem::offset_of!(GEMMSpiltKParams, tiles_n) - 24usize];
- ["Offset of field: GEMMSpiltKParams::tiles_m"]
- [::core::mem::offset_of!(GEMMSpiltKParams, tiles_m) - 28usize];
- ["Offset of field: GEMMSpiltKParams::split_k_partitions"][::core::mem::offset_of!(
- GEMMSpiltKParams,
- split_k_partitions
- ) - 32usize];
- ["Offset of field: GEMMSpiltKParams::split_k_partition_stride"][::core::mem::offset_of!(
- GEMMSpiltKParams,
- split_k_partition_stride
- )
- - 36usize];
- ["Offset of field: GEMMSpiltKParams::split_k_partition_size"][::core::mem::offset_of!(
- GEMMSpiltKParams,
- split_k_partition_size
- ) - 40usize];
- ["Offset of field: GEMMSpiltKParams::gemm_k_iterations_aligned"][::core::mem::offset_of!(
- GEMMSpiltKParams,
- gemm_k_iterations_aligned
- )
- - 44usize];
-};
-#[repr(C)]
-#[derive(Debug, Default, Copy, Clone)]
-pub struct GEMMSpiltKMlpFusedParams {
- pub M: ::core::ffi::c_int,
- pub N: ::core::ffi::c_int,
- pub K: ::core::ffi::c_int,
- pub lda: ::core::ffi::c_int,
- pub ldb: ::core::ffi::c_int,
- pub ldc: ::core::ffi::c_int,
- pub tiles_n: ::core::ffi::c_int,
- pub tiles_m: ::core::ffi::c_int,
- pub split_k_partitions: ::core::ffi::c_int,
- pub split_k_partition_stride: ::core::ffi::c_int,
- pub split_k_partition_size: ::core::ffi::c_int,
- pub gemm_k_iterations_aligned: ::core::ffi::c_int,
- pub hidden_dim: ::core::ffi::c_int,
-}
-#[allow(clippy::unnecessary_operation, clippy::identity_op)]
-const _: () = {
- ["Size of GEMMSpiltKMlpFusedParams"]
- [::core::mem::size_of::<GEMMSpiltKMlpFusedParams>() - 52usize];
- ["Alignment of GEMMSpiltKMlpFusedParams"]
- [::core::mem::align_of::<GEMMSpiltKMlpFusedParams>() - 4usize];
- ["Offset of field: GEMMSpiltKMlpFusedParams::M"]
- [::core::mem::offset_of!(GEMMSpiltKMlpFusedParams, M) - 0usize];
- ["Offset of field: GEMMSpiltKMlpFusedParams::N"]
- [::core::mem::offset_of!(GEMMSpiltKMlpFusedParams, N) - 4usize];
- ["Offset of field: GEMMSpiltKMlpFusedParams::K"]
- [::core::mem::offset_of!(GEMMSpiltKMlpFusedParams, K) - 8usize];
- ["Offset of field: GEMMSpiltKMlpFusedParams::lda"]
- [::core::mem::offset_of!(GEMMSpiltKMlpFusedParams, lda) - 12usize];
- ["Offset of field: GEMMSpiltKMlpFusedParams::ldb"]
- [::core::mem::offset_of!(GEMMSpiltKMlpFusedParams, ldb) - 16usize];
- ["Offset of field: GEMMSpiltKMlpFusedParams::ldc"]
- [::core::mem::offset_of!(GEMMSpiltKMlpFusedParams, ldc) - 20usize];
- ["Offset of field: GEMMSpiltKMlpFusedParams::tiles_n"]
- [::core::mem::offset_of!(GEMMSpiltKMlpFusedParams, tiles_n) - 24usize];
- ["Offset of field: GEMMSpiltKMlpFusedParams::tiles_m"]
- [::core::mem::offset_of!(GEMMSpiltKMlpFusedParams, tiles_m) - 28usize];
- ["Offset of field: GEMMSpiltKMlpFusedParams::split_k_partitions"][::core::mem::offset_of!(
- GEMMSpiltKMlpFusedParams,
- split_k_partitions
- )
- - 32usize];
- ["Offset of field: GEMMSpiltKMlpFusedParams::split_k_partition_stride"][::core::mem::offset_of!(
- GEMMSpiltKMlpFusedParams,
- split_k_partition_stride
- )
- - 36usize];
- ["Offset of field: GEMMSpiltKMlpFusedParams::split_k_partition_size"][::core::mem::offset_of!(
- GEMMSpiltKMlpFusedParams,
- split_k_partition_size
- )
- - 40usize];
- ["Offset of field: GEMMSpiltKMlpFusedParams::gemm_k_iterations_aligned"][::core::mem::offset_of!(
- GEMMSpiltKMlpFusedParams,
- gemm_k_iterations_aligned
- )
- - 44usize];
- ["Offset of field: GEMMSpiltKMlpFusedParams::hidden_dim"][::core::mem::offset_of!(
- GEMMSpiltKMlpFusedParams,
- hidden_dim
- ) - 48usize];
-};
-#[repr(C)]
-#[derive(Debug, Default, Copy, Clone)]
-pub struct GEMMAddMMParams {
- pub ldc: ::core::ffi::c_int,
- pub fdc: ::core::ffi::c_int,
- pub batch_stride_c: i64,
- pub alpha: f32,
- pub beta: f32,
-}
-#[allow(clippy::unnecessary_operation, clippy::identity_op)]
-const _: () = {
- ["Size of GEMMAddMMParams"]
- [::core::mem::size_of::<GEMMAddMMParams>() - 24usize];
- ["Alignment of GEMMAddMMParams"]
- [::core::mem::align_of::<GEMMAddMMParams>() - 8usize];
- ["Offset of field: GEMMAddMMParams::ldc"]
- [::core::mem::offset_of!(GEMMAddMMParams, ldc) - 0usize];
- ["Offset of field: GEMMAddMMParams::fdc"]
- [::core::mem::offset_of!(GEMMAddMMParams, fdc) - 4usize];
- ["Offset of field: GEMMAddMMParams::batch_stride_c"]
- [::core::mem::offset_of!(GEMMAddMMParams, batch_stride_c) - 8usize];
- ["Offset of field: GEMMAddMMParams::alpha"]
- [::core::mem::offset_of!(GEMMAddMMParams, alpha) - 16usize];
- ["Offset of field: GEMMAddMMParams::beta"]
- [::core::mem::offset_of!(GEMMAddMMParams, beta) - 20usize];
-};
diff --git a/crates/uzu/src/backends/metal/kernel/matmul/common/steel/utils/integral_constant.h b/crates/uzu/src/backends/metal/kernel/matmul/common/steel/utils/integral_constant.h
index 1ad59d9a7..5cf7e9645 100644
--- a/crates/uzu/src/backends/metal/kernel/matmul/common/steel/utils/integral_constant.h
+++ b/crates/uzu/src/backends/metal/kernel/matmul/common/steel/utils/integral_constant.h
@@ -1,9 +1,8 @@
-
-
#pragma once
#include <metal_stdlib>
#include "type_traits.h"
+#include "../../../../definitions.metal"
#pragma METAL internals : enable
@@ -104,7 +103,7 @@ void dispatch_bool(bool v, F f) {
template <int start, int stop, int step, typename F>
constexpr void const_for_loop(F f) {
- if constexpr (start < stop) {
+ IF_CONSTEXPR(start < stop) {
constexpr auto idx = Int<start>{};
f(idx);
const_for_loop<start + step, stop, step, F>(f);
diff --git a/crates/uzu/src/backends/metal/kernel/matmul/gemm/kernel.rs b/crates/uzu/src/backends/metal/kernel/matmul/gemm/kernel.rs
index 997f4fa9c..d115da827 100644
--- a/crates/uzu/src/backends/metal/kernel/matmul/gemm/kernel.rs
+++ b/crates/uzu/src/backends/metal/kernel/matmul/gemm/kernel.rs
@@ -7,15 +7,19 @@ use super::{
use crate::{
DataType,
backends::metal::{
- ComputeEncoderSetValue, MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext,
- MTLError, MTLFunctionConstantValues, ProtocolObject, Retained,
+ ComputeEncoderSetValue, MTLComputeCommandEncoder,
+ MTLComputePipelineState, MTLContext, MTLError,
+ MTLFunctionConstantValues, ProtocolObject, Retained,
kernel::matmul::common::{MatmulArguments, transpose_configuration},
},
};
pub struct Kernel {
data_type: DataType,
- pipelines: HashMap<PipelineConfiguration, Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
+ pipelines: HashMap<
+ PipelineConfiguration,
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ >,
}
impl Kernel {
@@ -131,7 +135,8 @@ impl Kernel {
&mut self,
context: &MTLContext,
configuration: &PipelineConfiguration,
- ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError> {
+ ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError>
+ {
if !self.pipelines.contains_key(configuration) {
let kernel_name = self.kernel_name(configuration);
let function_constants = MTLFunctionConstantValues::new();
diff --git a/crates/uzu/src/backends/metal/kernel/matmul/gemv/kernel.rs b/crates/uzu/src/backends/metal/kernel/matmul/gemv/kernel.rs
index c315a19d8..a6135614b 100644
--- a/crates/uzu/src/backends/metal/kernel/matmul/gemv/kernel.rs
+++ b/crates/uzu/src/backends/metal/kernel/matmul/gemv/kernel.rs
@@ -7,8 +7,9 @@ use super::{
use crate::{
DataType,
backends::metal::{
- ComputeEncoderSetValue, MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext,
- MTLError, ProtocolObject, Retained, kernel::matmul::common::MatmulArguments,
+ ComputeEncoderSetValue, MTLComputeCommandEncoder,
+ MTLComputePipelineState, MTLContext, MTLError, ProtocolObject,
+ Retained, kernel::matmul::common::MatmulArguments,
},
};
@@ -49,7 +50,10 @@ fn gemv_kernel_name(
pub struct Kernel {
data_type: DataType,
- pipelines: HashMap<PipelineConfiguration, Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
+ pipelines: HashMap<
+ PipelineConfiguration,
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ >,
}
impl Kernel {
@@ -102,7 +106,8 @@ impl Kernel {
&mut self,
context: &MTLContext,
config: &PipelineConfiguration,
- ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError> {
+ ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError>
+ {
if !self.pipelines.contains_key(config) {
let kernel_name = gemv_kernel_name(self.data_type, config)?;
let pipeline =
diff --git a/crates/uzu/src/backends/metal/kernel/matmul/kernel.rs b/crates/uzu/src/backends/metal/kernel/matmul/kernel.rs
index 745e0a50d..4b6436e15 100644
--- a/crates/uzu/src/backends/metal/kernel/matmul/kernel.rs
+++ b/crates/uzu/src/backends/metal/kernel/matmul/kernel.rs
@@ -1,5 +1,4 @@
use super::{
- super::{KernelDataType, TensorAddBias},
common::MatmulArguments,
dispatch_descriptor::{
MatmulDispatchDescriptor, choose_dispatch_descriptor,
@@ -10,9 +9,12 @@ use super::{
};
use crate::{
DataType,
- backends::metal::{
- MTLBuffer, MTLComputeCommandEncoder, MTLContext, MTLError,
- ProtocolObject,
+ backends::{
+ common::kernel::TensorAddBiasKernel,
+ metal::{
+ MTLBuffer, MTLComputeCommandEncoder, MTLContext, MTLError,
+ ProtocolObject, Retained, kernel::dsl::TensorAddBiasMetalKernel,
+ },
},
};
@@ -21,7 +23,7 @@ pub struct MatmulKernel {
gemm: Option<gemm::GemmKernel>,
gemv: Option<GemvKernel>,
splitk: Option<SplitKGemm>,
- bias_add: Option<TensorAddBias>,
+ bias_add: Option<TensorAddBiasMetalKernel>,
}
impl MatmulKernel {
@@ -137,7 +139,7 @@ impl MatmulKernel {
context: &MTLContext,
encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
arguments: &MatmulArguments,
- bias: &ProtocolObject<dyn MTLBuffer>,
+ bias: &Retained<ProtocolObject<dyn MTLBuffer>>,
) -> Result<(), MTLError> {
let m = arguments.batch as usize;
let n = arguments.output_dim as usize;
@@ -148,20 +150,17 @@ impl MatmulKernel {
}
if self.bias_add.is_none() {
- self.bias_add = Some(TensorAddBias::new(
- context,
- KernelDataType::from(self.data_type),
- )?);
+ self.bias_add =
+ Some(TensorAddBiasMetalKernel::new(context, self.data_type)?);
}
let bias_add = self.bias_add.as_ref().unwrap();
- bias_add.encode_with_encoder(
+ bias_add.encode(
arguments.d,
bias,
arguments.d,
- n,
- total_len,
+ n as u32,
+ total_len as u32,
encoder,
- None,
);
Ok(())
}
diff --git a/crates/uzu/src/backends/metal/kernel/matmul/split_k/kernel.rs b/crates/uzu/src/backends/metal/kernel/matmul/split_k/kernel.rs
index 7a768f81f..e08b03516 100644
--- a/crates/uzu/src/backends/metal/kernel/matmul/split_k/kernel.rs
+++ b/crates/uzu/src/backends/metal/kernel/matmul/split_k/kernel.rs
@@ -1,19 +1,31 @@
use std::collections::HashMap;
-use super::{DispatchDescriptor, pipeline_configuration::PipelineConfiguration};
+use super::{
+ DispatchDescriptor, pipeline_configuration::PipelineConfiguration,
+};
use crate::{
DataType,
- backends::metal::{
- ComputeEncoderSetValue, MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState,
- MTLContext, MTLDeviceExt, MTLError, MTLResourceOptions, ProtocolObject, Retained,
- kernel::matmul::common::{MatmulArguments, transpose_configuration},
+ backends::{
+ common::Context,
+ metal::{
+ ComputeEncoderSetValue, MTLBuffer, MTLComputeCommandEncoder,
+ MTLComputePipelineState, MTLContext, MTLError, ProtocolObject,
+ Retained,
+ kernel::matmul::common::{
+ MatmulArguments, transpose_configuration,
+ },
+ },
},
};
pub struct Kernel {
data_type: DataType,
- partial_pipelines: HashMap<PipelineConfiguration, Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
- accum_pipeline: Option<Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
+ partial_pipelines: HashMap<
+ PipelineConfiguration,
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ >,
+ accum_pipeline:
+ Option<Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
accumulator_buffer: Option<Retained<ProtocolObject<dyn MTLBuffer>>>,
accumulator_buffer_bytes: usize,
}
@@ -158,7 +170,8 @@ impl Kernel {
&mut self,
mtl: &MTLContext,
config: &PipelineConfiguration,
- ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError> {
+ ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError>
+ {
if !self.partial_pipelines.contains_key(config) {
let name = self.partial_kernel_name(config)?;
let ps = mtl.compute_pipeline_state(&name, None)?;
@@ -170,7 +183,8 @@ impl Kernel {
fn get_accum_pipeline(
&mut self,
mtl: &MTLContext,
- ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError> {
+ ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError>
+ {
if self.accum_pipeline.is_none() {
let name = self.accum_kernel_name()?;
let ps = mtl.compute_pipeline_state(&name, None)?;
@@ -257,11 +271,7 @@ impl Kernel {
return;
}
self.accumulator_buffer = Some(
- mtl.device
- .new_buffer(
- required_bytes as usize,
- MTLResourceOptions::STORAGE_MODE_PRIVATE,
- )
+ mtl.create_buffer(required_bytes)
.expect("Failed to create accumulator buffer"),
);
self.accumulator_buffer_bytes = required_bytes;
diff --git a/crates/uzu/src/backends/metal/kernel/media_kernels/mod.rs b/crates/uzu/src/backends/metal/kernel/media_kernels/mod.rs
index dd8472031..7a945dd23 100644
--- a/crates/uzu/src/backends/metal/kernel/media_kernels/mod.rs
+++ b/crates/uzu/src/backends/metal/kernel/media_kernels/mod.rs
@@ -1,10 +1,11 @@
use std::{mem, ptr::NonNull};
+use crate::backends::metal::Metal;
use objc2::rc::Retained;
use crate::backends::metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoderExt,
- MTLCommandEncoder, MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext, MTLSize,
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandEncoderExt,
+ MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext, MTLSize,
ProtocolObject,
error::MTLError,
forward_pass::{EncodableBlock, EncodingParameters, ForwardPassState},
@@ -38,7 +39,8 @@ pub struct ScalePadNormalizeImage {
impl ScalePadNormalizeImage {
pub fn new(context: &MTLContext) -> Result<Self, MTLError> {
let function_name = "scalePadNormalizeImage";
- let pipeline_state = context.compute_pipeline_state(&function_name, None)?;
+ let pipeline_state =
+ context.compute_pipeline_state(&function_name, None)?;
Ok(Self {
pipeline_state,
})
@@ -49,7 +51,7 @@ impl ScalePadNormalizeImage {
input_image: &Image,
output_image: &Image,
image_params_ptr: *const std::ffi::c_void,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
) {
let compute_encoder = command_buffer
.new_compute_command_encoder()
@@ -78,11 +80,11 @@ impl ScalePadNormalizeImage {
}
}
-impl EncodableBlock for ScalePadNormalizeImage {
+impl EncodableBlock<Metal> for ScalePadNormalizeImage {
fn encode(
&self,
_state: &mut ForwardPassState,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
_parameters: &EncodingParameters,
) {
let _ = command_buffer;
@@ -112,7 +114,8 @@ pub struct ExtractImagePatches {
impl ExtractImagePatches {
pub fn new(context: &MTLContext) -> Result<Self, MTLError> {
let function_name = "extractImagePatches";
- let pipeline_state = context.compute_pipeline_state(&function_name, None)?;
+ let pipeline_state =
+ context.compute_pipeline_state(&function_name, None)?;
Ok(Self {
pipeline_state,
})
@@ -121,9 +124,9 @@ impl ExtractImagePatches {
pub fn encode_internal(
&self,
padded_normalized_image: &Image,
- output_buffer_mtl: &ProtocolObject<dyn MTLBuffer>,
+ output_buffer_mtl: &Retained<ProtocolObject<dyn MTLBuffer>>,
patch_params_ptr: *const std::ffi::c_void,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
) {
let compute_encoder = command_buffer
.new_compute_command_encoder()
diff --git a/crates/uzu/src/backends/metal/kernel/mlp/activation_mul.metal b/crates/uzu/src/backends/metal/kernel/mlp/activation_mul.metal
index 81c5b1a25..521cf3ca0 100644
--- a/crates/uzu/src/backends/metal/kernel/mlp/activation_mul.metal
+++ b/crates/uzu/src/backends/metal/kernel/mlp/activation_mul.metal
@@ -1,84 +1,23 @@
#include <metal_stdlib>
#include "../definitions.metal"
-using namespace metal;
-enum ActivationType : ushort {
- ACT_SILU = 0,
- ACT_GELU = 1,
-};
+#include "../activation/activation.h"
template <typename T>
-inline T silu(T x) {
- float xf = float(x);
- float yf = xf / (1.0f + exp(-xf));
- return T(yf);
-}
-
-template <typename T>
-inline T gelu_approx(T x) {
- constexpr float k0 = 0.044715f;
- constexpr float k1 = 0.7978845608f; // sqrt(2/pi)
- float xf = float(x);
- float t = k1 * (xf + k0 * xf * xf * xf);
- float yf = 0.5f * xf * (1.0f + metal::precise::tanh(t));
- return T(yf);
-}
-
-template <typename T>
-inline T activate(T x, ushort act) {
- if (act == ACT_SILU)
- return silu(x);
- return gelu_approx(x);
-}
-
-template <typename T>
-[[kernel, max_total_threads_per_threadgroup(64)]] void mlp_activation_mul(
- const device T* fused_up [[buffer(0)]],
- device T* hidden [[buffer(1)]],
- const constant int& H [[buffer(2)]],
- const constant int& M [[buffer(3)]],
- const constant ushort& act_type [[buffer(4)]],
- uint2 tid [[thread_position_in_grid]]
+VARIANTS(T, float, half, bfloat)
+KERNEL(MlpGateActMul) (
+ const device T* fused_up,
+ device T* hidden,
+ const constant int& h,
+ const constant int& m,
+ const constant uint& act_type,
+ uint j AXIS(h, 64),
+ uint row AXIS(m, 1)
) {
- int row = tid.y;
- int j = tid.x;
- if (row >= M || j >= H)
- return;
- int base = row * (2 * H);
+ int base = row * (2 * h);
T up = fused_up[base + j];
- T gate = fused_up[base + H + j];
+ T gate = fused_up[base + h + j];
T g = activate(gate, act_type);
float out_f = float(up) * float(g);
- hidden[row * H + j] = T(out_f);
+ hidden[row * h + j] = T(out_f);
}
-
-// Explicit instantiations with stable host names
-template [[host_name("mlp_activation_mul_f16")]] [[kernel]] void
-mlp_activation_mul<half>(
- const device half* fused_up [[buffer(0)]],
- device half* hidden [[buffer(1)]],
- const constant int& H [[buffer(2)]],
- const constant int& M [[buffer(3)]],
- const constant ushort& act_type [[buffer(4)]],
- uint2 tid [[thread_position_in_grid]]
-);
-
-template [[host_name("mlp_activation_mul_f32")]] [[kernel]] void
-mlp_activation_mul<float>(
- const device float* fused_up [[buffer(0)]],
- device float* hidden [[buffer(1)]],
- const constant int& H [[buffer(2)]],
- const constant int& M [[buffer(3)]],
- const constant ushort& act_type [[buffer(4)]],
- uint2 tid [[thread_position_in_grid]]
-);
-
-template [[host_name("mlp_activation_mul_bf16")]] [[kernel]] void
-mlp_activation_mul<bfloat>(
- const device bfloat* fused_up [[buffer(0)]],
- device bfloat* hidden [[buffer(1)]],
- const constant int& H [[buffer(2)]],
- const constant int& M [[buffer(3)]],
- const constant ushort& act_type [[buffer(4)]],
- uint2 tid [[thread_position_in_grid]]
-);
diff --git a/crates/uzu/src/backends/metal/kernel/mlp/mod.rs b/crates/uzu/src/backends/metal/kernel/mlp/mod.rs
index 83958a51b..a6eb8f90e 100644
--- a/crates/uzu/src/backends/metal/kernel/mlp/mod.rs
+++ b/crates/uzu/src/backends/metal/kernel/mlp/mod.rs
@@ -2,10 +2,13 @@ use std::ptr::NonNull;
use crate::{
DataType,
- backends::metal::{
- ComputeEncoderSetValue, MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState,
- MTLContext, MTLDataType, MTLError, MTLFunctionConstantValues, MTLSize, ProtocolObject,
- Retained,
+ backends::{
+ common::kernel::MlpGateActMulKernel,
+ metal::{
+ MTLBuffer, MTLComputeCommandEncoder, MTLContext, MTLDataType,
+ MTLError, MTLFunctionConstantValues, ProtocolObject, Retained,
+ kernel::dsl::MlpGateActMulMetalKernel,
+ },
},
config::Activation,
};
@@ -57,7 +60,9 @@ impl MlpFusedConfig {
}
/// Create function constants for MLP fused matmul
- pub fn make_function_constants(&self) -> Retained<MTLFunctionConstantValues> {
+ pub fn make_function_constants(
+ &self
+ ) -> Retained<MTLFunctionConstantValues> {
let fcv = MTLFunctionConstantValues::new();
let fused = true;
fcv.set_constant_value_type_at_index(
@@ -81,7 +86,8 @@ impl MlpFusedConfig {
}
/// Create function constants for non-fused (standard) matmul
-pub fn make_non_fused_function_constants() -> Retained<MTLFunctionConstantValues> {
+pub fn make_non_fused_function_constants() -> Retained<MTLFunctionConstantValues>
+{
let fcv = MTLFunctionConstantValues::new();
let fused = false;
fcv.set_constant_value_type_at_index(
@@ -92,12 +98,8 @@ pub fn make_non_fused_function_constants() -> Retained<MTLFunctionConstantValues
fcv
}
-pub struct MlpGateActMulKernel {
- pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
pub struct MlpGateActMulEncodable {
- kernel: MlpGateActMulKernel,
+ kernel: MlpGateActMulMetalKernel,
activation: Activation,
hidden_dim: usize,
}
@@ -109,7 +111,7 @@ impl MlpGateActMulEncodable {
activation: Activation,
hidden_dim: usize,
) -> Result<Self, MTLError> {
- let kernel = MlpGateActMulKernel::new(context, data_type)?;
+ let kernel = MlpGateActMulMetalKernel::new(context, data_type.into())?;
Ok(Self {
kernel,
activation,
@@ -120,79 +122,27 @@ impl MlpGateActMulEncodable {
pub fn encode(
&self,
encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- fused_up: &ProtocolObject<dyn MTLBuffer>,
- hidden: &ProtocolObject<dyn MTLBuffer>,
+ fused_up: &Retained<ProtocolObject<dyn MTLBuffer>>,
+ hidden: &Retained<ProtocolObject<dyn MTLBuffer>>,
m: i32,
) -> Result<(), MTLError> {
- self.kernel.encode(
- encoder,
- &self.activation,
- fused_up,
- hidden,
- m,
- self.hidden_dim as i32,
- )
- }
-}
-
-impl MlpGateActMulKernel {
- fn kernel_name_for_type(
- data_type: DataType
- ) -> Result<&'static str, MTLError> {
- match data_type {
- DataType::F16 => Ok("mlp_activation_mul_f16"),
- DataType::F32 => Ok("mlp_activation_mul_f32"),
- DataType::BF16 => Ok("mlp_activation_mul_bf16"),
- other => Err(MTLError::Generic(format!(
- "Unsupported dtype for MLP activation: {:?}",
- other
- ))),
- }
- }
-
- pub fn new(
- context: &MTLContext,
- data_type: DataType,
- ) -> Result<Self, MTLError> {
- let fn_name = Self::kernel_name_for_type(data_type)?;
- let pipeline = context.compute_pipeline_state(fn_name, None)?;
- Ok(Self {
- pipeline,
- })
- }
-
- fn act_code(act: &Activation) -> u16 {
- match act {
+ let act_type = match self.activation {
Activation::SiLU {
..
} => 0,
Activation::Gelu => 1,
Activation::Identity => {
- panic!("Identity activation is not supported for MLP kernels")
+ panic!("Identity activation is not supported for kernel")
},
- }
- }
-
- pub fn encode(
- &self,
- encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- activation: &Activation,
- fused_up_buffer: &ProtocolObject<dyn MTLBuffer>,
- hidden_buffer: &ProtocolObject<dyn MTLBuffer>,
- m: i32,
- h: i32,
- ) -> Result<(), MTLError> {
- let act_code = Self::act_code(activation);
- let threads_per_tg = MTLSize::new(64, 1, 1);
- encoder.set_compute_pipeline_state(&self.pipeline);
- encoder.set_buffer(Some(fused_up_buffer), 0, 0);
- encoder.set_buffer(Some(hidden_buffer), 0, 1);
- encoder.set_value(&h, 2);
- encoder.set_value(&m, 3);
- encoder.set_value(&act_code, 4);
-
- let grid = MTLSize::new(h as usize, m as usize, 1);
- encoder.dispatch_threads(grid, threads_per_tg);
+ };
+ self.kernel.encode(
+ fused_up,
+ hidden,
+ self.hidden_dim as i32,
+ m,
+ act_type,
+ encoder,
+ );
Ok(())
}
}
diff --git a/crates/uzu/src/backends/metal/kernel/mlp_fused/gemm/kernel.rs b/crates/uzu/src/backends/metal/kernel/mlp_fused/gemm/kernel.rs
index ddb0a0f19..002bc9a42 100644
--- a/crates/uzu/src/backends/metal/kernel/mlp_fused/gemm/kernel.rs
+++ b/crates/uzu/src/backends/metal/kernel/mlp_fused/gemm/kernel.rs
@@ -1,11 +1,14 @@
use std::{collections::HashMap, ptr::NonNull};
-use super::{DispatchDescriptor, pipeline_configuration::PipelineConfiguration};
+use super::{
+ DispatchDescriptor, pipeline_configuration::PipelineConfiguration,
+};
use crate::{
DataType,
backends::metal::{
- ComputeEncoderSetValue, MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext,
- MTLError, MTLFunctionConstantValues, ProtocolObject, Retained,
+ ComputeEncoderSetValue, MTLComputeCommandEncoder,
+ MTLComputePipelineState, MTLContext, MTLError,
+ MTLFunctionConstantValues, ProtocolObject, Retained,
kernel::mlp_fused::common::MlpFusedArguments,
},
};
@@ -13,7 +16,10 @@ use crate::{
pub struct Kernel {
data_type: DataType,
weights_transposed: bool,
- pipelines: HashMap<PipelineConfiguration, Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
+ pipelines: HashMap<
+ PipelineConfiguration,
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ >,
}
impl Kernel {
@@ -80,7 +86,8 @@ impl Kernel {
&mut self,
context: &MTLContext,
configuration: &PipelineConfiguration,
- ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError> {
+ ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError>
+ {
if !self.pipelines.contains_key(configuration) {
let kernel_name = self.kernel_name(configuration);
@@ -114,7 +121,11 @@ impl Kernel {
)?;
encoder.set_compute_pipeline_state(pipeline_state);
- encoder.set_buffer(Some(arguments.input), arguments.input_offset as usize, 0);
+ encoder.set_buffer(
+ Some(arguments.input),
+ arguments.input_offset as usize,
+ 0,
+ );
encoder.set_buffer(Some(arguments.weights), 0, 1);
encoder.set_buffer(Some(arguments.output), 0, 2);
diff --git a/crates/uzu/src/backends/metal/kernel/mlp_fused/gemv/kernel.rs b/crates/uzu/src/backends/metal/kernel/mlp_fused/gemv/kernel.rs
index 6d63c6968..e113279eb 100644
--- a/crates/uzu/src/backends/metal/kernel/mlp_fused/gemv/kernel.rs
+++ b/crates/uzu/src/backends/metal/kernel/mlp_fused/gemv/kernel.rs
@@ -1,11 +1,14 @@
use std::{collections::HashMap, ptr::NonNull};
-use super::{DispatchDescriptor, pipeline_configuration::PipelineConfiguration};
+use super::{
+ DispatchDescriptor, pipeline_configuration::PipelineConfiguration,
+};
use crate::{
DataType,
backends::metal::{
- ComputeEncoderSetValue, MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext,
- MTLError, MTLFunctionConstantValues, ProtocolObject, Retained,
+ ComputeEncoderSetValue, MTLComputeCommandEncoder,
+ MTLComputePipelineState, MTLContext, MTLError,
+ MTLFunctionConstantValues, ProtocolObject, Retained,
kernel::mlp_fused::common::MlpFusedArguments,
},
};
@@ -39,7 +42,10 @@ fn kernel_name(
pub struct Kernel {
data_type: DataType,
- pipelines: HashMap<PipelineConfiguration, Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
+ pipelines: HashMap<
+ PipelineConfiguration,
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ >,
}
impl Kernel {
@@ -60,7 +66,8 @@ impl Kernel {
&mut self,
context: &MTLContext,
configuration: &PipelineConfiguration,
- ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError> {
+ ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError>
+ {
if !self.pipelines.contains_key(configuration) {
let kernel_name = kernel_name(self.data_type, configuration)?;
diff --git a/crates/uzu/src/backends/metal/kernel/mlp_fused/split_k/kernel.rs b/crates/uzu/src/backends/metal/kernel/mlp_fused/split_k/kernel.rs
index 9a55244f8..dced3cfb7 100644
--- a/crates/uzu/src/backends/metal/kernel/mlp_fused/split_k/kernel.rs
+++ b/crates/uzu/src/backends/metal/kernel/mlp_fused/split_k/kernel.rs
@@ -1,20 +1,33 @@
use std::{collections::HashMap, ptr::NonNull};
-use super::{DispatchDescriptor, pipeline_configuration::PipelineConfiguration};
+use super::{
+ DispatchDescriptor, pipeline_configuration::PipelineConfiguration,
+};
use crate::{
DataType,
- backends::metal::{
- ComputeEncoderSetValue, MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState,
- MTLContext, MTLDeviceExt, MTLError, MTLFunctionConstantValues, MTLResourceOptions,
- ProtocolObject, Retained,
- kernel::{mlp::MlpActivationType, mlp_fused::common::MlpFusedArguments},
+ backends::{
+ common::Context,
+ metal::{
+ ComputeEncoderSetValue, MTLBuffer, MTLComputeCommandEncoder,
+ MTLComputePipelineState, MTLContext, MTLError,
+ MTLFunctionConstantValues, ProtocolObject, Retained,
+ kernel::{
+ mlp::MlpActivationType, mlp_fused::common::MlpFusedArguments,
+ },
+ },
},
};
pub struct Kernel {
data_type: DataType,
- partial_pipelines: HashMap<PipelineConfiguration, Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
- accum_pipelines: HashMap<MlpActivationType, Retained<ProtocolObject<dyn MTLComputePipelineState>>>,
+ partial_pipelines: HashMap<
+ PipelineConfiguration,
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ >,
+ accum_pipelines: HashMap<
+ MlpActivationType,
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ >,
up_accumulator_buffer: Option<Retained<ProtocolObject<dyn MTLBuffer>>>,
gate_accumulator_buffer: Option<Retained<ProtocolObject<dyn MTLBuffer>>>,
accumulator_buffer_bytes: usize,
@@ -85,7 +98,8 @@ impl Kernel {
&mut self,
context: &MTLContext,
configuration: &PipelineConfiguration,
- ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError> {
+ ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError>
+ {
if !self.partial_pipelines.contains_key(configuration) {
let kernel_name = self.partial_kernel_name(configuration)?;
let pipeline_state =
@@ -100,7 +114,8 @@ impl Kernel {
&mut self,
context: &MTLContext,
activation: MlpActivationType,
- ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError> {
+ ) -> Result<&Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError>
+ {
if !self.accum_pipelines.contains_key(&activation) {
let kernel_name = self.accum_kernel_name()?;
let function_constants = MTLFunctionConstantValues::new();
@@ -130,14 +145,9 @@ impl Kernel {
{
return;
}
- self.up_accumulator_buffer = context.device.new_buffer(
- required_bytes,
- MTLResourceOptions::STORAGE_MODE_PRIVATE,
- );
- self.gate_accumulator_buffer = context.device.new_buffer(
- required_bytes,
- MTLResourceOptions::STORAGE_MODE_PRIVATE,
- );
+ self.up_accumulator_buffer = context.create_buffer(required_bytes).ok();
+ self.gate_accumulator_buffer =
+ context.create_buffer(required_bytes).ok();
self.accumulator_buffer_bytes = required_bytes;
}
diff --git a/crates/uzu/src/backends/metal/kernel/mlp_fused/split_k/steel_gemm_splitk_mlp_fused.h b/crates/uzu/src/backends/metal/kernel/mlp_fused/split_k/steel_gemm_splitk_mlp_fused.h
index d37245d71..c1bf82170 100644
--- a/crates/uzu/src/backends/metal/kernel/mlp_fused/split_k/steel_gemm_splitk_mlp_fused.h
+++ b/crates/uzu/src/backends/metal/kernel/mlp_fused/split_k/steel_gemm_splitk_mlp_fused.h
@@ -5,7 +5,7 @@
#include <metal_simdgroup_matrix>
#include "../../matmul/common/steel/gemm/gemm.h"
-#include "../../matmul/common/shared_types.h"
+#include "matmul.h"
#include "../../common/mlp_epilogue.h"
using namespace metal;
diff --git a/crates/uzu/src/backends/metal/kernel/mod.rs b/crates/uzu/src/backends/metal/kernel/mod.rs
index 64b06222d..c3c1f8d38 100644
--- a/crates/uzu/src/backends/metal/kernel/mod.rs
+++ b/crates/uzu/src/backends/metal/kernel/mod.rs
@@ -1,13 +1,12 @@
-pub mod activation;
pub mod attention;
mod data_type;
pub mod dsl {
include!(concat!(env!("OUT_DIR"), "/dsl.rs"));
}
-pub mod embedding;
+pub(super) const MTLB: &[u8] =
+ include_bytes!(concat!(env!("OUT_DIR"), "/default.metallib"));
+
pub mod kv_cache_update;
-pub mod layer_norm;
-pub mod mask_update;
pub mod matmul;
pub mod media_kernels;
pub mod mlp;
@@ -17,20 +16,13 @@ pub use mlp::{
};
pub mod mlp_fused;
pub mod moe;
-pub mod pooling;
pub mod quant_matmul;
pub mod rms_norm;
pub mod rope;
-pub mod sampling;
pub mod short_conv;
-pub mod sigmoid;
pub mod ssm;
-mod tensor_add_bias;
-mod tensor_add_swap;
-mod tensor_copy;
pub mod token_copy;
-pub use activation::ActivationKernel;
pub use attention::{
AttentionError, AttentionKernel, AttentionKernelVariant,
AttentionSinglePassArguments, AttentionTwoPassArguments,
@@ -38,42 +30,29 @@ pub use attention::{
};
pub use data_type::KernelDataType;
pub use kv_cache_update::KVCacheUpdate;
-pub use layer_norm::{LayerNormArguments, LayerNormError, LayerNormKernel};
-pub use mask_update::MaskUpdateKernel;
pub use matmul::{MatmulArguments, MatmulKernel};
pub use moe::{
- MoeBlockBasesArguments, MoeCountsOffsetsFusedArguments,
- MoeCountsOffsetsFusedError, MoeCountsOffsetsFusedKernel,
- MoeExpertsArguments, MoeExpertsError, MoeExpertsTwoPassArguments,
- MoeExpertsTwoPassDecodeKernel, MoeExpertsTwoPassPrefillKernel,
- MoeFinalizeArguments, MoeFinalizeError, MoeFinalizeKernel,
- MoeGatherArguments, MoeGatherKernel, MoeRouterTopKArguments,
- MoeRouterTopKKernel, MoeScatterArguments, MoeScatterError,
- MoeScatterKernels, MoeScatterWithMapArguments,
+ MoeBlockBasesArguments, MoeExpertsArguments, MoeExpertsError,
+ MoeExpertsTwoPassArguments, MoeExpertsTwoPassDecodeKernels,
+ MoeExpertsTwoPassPrefillKernel, MoeGatherArguments, MoeGatherKernels,
+ MoeRouterTopKArguments, MoeRouterTopKKernel, MoeScatterArguments,
+ MoeScatterError, MoeScatterKernels, MoeScatterWithMapArguments,
};
-pub use pooling::PoolingKernel;
pub use rms_norm::{
QKNormArguments, QKNormTarget, RMSNormArguments, RMSNormError,
RMSNormKernel, RMSNormKernelType,
};
pub use rope::{RopeError, RopeKernel, RopeKernelArguments};
-pub use sampling::{ArgmaxStrategy, SamplingError, SamplingKernel};
pub use short_conv::{
ShortConvDecodeArguments, ShortConvKernel, ShortConvKernelError,
ShortConvPrefillArguments,
};
-pub use sigmoid::SigmoidKernel;
pub use ssm::{
Conv1dPackArguments, Conv1dScanArguments, Conv1dScanKernel,
- SSDPrefillArguments, SSDPrefillKernel, SSDPrefillMode, SSDUpdateArguments,
- SSDUpdateKernel, SSMKernelError, SplitInProjArguments, SplitInProjKernel,
+ SSDPrefillArguments, SSDPrefillMode, SSMKernelError,
};
-pub use tensor_add_bias::TensorAddBias;
-pub use tensor_add_swap::TensorAddSwapKernel;
-pub use tensor_copy::TensorCopyKernel;
pub use token_copy::TokenCopyKernel;
-
use super::{MTLContext, metal_extensions};
pub mod media {
diff --git a/crates/uzu/src/backends/metal/kernel/moe/counts_offsets_fused.metal b/crates/uzu/src/backends/metal/kernel/moe/counts_offsets_fused.metal
index a6e14eb6a..653a9a796 100644
--- a/crates/uzu/src/backends/metal/kernel/moe/counts_offsets_fused.metal
+++ b/crates/uzu/src/backends/metal/kernel/moe/counts_offsets_fused.metal
@@ -1,6 +1,5 @@
#include <metal_stdlib>
#include <metal_atomic>
-using namespace metal;
#include "../definitions.metal"
@@ -9,19 +8,22 @@ using namespace metal;
// Single-kernel fused: count all experts + scan to offsets
// This kernel is launched with SINGLE threadgroup
-[[max_total_threads_per_threadgroup(128)]]
-kernel void moe_counts_offsets_fused(
- device const int* topk_ids [[buffer(0)]],
- device uint* offsets [[buffer(1)]], // output: exclusive scan [E+1]
- device uint* sum_k_out [[buffer(2)]], // output: total count [1]
- device uint* partials
- [[buffer(3)]], // output: partials [num_tiles * TILE_E] (for block_bases)
- constant uint& T [[buffer(4)]],
- constant uint& E [[buffer(5)]],
- constant uint& K [[buffer(6)]],
- uint lid [[thread_index_in_threadgroup]]
+KERNEL(MoeCountsOffsetsFused)(
+ device const int* topk_ids,
+ device uint* offsets, // output: exclusive scan [E+1]
+ device uint* sum_k_out, // output: total count [1]
+ device uint* partials, // output: partials [num_tiles * TILE_E] (for block_bases)
+ constant uint& t_input,
+ constant uint& e_input,
+ constant uint& k_input,
+ threadgroup _atomic<uint> tg_hist[TILE_E],
+ threadgroup uint scan_shared[BLOCK_SIZE],
+ threadgroup uint reduce_shared[BLOCK_SIZE],
+ threadgroup uint counts_shared[BLOCK_SIZE], // Cache counts in threadgroup memory
+ threadgroup uint& carry,
+ const uint lid THREADS(128)
) {
- if (E == 0) {
+ if (e_input == 0) {
if (lid == 0) {
offsets[0] = 0u;
sum_k_out[0] = 0u;
@@ -29,18 +31,12 @@ kernel void moe_counts_offsets_fused(
return;
}
- threadgroup _atomic<uint> tg_hist[TILE_E];
- threadgroup uint scan_shared[BLOCK_SIZE];
- threadgroup uint reduce_shared[BLOCK_SIZE];
- threadgroup uint
- counts_shared[BLOCK_SIZE]; // Cache counts in threadgroup memory
-
// ═══════════════════════════════════════════════════════════
// PHASE 1: Count tokens per expert using tiled histogram
// ═══════════════════════════════════════════════════════════
// Tile over E dimension
- for (uint e0 = 0; e0 < E; e0 += TILE_E) {
- const uint tile_e = (e0 + TILE_E <= E) ? TILE_E : (E - e0);
+ for (uint e0 = 0; e0 < e_input; e0 += TILE_E) {
+ const uint tile_e = (e0 + TILE_E <= e_input) ? TILE_E : (e_input - e0);
// Zero threadgroup histogram
for (uint e = lid; e < tile_e; e += BLOCK_SIZE) {
@@ -50,9 +46,9 @@ kernel void moe_counts_offsets_fused(
// Process all tokens in strided fashion
// Since we have only 1 threadgroup, we need to cover all T tokens
- for (uint t = lid; t < T; t += BLOCK_SIZE) {
- const uint base = t * K;
- for (uint k = 0; k < K; ++k) {
+ for (uint t = lid; t < t_input; t += BLOCK_SIZE) {
+ const uint base = t * k_input;
+ for (uint k = 0; k < k_input; ++k) {
int eid = topk_ids[base + k];
if (eid >= 0) {
uint ue = uint(eid);
@@ -78,14 +74,13 @@ kernel void moe_counts_offsets_fused(
// ═══════════════════════════════════════════════════════════
// PHASE 2: Compute exclusive prefix scan on counts
// ═══════════════════════════════════════════════════════════
- threadgroup uint carry;
if (lid == 0) {
carry = 0u;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
- for (uint base = 0; base < E; base += BLOCK_SIZE) {
- uint remaining = E - base;
+ for (uint base = 0; base < e_input; base += BLOCK_SIZE) {
+ uint remaining = e_input - base;
uint chunk_n = remaining < BLOCK_SIZE ? remaining : BLOCK_SIZE;
uint v = (lid < chunk_n) ? counts_shared[lid] : 0u;
@@ -113,7 +108,7 @@ kernel void moe_counts_offsets_fused(
// Write final offset and total
if (lid == 0) {
- offsets[E] = carry;
+ offsets[e_input] = carry;
sum_k_out[0] = carry;
}
}
diff --git a/crates/uzu/src/backends/metal/kernel/moe/counts_offsets_fused.rs b/crates/uzu/src/backends/metal/kernel/moe/counts_offsets_fused.rs
deleted file mode 100644
index 73c32dfd6..000000000
--- a/crates/uzu/src/backends/metal/kernel/moe/counts_offsets_fused.rs
+++ /dev/null
@@ -1,91 +0,0 @@
-use crate::backends::metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- MTLComputePipelineState, MTLContext, MTLError, MTLSize, ProtocolObject, Retained,
- metal_extensions::ComputeEncoderSetValue,
-};
-
-// ---- Fused Counts + Offsets Kernel ----
-
-#[derive(Debug, thiserror::Error)]
-pub enum MoeCountsOffsetsFusedError {
- #[error("Metal error: {0}")]
- MetalError(#[from] MTLError),
- #[error("Invalid dimensions: T={t}, E={e}, K={k}")]
- InvalidDimensions {
- t: usize,
- e: usize,
- k: usize,
- },
-}
-
-pub struct MoeCountsOffsetsFusedKernel {
- pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-#[derive(Debug)]
-pub struct MoeCountsOffsetsFusedArguments<'a> {
- pub topk_ids_buffer: &'a ProtocolObject<dyn MTLBuffer>,
- pub offsets_buffer: &'a ProtocolObject<dyn MTLBuffer>, // output [E+1]
- pub sum_k_buffer: &'a ProtocolObject<dyn MTLBuffer>, // output [1]
- pub partials_buffer: &'a ProtocolObject<dyn MTLBuffer>, // output [num_tiles * 512] (for block_bases)
- pub t: usize,
- pub e: usize,
- pub k: usize,
-}
-
-impl MoeCountsOffsetsFusedKernel {
- pub fn new(
- mtl_context: &MTLContext
- ) -> Result<Self, MoeCountsOffsetsFusedError> {
- let pipeline = mtl_context
- .compute_pipeline_state("moe_counts_offsets_fused", None)?;
- Ok(Self {
- pipeline,
- })
- }
-
- pub fn encode(
- &self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- args: MoeCountsOffsetsFusedArguments,
- ) -> Result<(), MoeCountsOffsetsFusedError> {
- if args.k == 0 || args.e == 0 {
- return Ok(());
- }
- if args.t == 0 {
- return Ok(());
- }
- if args.e < 1 {
- return Err(MoeCountsOffsetsFusedError::InvalidDimensions {
- t: args.t,
- e: args.e,
- k: args.k,
- });
- }
-
- let encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- encoder.set_compute_pipeline_state(&self.pipeline);
-
- encoder.set_buffer(Some(args.topk_ids_buffer), 0, 0);
- encoder.set_buffer(Some(args.offsets_buffer), 0, 1);
- encoder.set_buffer(Some(args.sum_k_buffer), 0, 2);
- encoder.set_buffer(Some(args.partials_buffer), 0, 3);
-
- let t_u32 = args.t as u32;
- let e_u32 = args.e as u32;
- let k_u32 = args.k as u32;
-
- encoder.set_value(&t_u32, 4);
- encoder.set_value(&e_u32, 5);
- encoder.set_value(&k_u32, 6);
-
- let threads_per_threadgroup = MTLSize::new(128, 1, 1);
- let tg = MTLSize::new(1, 1, 1);
- encoder.dispatch_threadgroups(tg, threads_per_threadgroup);
-
- encoder.end_encoding();
- Ok(())
- }
-}
diff --git a/crates/uzu/src/backends/metal/kernel/moe/experts.rs b/crates/uzu/src/backends/metal/kernel/moe/experts.rs
index 786b5e650..b0293716f 100644
--- a/crates/uzu/src/backends/metal/kernel/moe/experts.rs
+++ b/crates/uzu/src/backends/metal/kernel/moe/experts.rs
@@ -1,16 +1,15 @@
use std::{mem::size_of, ptr::NonNull};
-use super::{
- MoePassARowMapArguments, MoePassATileBuildArguments,
- MoePassATileCountsArguments, MoePassATileDispatchArguments,
- MoePassATileKernel, MoePassATileScanArguments, MoeTileCountsArguments,
- MoeTileDispatchArguments, MoeTileError, MoeTileMapBuildArguments,
- MoeTileMapKernel, MoeTileScanArguments, dtype_index, dtype_suffix,
-};
+use super::{dtype_index, dtype_suffix};
use crate::backends::metal::{
- KernelDataType, MTLBlitCommandEncoder, MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
- MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext, MTLDataType, MTLError,
- MTLFunctionConstantValues, MTLSize, NSRange, ProtocolObject, Retained,
+ KernelDataType, MTLBlitCommandEncoder, MTLBuffer, MTLCommandBuffer,
+ MTLCommandEncoder, MTLComputeCommandEncoder, MTLComputePipelineState,
+ MTLContext, MTLDataType, MTLError, MTLFunctionConstantValues, MTLSize,
+ NSRange, ProtocolObject, Retained,
+ kernel::moe::tiles_map::{
+ MoeTileCountsArguments, MoeTileDispatchArguments,
+ MoeTileMapBuildArguments, MoeTileMapKernels, MoeTileScanArguments,
+ },
};
#[derive(Debug, thiserror::Error)]
@@ -26,16 +25,6 @@ pub enum MoeExpertsError {
Generic(String),
}
-impl From<MoeTileError> for MoeExpertsError {
- fn from(err: MoeTileError) -> Self {
- match err {
- MoeTileError::MetalError(inner) => {
- MoeExpertsError::MetalError(inner)
- },
- }
- }
-}
-
#[derive(Debug)]
pub struct MoeExpertsArguments<'a> {
pub x_perm_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [sum_k, d_model] - permuted input
@@ -47,7 +36,7 @@ pub struct MoeExpertsArguments<'a> {
pub down_biases: &'a ProtocolObject<dyn MTLBuffer>, // [E, d_model] - down projection biases
pub tile_counts: &'a ProtocolObject<dyn MTLBuffer>, // [E]
pub tile_row_offsets: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
- pub tile_map: &'a ProtocolObject<dyn MTLBuffer>, // [max_tiles * 3]
+ pub tile_map: &'a ProtocolObject<dyn MTLBuffer>, // [max_tiles * 3]
pub total_tiles: &'a ProtocolObject<dyn MTLBuffer>, // [2]
pub dispatch_args: &'a ProtocolObject<dyn MTLBuffer>, // [3]
pub num_tiles_n: usize,
@@ -65,16 +54,11 @@ pub struct MoeExpertsArguments<'a> {
pub data_type: KernelDataType,
}
-pub struct MoeExpertsTwoPassDecodeKernel {
- pass_a_tile: MoePassATileKernel,
- pass_a_indirect: Vec<Vec<Retained<ProtocolObject<dyn MTLComputePipelineState>>>>, // [gate][dtype]
- fused_down: Vec<Retained<ProtocolObject<dyn MTLComputePipelineState>>>, // [dtype]
-}
-
pub struct MoeExpertsTwoPassPrefillKernel {
- tile_map: MoeTileMapKernel,
- pass_a_indirect: Vec<Vec<Retained<ProtocolObject<dyn MTLComputePipelineState>>>>, // [gate][dtype]
- pass_b_indirect: Vec<Retained<ProtocolObject<dyn MTLComputePipelineState>>>, // [dtype]
+ tile_map: MoeTileMapKernels,
+ pass_a_indirect:
+ Vec<Vec<Retained<ProtocolObject<dyn MTLComputePipelineState>>>>, // [gate][dtype]
+ pass_b_indirect: Vec<Retained<ProtocolObject<dyn MTLComputePipelineState>>>, // [dtype]
}
#[derive(Debug)]
@@ -107,228 +91,6 @@ pub struct MoeExpertsTwoPassArguments<'a> {
pub data_type: KernelDataType,
}
-impl MoeExpertsTwoPassDecodeKernel {
- pub fn new(ctx: &MTLContext) -> Result<Self, MoeExpertsError> {
- let dtypes = [
- KernelDataType::Float16,
- KernelDataType::BFloat16,
- KernelDataType::Float32,
- ];
- let mut pass_a_indirect = vec![Vec::with_capacity(dtypes.len()); 4];
- for gate in 0u32..4u32 {
- for dtype in &dtypes {
- let dtype_suffix = dtype_suffix(*dtype);
- let fcv = MTLFunctionConstantValues::new();
- fcv.set_constant_value_type_at_index(
- NonNull::from(&gate).cast(),
- MTLDataType::UInt,
- 30,
- );
- let tile_h: u32 = 512;
- fcv.set_constant_value_type_at_index(
- NonNull::from(&tile_h).cast(),
- MTLDataType::UInt,
- 32,
- );
- let kernel_name = format!(
- "moe_experts_decode_pass_a_indirect_{}",
- dtype_suffix
- );
- let cache_key =
- format!("{}_gate_{}_tile_{}", kernel_name, gate, tile_h);
- pass_a_indirect[gate as usize].push(
- ctx.compute_pipeline_state_cached(
- &cache_key,
- &kernel_name,
- Some(&fcv),
- )?,
- );
- }
- }
- let mut fused_down = Vec::with_capacity(dtypes.len());
- for dtype in &dtypes {
- let dtype_suffix = dtype_suffix(*dtype);
- let kernel_name =
- format!("moe_experts_decode_down_fused_2d_{}", dtype_suffix);
- fused_down.push(ctx.compute_pipeline_state(&kernel_name, None)?);
- }
- Ok(Self {
- pass_a_tile: MoePassATileKernel::new(ctx)?,
- pass_a_indirect,
- fused_down,
- })
- }
-
- pub fn encode(
- &self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- args: MoeExpertsTwoPassArguments,
- ) -> Result<(), MoeExpertsError> {
- if args.total_rows == 0 {
- return Ok(());
- }
- let gate_idx = (args.gating_code.min(3)) as usize;
- let dtype_idx = dtype_index(args.data_type);
- const BLOCK_M: u32 = 4;
- let h_blocks = (args.d_ff as u32 + BLOCK_M - 1) / BLOCK_M;
- self.pass_a_tile.encode_counts(
- command_buffer,
- &MoePassATileCountsArguments {
- expert_offsets: args.expert_offsets,
- tile_counts: args.tile_counts,
- e: args.e,
- h_blocks,
- },
- )?;
- self.pass_a_tile.encode_scan(
- command_buffer,
- &MoePassATileScanArguments {
- tile_counts: args.tile_counts,
- tile_offsets: args.tile_offsets,
- total_tiles: args.total_tiles,
- e: args.e,
- },
- )?;
- self.pass_a_tile.encode_row_map(
- command_buffer,
- &MoePassARowMapArguments {
- expert_offsets: args.expert_offsets,
- row_expert_map: args.row_expert_map,
- total_rows: args.total_rows,
- e: args.e,
- },
- )?;
- self.pass_a_tile.encode_build_map(
- command_buffer,
- &MoePassATileBuildArguments {
- expert_offsets: args.expert_offsets,
- tile_offsets: args.tile_offsets,
- row_expert_map: args.row_expert_map,
- tile_map: args.tile_map,
- total_rows: args.total_rows,
- h_blocks,
- },
- )?;
- self.pass_a_tile.encode_dispatch_args(
- command_buffer,
- &MoePassATileDispatchArguments {
- total_tiles: args.total_tiles,
- dispatch_args: args.dispatch_args,
- num_tiles_y: 1,
- },
- )?;
- let d_model_u32 = args.d_model as u32;
- let d_ff_u32 = args.d_ff as u32;
- let e_u32 = args.e as u32;
- let pass_a_pipeline = &self.pass_a_indirect[gate_idx][dtype_idx];
- let encoder_a = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- encoder_a.set_compute_pipeline_state(pass_a_pipeline);
- encoder_a.set_buffer(Some(args.x_perm_buffer), 0, 0);
- encoder_a.set_buffer(Some(args.expert_offsets), 0, 1);
- encoder_a.set_buffer(Some(args.w13_all), 0, 2);
- encoder_a.set_buffer(Some(args.hidden_buffer), 0, 3);
- encoder_a.set_buffer(Some(args.up_biases), 0, 4);
- unsafe {
- encoder_a.set_bytes(
- NonNull::new_unchecked(&d_model_u32 as *const _ as *mut _),
- size_of::<u32>(),
- 5,
- );
- encoder_a.set_bytes(
- NonNull::new_unchecked(&d_ff_u32 as *const _ as *mut _),
- size_of::<u32>(),
- 6,
- );
- encoder_a.set_bytes(
- NonNull::new_unchecked(&e_u32 as *const _ as *mut _),
- size_of::<u32>(),
- 7,
- );
- encoder_a.set_bytes(
- NonNull::new_unchecked(
- &args.gate_clip_min as *const _ as *mut _,
- ),
- size_of::<f32>(),
- 8,
- );
- encoder_a.set_bytes(
- NonNull::new_unchecked(
- &args.gate_clip_max as *const _ as *mut _,
- ),
- size_of::<f32>(),
- 9,
- );
- encoder_a.set_bytes(
- NonNull::new_unchecked(&args.up_clip_min as *const _ as *mut _),
- size_of::<f32>(),
- 10,
- );
- encoder_a.set_bytes(
- NonNull::new_unchecked(&args.up_clip_max as *const _ as *mut _),
- size_of::<f32>(),
- 11,
- );
- encoder_a.set_bytes(
- NonNull::new_unchecked(&args.silu_alpha as *const _ as *mut _),
- size_of::<f32>(),
- 12,
- );
- }
- encoder_a.set_buffer(Some(args.tile_map), 0, 13);
- encoder_a.dispatch_threadgroups_indirect(
- args.dispatch_args,
- 0,
- MTLSize::new(128, 1, 1),
- );
- encoder_a.end_encoding();
- let total_rows_u32 = args.total_rows as u32;
- let pass_b_pipeline = &self.fused_down[dtype_idx];
- let encoder_b = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- encoder_b.set_compute_pipeline_state(pass_b_pipeline);
- encoder_b.set_buffer(Some(args.hidden_buffer), 0, 0);
- encoder_b.set_buffer(Some(args.row_expert_map), 0, 1);
- encoder_b.set_buffer(Some(args.w2_all), 0, 2);
- encoder_b.set_buffer(Some(args.down_biases), 0, 3);
- encoder_b.set_buffer(Some(args.output_buffer), 0, 4);
- unsafe {
- encoder_b.set_bytes(
- NonNull::new_unchecked(&total_rows_u32 as *const _ as *mut _),
- size_of::<u32>(),
- 5,
- );
- encoder_b.set_bytes(
- NonNull::new_unchecked(&d_model_u32 as *const _ as *mut _),
- size_of::<u32>(),
- 6,
- );
- encoder_b.set_bytes(
- NonNull::new_unchecked(&d_ff_u32 as *const _ as *mut _),
- size_of::<u32>(),
- 7,
- );
- encoder_b.set_bytes(
- NonNull::new_unchecked(&e_u32 as *const _ as *mut _),
- size_of::<u32>(),
- 8,
- );
- }
- const SIMDGROUPS_PER_TG: u32 = 8;
- const THREADS_PER_TG: u32 = 256;
- let col_blocks =
- (args.d_model as u32 + SIMDGROUPS_PER_TG - 1) / SIMDGROUPS_PER_TG;
- encoder_b.dispatch_threadgroups(
- MTLSize::new(col_blocks as usize, args.total_rows, 1),
- MTLSize::new(THREADS_PER_TG as usize, 1, 1),
- );
- encoder_b.end_encoding();
- Ok(())
- }
-}
-
impl MoeExpertsTwoPassPrefillKernel {
pub fn new(ctx: &MTLContext) -> Result<Self, MoeExpertsError> {
let dtypes = [
@@ -371,7 +133,7 @@ impl MoeExpertsTwoPassPrefillKernel {
.push(ctx.compute_pipeline_state(&kernel_name, None)?);
}
Ok(Self {
- tile_map: MoeTileMapKernel::new(ctx)?,
+ tile_map: MoeTileMapKernels::new(ctx)?,
pass_a_indirect,
pass_b_indirect,
})
@@ -379,7 +141,7 @@ impl MoeExpertsTwoPassPrefillKernel {
pub fn encode(
&self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
args: MoeExpertsTwoPassArguments,
) -> Result<(), MoeExpertsError> {
if args.total_rows == 0 {
@@ -408,7 +170,7 @@ impl MoeExpertsTwoPassPrefillKernel {
tile_counts_buffer: args.tile_counts,
e: args.e,
},
- )?;
+ );
self.tile_map.encode_scan(
command_buffer,
&MoeTileScanArguments {
@@ -417,7 +179,7 @@ impl MoeExpertsTwoPassPrefillKernel {
total_tiles_buffer: args.total_tiles,
e: args.e,
},
- )?;
+ );
self.tile_map.encode_build_map(
command_buffer,
&MoeTileMapBuildArguments {
@@ -427,7 +189,7 @@ impl MoeExpertsTwoPassPrefillKernel {
tile_map: args.tile_map,
e: args.e,
},
- )?;
+ );
let d_model_u32 = args.d_model as u32;
let d_ff_u32 = args.d_ff as u32;
let e_u32 = args.e as u32;
@@ -450,7 +212,7 @@ impl MoeExpertsTwoPassPrefillKernel {
dispatch_args: args.dispatch_args,
num_tiles_x: n_tiles_ff,
},
- )?;
+ );
let pass_a_pipeline = &self.pass_a_indirect[gate_idx][dtype_idx];
let encoder_a = command_buffer
.new_compute_command_encoder()
@@ -524,7 +286,7 @@ impl MoeExpertsTwoPassPrefillKernel {
dispatch_args: args.dispatch_args,
num_tiles_x: n_tiles_model,
},
- )?;
+ );
let pass_b_pipeline = &self.pass_b_indirect[dtype_idx];
let encoder_b = command_buffer
.new_compute_command_encoder()
@@ -562,239 +324,3 @@ impl MoeExpertsTwoPassPrefillKernel {
Ok(())
}
}
-
-/// Arguments for single-token MoE decode (T=1 optimized path)
-#[derive(Debug)]
-pub struct MoeExpertsSingleDecodeArguments<'a> {
- /// Input activation [d_model]
- pub x: &'a ProtocolObject<dyn MTLBuffer>,
- /// Top-K expert indices from router [K]
- pub topk_ids: &'a ProtocolObject<dyn MTLBuffer>,
- /// Top-K probabilities from router [K]
- pub topk_probs: &'a ProtocolObject<dyn MTLBuffer>,
- /// Up/gate projection weights [E, 2*d_ff, d_model]
- pub w13_all: &'a ProtocolObject<dyn MTLBuffer>,
- /// Down projection weights [E, d_model, d_ff]
- pub w2_all: &'a ProtocolObject<dyn MTLBuffer>,
- /// Up/gate biases [E, 2*d_ff]
- pub up_biases: &'a ProtocolObject<dyn MTLBuffer>,
- /// Down biases [E, d_model]
- pub down_biases: &'a ProtocolObject<dyn MTLBuffer>,
- /// Hidden buffer [K, d_ff] - intermediate storage (f32)
- pub hidden: &'a ProtocolObject<dyn MTLBuffer>,
- /// Final output [d_model]
- pub y: &'a ProtocolObject<dyn MTLBuffer>,
- /// Model dimension
- pub d_model: usize,
- /// FFN hidden dimension
- pub d_ff: usize,
- /// Number of active experts per token
- pub k: usize,
- /// Gating activation: 0=GELU, 1=SiLU, 2=SwiGLU, 3=GEGLU
- pub gating_code: u32,
- /// SiLU activation alpha parameter
- pub silu_alpha: f32,
- /// Gate clipping min
- pub gate_clip_min: f32,
- /// Gate clipping max
- pub gate_clip_max: f32,
- /// Up clipping min
- pub up_clip_min: f32,
- /// Up clipping max
- pub up_clip_max: f32,
- /// Data type
- pub data_type: KernelDataType,
-}
-
-pub struct MoeExpertsSingleDecodeKernel {
- pass_a: Vec<Vec<Retained<ProtocolObject<dyn MTLComputePipelineState>>>>, // [gate][dtype]
- pass_b: Vec<Retained<ProtocolObject<dyn MTLComputePipelineState>>>, // [dtype]
-}
-
-impl MoeExpertsSingleDecodeKernel {
- pub fn new(ctx: &MTLContext) -> Result<Self, MoeExpertsError> {
- let dtypes = [
- KernelDataType::Float16,
- KernelDataType::BFloat16,
- KernelDataType::Float32,
- ];
-
- // Pass A: needs gating function constant
- let mut pass_a = vec![Vec::with_capacity(dtypes.len()); 4];
- for gate in 0u32..4u32 {
- for dtype in &dtypes {
- let suffix = dtype_suffix(*dtype);
- let fcv = MTLFunctionConstantValues::new();
- fcv.set_constant_value_type_at_index(
- NonNull::from(&gate).cast(),
- MTLDataType::UInt,
- 30,
- );
- let kernel_name =
- format!("moe_experts_decode_single_pass_a_{}", suffix);
- let cache_key = format!("{}_gate_{}", kernel_name, gate);
- pass_a[gate as usize].push(ctx.compute_pipeline_state_cached(
- &cache_key,
- &kernel_name,
- Some(&fcv),
- )?);
- }
- }
-
- // Pass B (fused with finalize): no function constants needed
- let mut pass_b = Vec::with_capacity(dtypes.len());
- for dtype in &dtypes {
- let suffix = dtype_suffix(*dtype);
- let kernel_name =
- format!("moe_experts_decode_single_pass_b_{}", suffix);
- pass_b.push(ctx.compute_pipeline_state(&kernel_name, None)?);
- }
-
- Ok(Self {
- pass_a,
- pass_b,
- })
- }
-
- /// Encode the single-token decode pipeline
- ///
- /// Dispatches:
- /// 1. Pass A: grid = (h_blocks, K) - K parallel expert up+gate projections
- /// 2. Pass B: grid = (d_blocks, 1) - computes final y directly (fused finalize)
- pub fn encode(
- &self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- args: MoeExpertsSingleDecodeArguments,
- ) -> Result<(), MoeExpertsError> {
- if args.k == 0 {
- return Ok(());
- }
-
- let gate_idx = (args.gating_code.min(3)) as usize;
- let dtype_idx = dtype_index(args.data_type);
-
- let d_model_u32 = args.d_model as u32;
- let d_ff_u32 = args.d_ff as u32;
- let k_u32 = args.k as u32;
-
- // Pass A: 4 simdgroups (128 threads), each threadgroup outputs 4 elements
- const PASSA_BLOCK_H: u32 = 4;
- const PASSA_THREADS: u32 = 128;
- let h_blocks = (d_ff_u32 + PASSA_BLOCK_H - 1) / PASSA_BLOCK_H;
-
- // Pass A: x @ W13[expert] -> hidden
- {
- let pipeline = &self.pass_a[gate_idx][dtype_idx];
- let encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- encoder.set_compute_pipeline_state(pipeline);
- encoder.set_buffer(Some(args.x), 0, 0);
- encoder.set_buffer(Some(args.topk_ids), 0, 1);
- encoder.set_buffer(Some(args.w13_all), 0, 2);
- encoder.set_buffer(Some(args.up_biases), 0, 3);
- encoder.set_buffer(Some(args.hidden), 0, 4);
- unsafe {
- encoder.set_bytes(
- NonNull::new_unchecked(&d_model_u32 as *const _ as *mut _),
- size_of::<u32>(),
- 5,
- );
- encoder.set_bytes(
- NonNull::new_unchecked(&d_ff_u32 as *const _ as *mut _),
- size_of::<u32>(),
- 6,
- );
- encoder.set_bytes(
- NonNull::new_unchecked(&k_u32 as *const _ as *mut _),
- size_of::<u32>(),
- 7,
- );
- encoder.set_bytes(
- NonNull::new_unchecked(
- &args.silu_alpha as *const _ as *mut _,
- ),
- size_of::<f32>(),
- 8,
- );
- encoder.set_bytes(
- NonNull::new_unchecked(
- &args.gate_clip_min as *const _ as *mut _,
- ),
- size_of::<f32>(),
- 9,
- );
- encoder.set_bytes(
- NonNull::new_unchecked(
- &args.gate_clip_max as *const _ as *mut _,
- ),
- size_of::<f32>(),
- 10,
- );
- encoder.set_bytes(
- NonNull::new_unchecked(
- &args.up_clip_min as *const _ as *mut _,
- ),
- size_of::<f32>(),
- 11,
- );
- encoder.set_bytes(
- NonNull::new_unchecked(
- &args.up_clip_max as *const _ as *mut _,
- ),
- size_of::<f32>(),
- 12,
- );
- }
- encoder.dispatch_threadgroups(
- MTLSize::new(h_blocks as usize, args.k, 1),
- MTLSize::new(PASSA_THREADS as usize, 1, 1),
- );
- encoder.end_encoding();
- }
-
- // Pass B: 8 simdgroups (256 threads), outputs final y directly
- const PASSB_BLOCK_D: u32 = 8;
- const PASSB_THREADS: u32 = 256;
- let d_blocks = (d_model_u32 + PASSB_BLOCK_D - 1) / PASSB_BLOCK_D;
-
- // Pass B: hidden @ W2[expert] -> y (with weighted sum)
- {
- let pipeline = &self.pass_b[dtype_idx];
- let encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- encoder.set_compute_pipeline_state(pipeline);
- encoder.set_buffer(Some(args.hidden), 0, 0);
- encoder.set_buffer(Some(args.topk_ids), 0, 1);
- encoder.set_buffer(Some(args.topk_probs), 0, 2);
- encoder.set_buffer(Some(args.w2_all), 0, 3);
- encoder.set_buffer(Some(args.down_biases), 0, 4);
- encoder.set_buffer(Some(args.y), 0, 5);
- unsafe {
- encoder.set_bytes(
- NonNull::new_unchecked(&d_model_u32 as *const _ as *mut _),
- size_of::<u32>(),
- 6,
- );
- encoder.set_bytes(
- NonNull::new_unchecked(&d_ff_u32 as *const _ as *mut _),
- size_of::<u32>(),
- 7,
- );
- encoder.set_bytes(
- NonNull::new_unchecked(&k_u32 as *const _ as *mut _),
- size_of::<u32>(),
- 8,
- );
- }
- encoder.dispatch_threadgroups(
- MTLSize::new(d_blocks as usize, 1, 1),
- MTLSize::new(PASSB_THREADS as usize, 1, 1),
- );
- encoder.end_encoding();
- }
-
- Ok(())
- }
-}
diff --git a/crates/uzu/src/backends/metal/kernel/moe/experts_single.metal b/crates/uzu/src/backends/metal/kernel/moe/experts_single.metal
index 5f5d2ec28..bf345f1ff 100644
--- a/crates/uzu/src/backends/metal/kernel/moe/experts_single.metal
+++ b/crates/uzu/src/backends/metal/kernel/moe/experts_single.metal
@@ -1,24 +1,9 @@
#include <metal_stdlib>
#include <metal_simdgroup>
-using namespace metal;
+#include "../definitions.metal"
+#include "moe_commons.h"
-constant uint GATING_SEL
- [[function_constant(30)]]; // 0=GELU, 1=SiLU, 2=SwiGLU, 3=GEGLU
-
-static inline float gelu_approx(float x) {
- const float k0 = 0.7978845608f;
- const float k1 = 0.044715f;
- if (x > 10.0f)
- return x;
- if (x < -10.0f)
- return 0.0f;
- return 0.5f * x *
- (1.0f + tanh(clamp(k0 * (x + k1 * x * x * x), -10.0f, 10.0f)));
-}
-
-static inline float silu(float x, float alpha) {
- return x / (1.0f + exp(-alpha * x));
-}
+#define SIMD_SIZE 32
// ============================================================================
// Pass A: x @ W13[expert] → hidden[k]
@@ -26,27 +11,30 @@ static inline float silu(float x, float alpha) {
// Each simdgroup: 32 threads reduce d_model with float4 vectorized loads
// Grid: (ceil(d_ff/4), K)
// ============================================================================
-
-template <typename T, typename T4>
-inline void moe_experts_decode_single_pass_a_impl(
+template <typename T>
+VARIANTS(T, float, half, bfloat)
+KERNEL(MoeExpertsDecodeSinglePassA)(
device const T* x,
device const int* topk_ids,
- device const T* W13_all,
+ device const T* w13_all,
device const T* biases,
device float* hidden_out,
- uint d_model,
- uint d_ff,
- uint K,
- float silu_alpha,
- float gate_clip_min,
- float gate_clip_max,
- float up_clip_min,
- float up_clip_max,
- uint k_slot,
- uint h_block_idx,
- uint simd_gid,
- uint simd_lid
+ constant uint& d_model,
+ constant uint& d_ff,
+ constant uint& k,
+ constant float& silu_alpha,
+ constant float& gate_clip_min,
+ constant float& gate_clip_max,
+ constant float& up_clip_min,
+ constant float& up_clip_max,
+ const uint gating_sel SPECIALIZE, // 0=GELU, 1=SiLU, 2=SwiGLU, 3=GEGLU
+ const uint h_block_idx GROUPS(d_ff.div_ceil(4)),
+ const uint k_slot GROUPS(k),
+ const uint tid THREADS(128)
) {
+ const uint simd_gid = tid / SIMD_SIZE;
+ const uint simd_lid = tid % SIMD_SIZE;
+
const int expert_id = topk_ids[k_slot];
if (expert_id < 0)
return;
@@ -60,9 +48,9 @@ inline void moe_experts_decode_single_pass_a_impl(
const ulong w13_base = (ulong)expert_u * w13_stride;
const ulong bias_base = (ulong)expert_u * (ulong)(2 * d_ff);
- device const T* w_up_row = W13_all + w13_base + (ulong)h_idx * (ulong)d_model;
+ device const T* w_up_row = w13_all + w13_base + (ulong)h_idx * (ulong)d_model;
device const T* w_gate_row =
- W13_all + w13_base + (ulong)(d_ff + h_idx) * (ulong)d_model;
+ w13_all + w13_base + (ulong)(d_ff + h_idx) * (ulong)d_model;
float acc_up = 0.0f;
float acc_gate = 0.0f;
@@ -72,21 +60,19 @@ inline void moe_experts_decode_single_pass_a_impl(
for (uint i = 0; i < vec_iters; ++i) {
uint base_idx = i * 128 + simd_lid * 4;
- T4 x_vec = *reinterpret_cast<device const T4*>(x + base_idx);
- T4 w_up_vec = *reinterpret_cast<device const T4*>(w_up_row + base_idx);
-
- acc_up += float(x_vec.x) * float(w_up_vec.x);
- acc_up += float(x_vec.y) * float(w_up_vec.y);
- acc_up += float(x_vec.z) * float(w_up_vec.z);
- acc_up += float(x_vec.w) * float(w_up_vec.w);
-
- if (GATING_SEL > 1) {
- T4 w_gate_vec =
- *reinterpret_cast<device const T4*>(w_gate_row + base_idx);
- acc_gate += float(x_vec.x) * float(w_gate_vec.x);
- acc_gate += float(x_vec.y) * float(w_gate_vec.y);
- acc_gate += float(x_vec.z) * float(w_gate_vec.z);
- acc_gate += float(x_vec.w) * float(w_gate_vec.w);
+ device const T* x_vec2 = reinterpret_cast<device const T*>(x + base_idx);
+ device const T* w_up_vec2 = reinterpret_cast<device const T*>(w_up_row + base_idx);
+ acc_up += float(x_vec2[0]) * float(w_up_vec2[0]);
+ acc_up += float(x_vec2[1]) * float(w_up_vec2[1]);
+ acc_up += float(x_vec2[2]) * float(w_up_vec2[2]);
+ acc_up += float(x_vec2[3]) * float(w_up_vec2[3]);
+
+ if (gating_sel > 1) {
+ device const T* w_gate_vec2 = reinterpret_cast<device const T*>(w_gate_row + base_idx);
+ acc_gate += float(x_vec2[0]) * float(w_gate_vec2[0]);
+ acc_gate += float(x_vec2[1]) * float(w_gate_vec2[1]);
+ acc_gate += float(x_vec2[2]) * float(w_gate_vec2[2]);
+ acc_gate += float(x_vec2[3]) * float(w_gate_vec2[3]);
}
}
@@ -94,13 +80,13 @@ inline void moe_experts_decode_single_pass_a_impl(
for (uint idx = leftover_start; idx < d_model; idx += 32) {
float xv = float(x[idx]);
acc_up += xv * float(w_up_row[idx]);
- if (GATING_SEL > 1) {
+ if (gating_sel > 1) {
acc_gate += xv * float(w_gate_row[idx]);
}
}
acc_up = simd_sum(acc_up);
- if (GATING_SEL > 1) {
+ if (gating_sel > 1) {
acc_gate = simd_sum(acc_gate);
}
@@ -112,16 +98,16 @@ inline void moe_experts_decode_single_pass_a_impl(
);
float activated;
- if (GATING_SEL <= 1) {
+ if (gating_sel <= 1) {
activated =
- (GATING_SEL == 0) ? gelu_approx(up_val) : silu(up_val, silu_alpha);
+ (gating_sel == 0) ? gelu_approx(up_val) : silu(up_val, silu_alpha);
} else {
float gate_val = clamp(
acc_gate + float(biases[bias_base + d_ff + h_idx]),
gate_clip_min,
gate_clip_max
);
- float gate_act = (GATING_SEL == 2) ? silu(gate_val, silu_alpha)
+ float gate_act = (gating_sel == 2) ? silu(gate_val, silu_alpha)
: gelu_approx(gate_val);
activated = gate_act * up_val;
}
@@ -130,50 +116,6 @@ inline void moe_experts_decode_single_pass_a_impl(
}
}
-#define MOE_DECODE_SINGLE_PASS_A_KERNEL(DTYPE, DTYPE4, SUFFIX) \
- [[max_total_threads_per_threadgroup(128)]] \
- kernel void moe_experts_decode_single_pass_a_##SUFFIX( \
- device const DTYPE* x [[buffer(0)]], \
- device const int* topk_ids [[buffer(1)]], \
- device const DTYPE* W13_all [[buffer(2)]], \
- device const DTYPE* biases [[buffer(3)]], \
- device float* hidden_out [[buffer(4)]], \
- constant uint& d_model [[buffer(5)]], \
- constant uint& d_ff [[buffer(6)]], \
- constant uint& K [[buffer(7)]], \
- constant float& silu_alpha [[buffer(8)]], \
- constant float& gate_clip_min [[buffer(9)]], \
- constant float& gate_clip_max [[buffer(10)]], \
- constant float& up_clip_min [[buffer(11)]], \
- constant float& up_clip_max [[buffer(12)]], \
- uint2 tgpig [[threadgroup_position_in_grid]], \
- uint simd_gid [[simdgroup_index_in_threadgroup]], \
- uint simd_lid [[thread_index_in_simdgroup]] \
- ) { \
- moe_experts_decode_single_pass_a_impl<DTYPE, DTYPE4>( \
- x, \
- topk_ids, \
- W13_all, \
- biases, \
- hidden_out, \
- d_model, \
- d_ff, \
- K, \
- silu_alpha, \
- gate_clip_min, \
- gate_clip_max, \
- up_clip_min, \
- up_clip_max, \
- tgpig.y, \
- tgpig.x, \
- simd_gid, \
- simd_lid \
- ); \
- }
-
-MOE_DECODE_SINGLE_PASS_A_KERNEL(half, half4, f16)
-MOE_DECODE_SINGLE_PASS_A_KERNEL(bfloat, bfloat4, bf16)
-MOE_DECODE_SINGLE_PASS_A_KERNEL(float, float4, f32)
// ============================================================================
// Pass B (fused with finalize): hidden[k] @ W2[expert] → y (directly)
@@ -181,22 +123,24 @@ MOE_DECODE_SINGLE_PASS_A_KERNEL(float, float4, f32)
// Each simdgroup computes one final output element
// Grid: (ceil(d_model/8), 1) - NOT per K!
// ============================================================================
-
-template <typename T, typename T4>
-inline void moe_experts_decode_single_pass_b_impl(
+template <typename T>
+VARIANTS(T, float, half, bfloat)
+KERNEL(MoeExpertsDecodeSinglePassB)(
device const float* hidden, // [K, d_ff]
device const int* topk_ids, // [K]
device const T* topk_probs, // [K]
- device const T* W2_all, // [E, d_model, d_ff]
+ device const T* w2_all, // [E, d_model, d_ff]
device const T* biases, // [E, d_model]
device T* y, // [d_model]
- uint d_model,
- uint d_ff,
- uint K,
- uint d_block,
- uint simd_gid,
- uint simd_lid
+ constant uint& d_model,
+ constant uint& d_ff,
+ constant uint& k_input,
+ const uint d_block GROUPS(d_model.div_ceil(8)),
+ const uint tid THREADS(256)
) {
+ const uint simd_gid = tid / SIMD_SIZE;
+ const uint simd_lid = tid % SIMD_SIZE;
+
const uint my_col = d_block * 8 + simd_gid;
if (my_col >= d_model)
return;
@@ -206,13 +150,13 @@ inline void moe_experts_decode_single_pass_b_impl(
float final_acc = 0.0f;
- // Loop over K experts
- for (uint k = 0; k < K; ++k) {
+ // Loop over k_input experts
+ for (uint k = 0; k < k_input; ++k) {
const uint expert_u = uint(topk_ids[k]);
const float prob = float(topk_probs[k]);
device const float* hidden_ptr = hidden + (ulong)k * (ulong)d_ff;
- device const T* w2_ptr = W2_all + (ulong)expert_u * w2_expert_stride +
+ device const T* w2_ptr = w2_all + (ulong)expert_u * w2_expert_stride +
(ulong)my_col * (ulong)d_ff;
float acc = 0.0f;
@@ -221,14 +165,12 @@ inline void moe_experts_decode_single_pass_b_impl(
for (uint i = 0; i < vec_iters; ++i) {
uint base_idx = i * 128 + simd_lid * 4;
- float4 h_vec =
- *reinterpret_cast<device const float4*>(hidden_ptr + base_idx);
- T4 w_vec = *reinterpret_cast<device const T4*>(w2_ptr + base_idx);
-
- acc += h_vec.x * float(w_vec.x);
- acc += h_vec.y * float(w_vec.y);
- acc += h_vec.z * float(w_vec.z);
- acc += h_vec.w * float(w_vec.w);
+ float4 h_vec = *reinterpret_cast<device const float4*>(hidden_ptr + base_idx);
+ device const T* w_vec2 = reinterpret_cast<device const T*>(w2_ptr + base_idx);
+ acc += h_vec.x * float(w_vec2[0]);
+ acc += h_vec.y * float(w_vec2[1]);
+ acc += h_vec.z * float(w_vec2[2]);
+ acc += h_vec.w * float(w_vec2[3]);
}
// Remainder
@@ -247,40 +189,4 @@ inline void moe_experts_decode_single_pass_b_impl(
if (simd_lid == 0) {
y[my_col] = T(final_acc);
}
-}
-
-#define MOE_DECODE_SINGLE_PASS_B_KERNEL(DTYPE, DTYPE4, SUFFIX) \
- [[max_total_threads_per_threadgroup(256)]] \
- kernel void moe_experts_decode_single_pass_b_##SUFFIX( \
- device const float* hidden [[buffer(0)]], \
- device const int* topk_ids [[buffer(1)]], \
- device const DTYPE* topk_probs [[buffer(2)]], \
- device const DTYPE* W2_all [[buffer(3)]], \
- device const DTYPE* biases [[buffer(4)]], \
- device DTYPE* y [[buffer(5)]], \
- constant uint& d_model [[buffer(6)]], \
- constant uint& d_ff [[buffer(7)]], \
- constant uint& K [[buffer(8)]], \
- uint tgpig [[threadgroup_position_in_grid]], \
- uint simd_gid [[simdgroup_index_in_threadgroup]], \
- uint simd_lid [[thread_index_in_simdgroup]] \
- ) { \
- moe_experts_decode_single_pass_b_impl<DTYPE, DTYPE4>( \
- hidden, \
- topk_ids, \
- topk_probs, \
- W2_all, \
- biases, \
- y, \
- d_model, \
- d_ff, \
- K, \
- tgpig, \
- simd_gid, \
- simd_lid \
- ); \
- }
-
-MOE_DECODE_SINGLE_PASS_B_KERNEL(half, half4, f16)
-MOE_DECODE_SINGLE_PASS_B_KERNEL(bfloat, bfloat4, bf16)
-MOE_DECODE_SINGLE_PASS_B_KERNEL(float, float4, f32)
+}
\ No newline at end of file
diff --git a/crates/uzu/src/backends/metal/kernel/moe/experts_single.rs b/crates/uzu/src/backends/metal/kernel/moe/experts_single.rs
new file mode 100644
index 000000000..14ed8876a
--- /dev/null
+++ b/crates/uzu/src/backends/metal/kernel/moe/experts_single.rs
@@ -0,0 +1,158 @@
+use crate::backends::common::kernel::{
+ MoeExpertsDecodeSinglePassAKernel as _,
+ MoeExpertsDecodeSinglePassBKernel as _,
+};
+use crate::backends::metal::kernel::dsl::{
+ MoeExpertsDecodeSinglePassAMetalKernel,
+ MoeExpertsDecodeSinglePassBMetalKernel,
+};
+use crate::backends::metal::{KernelDataType, MTLContext, MTLError};
+use metal::{MTLBuffer, MTLCommandBuffer, MTLCommandEncoder};
+use objc2::__framework_prelude::ProtocolObject;
+use objc2::Message;
+use objc2::rc::Retained;
+
+static DTYPES: [KernelDataType; 3] = [
+ KernelDataType::Float16,
+ KernelDataType::BFloat16,
+ KernelDataType::Float32,
+];
+
+/// Arguments for single-token MoE decode (T=1 optimized path)
+#[derive(Debug)]
+pub struct MoeExpertsSingleDecodeArguments<'a> {
+ /// Input activation [d_model]
+ pub x: &'a ProtocolObject<dyn MTLBuffer>,
+ /// Top-K expert indices from router [K]
+ pub topk_ids: &'a ProtocolObject<dyn MTLBuffer>,
+ /// Top-K probabilities from router [K]
+ pub topk_probs: &'a ProtocolObject<dyn MTLBuffer>,
+ /// Up/gate projection weights [E, 2*d_ff, d_model]
+ pub w13_all: &'a ProtocolObject<dyn MTLBuffer>,
+ /// Down projection weights [E, d_model, d_ff]
+ pub w2_all: &'a ProtocolObject<dyn MTLBuffer>,
+ /// Up/gate biases [E, 2*d_ff]
+ pub up_biases: &'a ProtocolObject<dyn MTLBuffer>,
+ /// Down biases [E, d_model]
+ pub down_biases: &'a ProtocolObject<dyn MTLBuffer>,
+ /// Hidden buffer [K, d_ff] - intermediate storage (f32)
+ pub hidden: &'a ProtocolObject<dyn MTLBuffer>,
+ /// Final output [d_model]
+ pub y: &'a ProtocolObject<dyn MTLBuffer>,
+ /// Model dimension
+ pub d_model: usize,
+ /// FFN hidden dimension
+ pub d_ff: usize,
+ /// Number of active experts per token
+ pub k: usize,
+ /// Gating activation: 0=GELU, 1=SiLU, 2=SwiGLU, 3=GEGLU
+ pub gating_code: u32,
+ /// SiLU activation alpha parameter
+ pub silu_alpha: f32,
+ /// Gate clipping min
+ pub gate_clip_min: f32,
+ /// Gate clipping max
+ pub gate_clip_max: f32,
+ /// Up clipping min
+ pub up_clip_min: f32,
+ /// Up clipping max
+ pub up_clip_max: f32,
+ /// Data type
+ pub data_type: KernelDataType,
+}
+
+pub struct MoeExpertsSingleDecodeKernels {
+ pass_a: Vec<Vec<MoeExpertsDecodeSinglePassAMetalKernel>>,
+ pass_b: Vec<MoeExpertsDecodeSinglePassBMetalKernel>,
+}
+
+impl MoeExpertsSingleDecodeKernels {
+ pub fn new(ctx: &MTLContext) -> Result<Self, MTLError> {
+ let mut pass_a = vec![];
+ for gate in 0..4 {
+ let mut kernels = vec![];
+ for dtype in &DTYPES {
+ let kernel = MoeExpertsDecodeSinglePassAMetalKernel::new(
+ ctx,
+ (*dtype).into(),
+ gate,
+ )?;
+ kernels.push(kernel)
+ }
+ pass_a.push(kernels);
+ }
+
+ let mut pass_b = Vec::with_capacity(DTYPES.len());
+ for dtype in &DTYPES {
+ pass_b.push(MoeExpertsDecodeSinglePassBMetalKernel::new(
+ ctx,
+ (*dtype).into(),
+ )?);
+ }
+
+ Ok(Self {
+ pass_a,
+ pass_b,
+ })
+ }
+
+ pub fn encode(
+ &self,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
+ args: MoeExpertsSingleDecodeArguments,
+ ) {
+ if args.k == 0 {
+ return;
+ }
+
+ let gate_idx = args.gating_code.min(3) as usize;
+ let dtype_idx =
+ DTYPES.iter().position(|dtype| *dtype == args.data_type).unwrap();
+
+ // Pass A: x @ W13[expert] -> hidden
+ {
+ let kernel = &self.pass_a[gate_idx][dtype_idx];
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+ kernel.encode(
+ &args.x.retain(),
+ &args.topk_ids.retain(),
+ &args.w13_all.retain(),
+ &args.up_biases.retain(),
+ &args.hidden.retain(),
+ args.d_model as u32,
+ args.d_ff as u32,
+ args.k as u32,
+ args.silu_alpha,
+ args.gate_clip_min,
+ args.gate_clip_max,
+ args.up_clip_min,
+ args.up_clip_max,
+ &encoder,
+ );
+ encoder.end_encoding();
+ }
+
+ // Pass B: 8 simdgroups (256 threads), outputs final y directly
+ {
+ let kernel = &self.pass_b[dtype_idx];
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+ kernel.encode(
+ &args.hidden.retain(),
+ &args.topk_ids.retain(),
+ &args.topk_probs.retain(),
+ &args.w2_all.retain(),
+ &args.down_biases.retain(),
+ &args.y.retain(),
+ args.d_model as u32,
+ args.d_ff as u32,
+ args.k as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
+ }
+ }
+}
diff --git a/crates/uzu/src/backends/metal/kernel/moe/experts_two_pass_decode.metal b/crates/uzu/src/backends/metal/kernel/moe/experts_two_pass_decode.metal
index 8e78049db..7f50ee8b6 100644
--- a/crates/uzu/src/backends/metal/kernel/moe/experts_two_pass_decode.metal
+++ b/crates/uzu/src/backends/metal/kernel/moe/experts_two_pass_decode.metal
@@ -1,50 +1,44 @@
#include <metal_stdlib>
#include <metal_simdgroup>
+#include "../definitions.metal"
+#include "moe_commons.h"
using namespace metal;
-constant uint GATING_SEL
- [[function_constant(30)]]; // 0=GELU,1=SiLU,2=SwiGLU,3=GEGLU
-
-static inline float gelu_approx(float x) {
- const float k0 = 0.7978845608f;
- const float k1 = 0.044715f;
- if (x > 10.0f)
- return x;
- if (x < -10.0f)
- return 0.0f;
- return 0.5f * x *
- (1.0f + tanh(clamp(k0 * (x + k1 * x * x * x), -10.0f, 10.0f)));
-}
-
-static inline float silu(float x, float alpha) {
- return x / (1.0f + exp(-alpha * x));
-}
-
// === Pass A: Vectorized GEMV with float4 loads ===
// Structure: 4 simdgroups (128 threads), each outputs 1 hidden element
// Each simdgroup: 32 threads reduce d_model using float4 vectorized loads
// Grid: (h_blocks, rows, 1) where h_blocks = ceil(d_ff/4)
-template <typename T, typename T4>
-void moe_experts_decode_pass_a_impl(
- device const T* X_perm, // [total_rows, d_model]
- device const uint* expert_offsets, // [E + 1]
- device const T* W13_all, // [E, 2*d_ff, d_model]
- device const T* up_biases, // [E, 2*d_ff]
- device float* hidden_out, // [total_rows, d_ff]
- uint d_model,
- uint d_ff,
- uint E,
- float gate_clip_min,
- float gate_clip_max,
- float up_clip_min,
- float up_clip_max,
- float silu_alpha,
- uint expert_idx,
- uint row_in_expert,
- uint h_block_idx,
- uint simd_gid,
- uint simd_lid
+#define SIMD_SIZE 32
+
+template <typename T>
+VARIANTS(T, float, half, bfloat)
+KERNEL(MoeExpertsDecodePassA)(
+ device const T* x_perm, // [total_rows, d_model]
+ device const uint* expert_offsets, // [E + 1]
+ device const T* w13_all, // [E, 2*d_ff, d_model]
+ device float* hidden_out, // [E, 2*d_ff]
+ device const T* up_biases, // [total_rows, d_ff]
+ constant uint& d_model,
+ constant uint& d_ff,
+ constant uint& e,
+ constant float& gate_clip_min,
+ constant float& gate_clip_max,
+ constant float& up_clip_min,
+ constant float& up_clip_max,
+ constant float& silu_alpha,
+ device const uint* tile_map,
+ const uint gating_sel SPECIALIZE,
+ const uint tile_idx GROUPS(INDIRECT),
+ const uint tid THREADS(128)
) {
+ const uint simd_gid = tid / SIMD_SIZE;
+ const uint simd_lid = tid % SIMD_SIZE;
+
+ // Read tile descriptor
+ const uint h_block_idx = tile_map[tile_idx * 3 + 0];
+ const uint expert_idx = tile_map[tile_idx * 3 + 1];
+ const uint row_in_expert = tile_map[tile_idx * 3 + 2];
+
// Validate row bounds
const uint seg_start = expert_offsets[expert_idx];
const uint seg_end = expert_offsets[expert_idx + 1];
@@ -63,10 +57,10 @@ void moe_experts_decode_pass_a_impl(
const ulong bias_base = (ulong)expert_idx * (ulong)(2 * d_ff);
const ulong x_row_base = (ulong)global_row * (ulong)d_model;
- device const T* x_ptr = X_perm + x_row_base;
- device const T* w_up_row = W13_all + w13_base + (ulong)h_idx * (ulong)d_model;
+ device const T* x_ptr = x_perm + x_row_base;
+ device const T* w_up_row = w13_all + w13_base + (ulong)h_idx * (ulong)d_model;
device const T* w_gate_row =
- W13_all + w13_base + (ulong)(d_ff + h_idx) * (ulong)d_model;
+ w13_all + w13_base + (ulong)(d_ff + h_idx) * (ulong)d_model;
float acc_up = 0.0f;
float acc_gate = 0.0f;
@@ -77,21 +71,19 @@ void moe_experts_decode_pass_a_impl(
for (uint i = 0; i < vec_iters; ++i) {
uint base_idx = i * 128 + simd_lid * 4;
- T4 x_vec = *reinterpret_cast<device const T4*>(x_ptr + base_idx);
- T4 w_up_vec = *reinterpret_cast<device const T4*>(w_up_row + base_idx);
-
- acc_up += float(x_vec.x) * float(w_up_vec.x);
- acc_up += float(x_vec.y) * float(w_up_vec.y);
- acc_up += float(x_vec.z) * float(w_up_vec.z);
- acc_up += float(x_vec.w) * float(w_up_vec.w);
-
- if (GATING_SEL > 1) {
- T4 w_gate_vec =
- *reinterpret_cast<device const T4*>(w_gate_row + base_idx);
- acc_gate += float(x_vec.x) * float(w_gate_vec.x);
- acc_gate += float(x_vec.y) * float(w_gate_vec.y);
- acc_gate += float(x_vec.z) * float(w_gate_vec.z);
- acc_gate += float(x_vec.w) * float(w_gate_vec.w);
+ device const T* x_vec = reinterpret_cast<device const T*>(x_ptr + base_idx);
+ device const T* w_up_vec = reinterpret_cast<device const T*>(w_up_row + base_idx);
+ acc_up += float(x_vec[0]) * float(w_up_vec[0]);
+ acc_up += float(x_vec[1]) * float(w_up_vec[1]);
+ acc_up += float(x_vec[2]) * float(w_up_vec[2]);
+ acc_up += float(x_vec[3]) * float(w_up_vec[3]);
+
+ if (gating_sel > 1) {
+ device const T* w_gate_vec = reinterpret_cast<device const T*>(w_gate_row + base_idx);
+ acc_gate += float(x_vec[0]) * float(w_gate_vec[0]);
+ acc_gate += float(x_vec[1]) * float(w_gate_vec[1]);
+ acc_gate += float(x_vec[2]) * float(w_gate_vec[2]);
+ acc_gate += float(x_vec[3]) * float(w_gate_vec[3]);
}
}
@@ -100,14 +92,14 @@ void moe_experts_decode_pass_a_impl(
for (uint idx = leftover_start; idx < d_model; idx += 32) {
float xv = float(x_ptr[idx]);
acc_up += xv * float(w_up_row[idx]);
- if (GATING_SEL > 1) {
+ if (gating_sel > 1) {
acc_gate += xv * float(w_gate_row[idx]);
}
}
// Simdgroup reduction
acc_up = simd_sum(acc_up);
- if (GATING_SEL > 1) {
+ if (gating_sel > 1) {
acc_gate = simd_sum(acc_gate);
}
@@ -117,14 +109,12 @@ void moe_experts_decode_pass_a_impl(
up_val = clamp(up_val, up_clip_min, up_clip_max);
float activated;
- if (GATING_SEL <= 1) {
- activated =
- (GATING_SEL == 0) ? gelu_approx(up_val) : silu(up_val, silu_alpha);
+ if (gating_sel <= 1) {
+ activated = (gating_sel == 0) ? gelu_approx(up_val) : silu(up_val, silu_alpha);
} else {
float gate_val = acc_gate + float(up_biases[bias_base + d_ff + h_idx]);
gate_val = clamp(gate_val, gate_clip_min, gate_clip_max);
- float gate_act = (GATING_SEL == 2) ? silu(gate_val, silu_alpha)
- : gelu_approx(gate_val);
+ float gate_act = (gating_sel == 2) ? silu(gate_val, silu_alpha) : gelu_approx(gate_val);
activated = gate_act * up_val;
}
@@ -132,273 +122,35 @@ void moe_experts_decode_pass_a_impl(
}
}
-// === Helper kernels for indirect dispatch of Pass A ===
-
-// Count tiles per expert: tiles = (num_rows > 0) ? num_rows * h_blocks : 0
-[[max_total_threads_per_threadgroup(256)]]
-kernel void moe_pass_a_tile_counts(
- device const uint* expert_offsets [[buffer(0)]], // [E+1]
- device uint* tile_counts [[buffer(1)]], // [E]
- constant uint& E [[buffer(2)]],
- constant uint& h_blocks [[buffer(3)]],
- uint tid [[thread_position_in_grid]]
-) {
- if (tid >= E)
- return;
- const uint start = expert_offsets[tid];
- const uint end = expert_offsets[tid + 1];
- const uint num_rows = end - start;
- tile_counts[tid] = (num_rows > 0) ? (num_rows * h_blocks) : 0;
-}
-
-// Exclusive scan of tile_counts to get tile_offsets and total_tiles
-[[max_total_threads_per_threadgroup(1024)]]
-kernel void moe_pass_a_tile_scan(
- device const uint* tile_counts [[buffer(0)]], // [E]
- device uint* tile_offsets [[buffer(1)]], // [E+1]
- device uint* total_tiles [[buffer(2)]], // [1]
- constant uint& E [[buffer(3)]],
- uint lid [[thread_index_in_threadgroup]],
- threadgroup uint* scratch [[threadgroup(0)]]
-) {
- // Simple single-threadgroup scan (works for E <= 1024)
- const uint idx = lid;
-
- // Load into threadgroup memory
- if (idx < E) {
- scratch[idx] = tile_counts[idx];
- } else {
- scratch[idx] = 0;
- }
- threadgroup_barrier(mem_flags::mem_threadgroup);
-
- // Kogge-Stone scan
- uint val = scratch[idx];
- for (uint offset = 1; offset < 1024; offset *= 2) {
- uint temp = 0;
- if (idx >= offset && idx < E) {
- temp = scratch[idx - offset];
- }
- threadgroup_barrier(mem_flags::mem_threadgroup);
- if (idx >= offset && idx < E) {
- val += temp;
- scratch[idx] = val;
- }
- threadgroup_barrier(mem_flags::mem_threadgroup);
- }
-
- // Write exclusive scan (shift right by 1)
- if (idx == 0) {
- tile_offsets[0] = 0;
- }
- if (idx < E) {
- tile_offsets[idx + 1] = scratch[idx];
- if (idx == E - 1) {
- total_tiles[0] = scratch[idx];
- }
- }
-}
-
-// Build row→expert map: one thread per routed row
-[[max_total_threads_per_threadgroup(256)]]
-kernel void moe_pass_a_build_row_map(
- device const uint* expert_offsets [[buffer(0)]], // [E+1]
- device uint* row_expert_map [[buffer(1)]], // [total_rows]
- constant uint& total_rows [[buffer(2)]],
- constant uint& E [[buffer(3)]],
- uint tid [[thread_position_in_grid]]
-) {
- if (tid >= total_rows)
- return;
-
- uint left = 0u;
- uint right = E;
- const uint row = tid;
-
- while (left + 1u < right) {
- const uint mid = (left + right) >> 1;
- if (row < expert_offsets[mid]) {
- right = mid;
- } else {
- left = mid;
- }
- }
-
- row_expert_map[row] = left;
-}
-
-// Build tile map entries from row→expert map
-[[max_total_threads_per_threadgroup(256)]]
-kernel void moe_pass_a_build_tile_map(
- device const uint* expert_offsets [[buffer(0)]], // [E+1]
- device const uint* tile_offsets [[buffer(1)]], // [E+1]
- device const uint* row_expert_map [[buffer(2)]], // [total_rows]
- device uint* tile_map [[buffer(3)]], // [total_tiles * 3]
- constant uint& total_rows [[buffer(4)]],
- constant uint& h_blocks [[buffer(5)]],
- uint tid [[thread_position_in_grid]]
-) {
- const uint total_tiles = total_rows * h_blocks;
- if (tid >= total_tiles)
- return;
-
- const uint row_idx = tid / h_blocks;
- const uint h_block = tid % h_blocks;
-
- if (row_idx >= total_rows)
- return;
-
- const uint expert_idx = row_expert_map[row_idx];
- const uint row_start = expert_offsets[expert_idx];
- const uint row_in_expert = row_idx - row_start;
- const uint tile_base =
- tile_offsets[expert_idx] + row_in_expert * h_blocks + h_block;
-
- tile_map[tile_base * 3 + 0] = h_block;
- tile_map[tile_base * 3 + 1] = expert_idx;
- tile_map[tile_base * 3 + 2] = row_in_expert;
-}
-
-// Write dispatch args for indirect dispatch (reusable from tiled version)
-[[max_total_threads_per_threadgroup(1)]]
-kernel void moe_pass_a_write_dispatch_args(
- device const uint* total_tiles [[buffer(0)]], // [1]
- device uint* dispatch_args
- [[buffer(1)]], // [3] - MTLDispatchThreadgroupsIndirectArguments
- constant uint& num_tiles_y [[buffer(2)]], // usually 1 for Pass A
- uint tid [[thread_position_in_grid]]
-) {
- if (tid > 0)
- return;
- dispatch_args[0] = total_tiles[0]; // x dimension = total tiles
- dispatch_args[1] = num_tiles_y; // y dimension
- dispatch_args[2] = 1; // z dimension
-}
-
-// Modified Pass A that reads from tile map for indirect dispatch
-template <typename T, typename T4>
-void moe_experts_decode_pass_a_indirect_impl(
- device const T* X_perm,
- device const uint* expert_offsets,
- device const T* W13_all,
- device const T* up_biases,
- device float* hidden_out,
- device const uint* tile_map, // [total_tiles * 3]
- uint d_model,
- uint d_ff,
- uint E,
- float gate_clip_min,
- float gate_clip_max,
- float up_clip_min,
- float up_clip_max,
- float silu_alpha,
- uint tile_idx, // flat threadgroup index
- uint simd_gid,
- uint simd_lid
-) {
- // Read tile descriptor
- const uint h_block_idx = tile_map[tile_idx * 3 + 0];
- const uint expert_idx = tile_map[tile_idx * 3 + 1];
- const uint row_in_expert = tile_map[tile_idx * 3 + 2];
-
- // Call original implementation
- moe_experts_decode_pass_a_impl<T, T4>(
- X_perm,
- expert_offsets,
- W13_all,
- up_biases,
- hidden_out,
- d_model,
- d_ff,
- E,
- gate_clip_min,
- gate_clip_max,
- up_clip_min,
- up_clip_max,
- silu_alpha,
- expert_idx,
- row_in_expert,
- h_block_idx,
- simd_gid,
- simd_lid
- );
-}
-
-#define MOE_PASS_A_INDIRECT_KERNEL(DTYPE, DTYPE4, SUFFIX) \
- [[max_total_threads_per_threadgroup(128)]] \
- kernel void moe_experts_decode_pass_a_indirect_##SUFFIX( \
- device const DTYPE* X_perm [[buffer(0)]], \
- device const uint* expert_offsets [[buffer(1)]], \
- device const DTYPE* W13_all [[buffer(2)]], \
- device float* hidden_out [[buffer(3)]], \
- device const DTYPE* up_biases [[buffer(4)]], \
- constant uint& d_model [[buffer(5)]], \
- constant uint& d_ff [[buffer(6)]], \
- constant uint& E [[buffer(7)]], \
- constant float& gate_clip_min [[buffer(8)]], \
- constant float& gate_clip_max [[buffer(9)]], \
- constant float& up_clip_min [[buffer(10)]], \
- constant float& up_clip_max [[buffer(11)]], \
- constant float& silu_alpha [[buffer(12)]], \
- device const uint* tile_map [[buffer(13)]], \
- uint3 tgpig [[threadgroup_position_in_grid]], \
- uint simd_gid [[simdgroup_index_in_threadgroup]], \
- uint simd_lid [[thread_index_in_simdgroup]] \
- ) { \
- moe_experts_decode_pass_a_indirect_impl<DTYPE, DTYPE4>( \
- X_perm, \
- expert_offsets, \
- W13_all, \
- up_biases, \
- hidden_out, \
- tile_map, \
- d_model, \
- d_ff, \
- E, \
- gate_clip_min, \
- gate_clip_max, \
- up_clip_min, \
- up_clip_max, \
- silu_alpha, \
- tgpig.x, \
- simd_gid, \
- simd_lid \
- ); \
- }
-
-MOE_PASS_A_INDIRECT_KERNEL(bfloat, bfloat4, bf16)
-MOE_PASS_A_INDIRECT_KERNEL(half, half4, f16)
-MOE_PASS_A_INDIRECT_KERNEL(float, float4, f32)
-
// === Pass B: Simdgroup cooperation along K for coalescing ===
// W2 layout [E, d_model, d_ff] - 32 threads cooperate on one output, reading
// consecutive K elements
+#define THREADS_PER_SIMD 32
+#define SIMDGROUPS_PER_TG 8
template <typename T, typename AccumT>
-void moe_experts_decode_down_fused_2d_impl(
- device const float* hidden, // [total_rows, d_ff] - f32 from Pass A
- device const uint*
- row_expert_map, // [total_rows] - direct row->expert lookup
- device const T* w2_all, // [E, d_model, d_ff] - layout
- device const T* down_biases, // [E, d_model]
- device T* y_out, // [total_rows, d_model]
- uint total_rows,
- uint d_model,
- uint d_ff,
- uint E,
- uint2 tgpig,
- uint simd_gid,
- uint simd_lid
+VARIANTS(T, float, half, bfloat)
+VARIANTS(AccumT, float)
+KERNEL(MoeExpertsDecodeDownFused2D)(
+ device const float* hidden, // [total_rows, d_ff] - f32 from Pass A
+ device const uint* row_expert_map, // [total_rows] - direct row->expert lookup
+ device const T* w2_all, // [E, d_model, d_ff] - layout
+ device const T* down_biases, // [E, d_model]
+ device T* y_out, // [total_rows, d_model]
+ constant uint& total_rows,
+ constant uint& d_model,
+ constant uint& d_ff,
+ constant uint& e,
+ const uint tgpig_x GROUPS(d_model.div_ceil(SIMDGROUPS_PER_TG)),
+ const uint tgpig_y GROUPS(total_rows),
+ const uint tid THREADS(256)
) {
- constexpr uint THREADS_PER_SIMD = 32;
- constexpr uint SIMDGROUPS_PER_TG = 8;
-
- const uint row_idx = tgpig.y;
- if (row_idx >= total_rows)
- return;
+ const uint simd_gid = tid / SIMD_SIZE;
+ const uint simd_lid = tid % SIMD_SIZE;
+ const uint row_idx = tgpig_y;
// Each simdgroup computes one output column
- const uint my_col = tgpig.x * SIMDGROUPS_PER_TG + simd_gid;
+ const uint my_col = tgpig_x * SIMDGROUPS_PER_TG + simd_gid;
if (my_col >= d_model)
return;
@@ -432,22 +184,14 @@ void moe_experts_decode_down_fused_2d_impl(
const AccumT h7 = hidden[hidden_base + k_base + 7 * THREADS_PER_SIMD];
// W2: lane-coalesced
- const AccumT w0 =
- AccumT(w2_all[w2_col_base + k_base + 0 * THREADS_PER_SIMD]);
- const AccumT w1 =
- AccumT(w2_all[w2_col_base + k_base + 1 * THREADS_PER_SIMD]);
- const AccumT w2 =
- AccumT(w2_all[w2_col_base + k_base + 2 * THREADS_PER_SIMD]);
- const AccumT w3 =
- AccumT(w2_all[w2_col_base + k_base + 3 * THREADS_PER_SIMD]);
- const AccumT w4 =
- AccumT(w2_all[w2_col_base + k_base + 4 * THREADS_PER_SIMD]);
- const AccumT w5 =
- AccumT(w2_all[w2_col_base + k_base + 5 * THREADS_PER_SIMD]);
- const AccumT w6 =
- AccumT(w2_all[w2_col_base + k_base + 6 * THREADS_PER_SIMD]);
- const AccumT w7 =
- AccumT(w2_all[w2_col_base + k_base + 7 * THREADS_PER_SIMD]);
+ const AccumT w0 = AccumT(w2_all[w2_col_base + k_base + 0 * THREADS_PER_SIMD]);
+ const AccumT w1 = AccumT(w2_all[w2_col_base + k_base + 1 * THREADS_PER_SIMD]);
+ const AccumT w2 = AccumT(w2_all[w2_col_base + k_base + 2 * THREADS_PER_SIMD]);
+ const AccumT w3 = AccumT(w2_all[w2_col_base + k_base + 3 * THREADS_PER_SIMD]);
+ const AccumT w4 = AccumT(w2_all[w2_col_base + k_base + 4 * THREADS_PER_SIMD]);
+ const AccumT w5 = AccumT(w2_all[w2_col_base + k_base + 5 * THREADS_PER_SIMD]);
+ const AccumT w6 = AccumT(w2_all[w2_col_base + k_base + 6 * THREADS_PER_SIMD]);
+ const AccumT w7 = AccumT(w2_all[w2_col_base + k_base + 7 * THREADS_PER_SIMD]);
// dual trees for ILP
acc0 = fma(h0, w0, acc0);
@@ -493,40 +237,4 @@ void moe_experts_decode_down_fused_2d_impl(
const ulong out_idx = (ulong)row_idx * (ulong)d_model + (ulong)my_col;
y_out[out_idx] = T(result);
}
-}
-
-#define MOE_PASS_B_FUSED_2D_KERNEL(DTYPE, SUFFIX) \
- [[max_total_threads_per_threadgroup(256)]] \
- kernel void moe_experts_decode_down_fused_2d_##SUFFIX( \
- device const float* hidden [[buffer(0)]], \
- device const uint* row_expert_map [[buffer(1)]], \
- device const DTYPE* w2_all [[buffer(2)]], \
- device const DTYPE* down_biases [[buffer(3)]], \
- device DTYPE* y_out [[buffer(4)]], \
- constant uint& total_rows [[buffer(5)]], \
- constant uint& d_model [[buffer(6)]], \
- constant uint& d_ff [[buffer(7)]], \
- constant uint& E [[buffer(8)]], \
- uint2 tgpig [[threadgroup_position_in_grid]], \
- uint simd_gid [[simdgroup_index_in_threadgroup]], \
- uint simd_lid [[thread_index_in_simdgroup]] \
- ) { \
- moe_experts_decode_down_fused_2d_impl<DTYPE, float>( \
- hidden, \
- row_expert_map, \
- w2_all, \
- down_biases, \
- y_out, \
- total_rows, \
- d_model, \
- d_ff, \
- E, \
- tgpig, \
- simd_gid, \
- simd_lid \
- ); \
- }
-
-MOE_PASS_B_FUSED_2D_KERNEL(bfloat, bf16)
-MOE_PASS_B_FUSED_2D_KERNEL(half, f16)
-MOE_PASS_B_FUSED_2D_KERNEL(float, f32)
+}
\ No newline at end of file
diff --git a/crates/uzu/src/backends/metal/kernel/moe/experts_two_pass_decode.rs b/crates/uzu/src/backends/metal/kernel/moe/experts_two_pass_decode.rs
new file mode 100644
index 000000000..29cceb288
--- /dev/null
+++ b/crates/uzu/src/backends/metal/kernel/moe/experts_two_pass_decode.rs
@@ -0,0 +1,175 @@
+use metal::{MTLCommandBuffer, MTLCommandEncoder};
+use objc2::{
+ __framework_prelude::{ProtocolObject, Retained},
+ Message,
+};
+
+use crate::{
+ DataType,
+ backends::{
+ common::kernel::{
+ MoeExpertsDecodeDownFused2DKernel, MoeExpertsDecodePassAKernel,
+ },
+ metal::{
+ MTLContext, MTLError,
+ kernel::{
+ MoeExpertsTwoPassArguments,
+ dsl::{
+ MoeExpertsDecodeDownFused2DMetalKernel,
+ MoeExpertsDecodePassAMetalKernel,
+ },
+ moe::{
+ MoePassARowMapArguments, MoePassATileBuildArguments,
+ MoePassATileCountsArguments, MoePassATileDispatchArguments,
+ MoePassATileKernels, MoePassATileScanArguments,
+ dtype_index,
+ },
+ },
+ },
+ },
+};
+
+static DTYPES: [DataType; 3] = [DataType::F16, DataType::BF16, DataType::F32];
+
+pub struct MoeExpertsTwoPassDecodeKernels {
+ pass_a_tile: MoePassATileKernels,
+ pass_a_indirect: Vec<Vec<MoeExpertsDecodePassAMetalKernel>>,
+ fused_down: Vec<MoeExpertsDecodeDownFused2DMetalKernel>,
+}
+
+impl MoeExpertsTwoPassDecodeKernels {
+ pub fn new(ctx: &MTLContext) -> Result<Self, MTLError> {
+ let mut pass_a_indirect = vec![];
+ for gate in 0u32..4u32 {
+ let mut kernels = vec![];
+ for dtype in &DTYPES {
+ let kernel =
+ MoeExpertsDecodePassAMetalKernel::new(ctx, *dtype, gate)?;
+ kernels.push(kernel);
+ }
+ pass_a_indirect.push(kernels);
+ }
+
+ let mut fused_down = Vec::with_capacity(DTYPES.len());
+ for dtype in &DTYPES {
+ let kernel = MoeExpertsDecodeDownFused2DMetalKernel::new(
+ ctx,
+ *dtype,
+ DataType::F32,
+ )?;
+ fused_down.push(kernel);
+ }
+
+ Ok(Self {
+ pass_a_tile: MoePassATileKernels::new(ctx)?,
+ pass_a_indirect,
+ fused_down,
+ })
+ }
+
+ pub fn encode(
+ &self,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
+ args: &MoeExpertsTwoPassArguments,
+ ) {
+ if args.total_rows == 0 {
+ return;
+ }
+
+ // pass a tile
+ const BLOCK_M: u32 = 4;
+ let h_blocks = (args.d_ff as u32 + BLOCK_M - 1) / BLOCK_M;
+ self.pass_a_tile.encode_counts(
+ command_buffer,
+ &MoePassATileCountsArguments {
+ expert_offsets: args.expert_offsets,
+ tile_counts: args.tile_counts,
+ e: args.e,
+ h_blocks,
+ },
+ );
+ self.pass_a_tile.encode_scan(
+ command_buffer,
+ &MoePassATileScanArguments {
+ tile_counts: args.tile_counts,
+ tile_offsets: args.tile_offsets,
+ total_tiles: args.total_tiles,
+ e: args.e,
+ },
+ );
+ self.pass_a_tile.encode_row_map(
+ command_buffer,
+ &MoePassARowMapArguments {
+ expert_offsets: args.expert_offsets,
+ row_expert_map: args.row_expert_map,
+ total_rows: args.total_rows,
+ e: args.e,
+ },
+ );
+ self.pass_a_tile.encode_build_map(
+ command_buffer,
+ &MoePassATileBuildArguments {
+ expert_offsets: args.expert_offsets,
+ tile_offsets: args.tile_offsets,
+ row_expert_map: args.row_expert_map,
+ tile_map: args.tile_map,
+ total_rows: args.total_rows,
+ h_blocks,
+ },
+ );
+ self.pass_a_tile.encode_dispatch_args(
+ command_buffer,
+ &MoePassATileDispatchArguments {
+ total_tiles: args.total_tiles,
+ dispatch_args: args.dispatch_args,
+ num_tiles_y: 1,
+ },
+ );
+
+ // pass a
+ let gate_idx = args.gating_code.min(3) as usize;
+ let dtype_idx = dtype_index(args.data_type);
+ let pass_a_encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+ let pass_a_kernel = &self.pass_a_indirect[gate_idx][dtype_idx];
+ pass_a_kernel.encode(
+ &args.x_perm_buffer.retain(),
+ &args.expert_offsets.retain(),
+ &args.w13_all.retain(),
+ &args.hidden_buffer.retain(),
+ &args.up_biases.retain(),
+ args.d_model as u32,
+ args.d_ff as u32,
+ args.e as u32,
+ args.gate_clip_min,
+ args.gate_clip_max,
+ args.up_clip_min,
+ args.up_clip_max,
+ args.silu_alpha,
+ &args.tile_map.retain(),
+ &args.dispatch_args.retain(),
+ &pass_a_encoder,
+ );
+ pass_a_encoder.end_encoding();
+
+ // pass b
+ let pass_b_kernel = &self.fused_down[dtype_idx];
+ let pass_b_encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+ pass_b_kernel.encode(
+ &args.hidden_buffer.retain(),
+ &args.row_expert_map.retain(),
+ &args.w2_all.retain(),
+ &args.down_biases.retain(),
+ &args.output_buffer.retain(),
+ args.total_rows as u32,
+ args.d_model as u32,
+ args.d_ff as u32,
+ args.e as u32,
+ &pass_b_encoder,
+ );
+ pass_b_encoder.end_encoding();
+ }
+}
diff --git a/crates/uzu/src/backends/metal/kernel/moe/experts_two_pass_prefill.metal b/crates/uzu/src/backends/metal/kernel/moe/experts_two_pass_prefill.metal
index 42c13241e..f1818b697 100644
--- a/crates/uzu/src/backends/metal/kernel/moe/experts_two_pass_prefill.metal
+++ b/crates/uzu/src/backends/metal/kernel/moe/experts_two_pass_prefill.metal
@@ -2,6 +2,7 @@
#include <metal_simdgroup>
#include <metal_simdgroup_matrix>
#include "../quant_matmul/mma.h"
+#include "moe_commons.h"
using namespace metal;
@@ -12,23 +13,6 @@ static inline uint linear_tid(uint3 tid, uint3 tpg) {
return tid.z * (tpg.x * tpg.y) + tid.y * tpg.x + tid.x;
}
-static inline float gelu_approx(float x) {
- const float k0 = 0.7978845608f; // sqrt(2/pi)
- const float k1 = 0.044715f;
- if (x > 10.0f)
- return x;
- if (x < -10.0f)
- return 0.0f;
- float x3 = x * x * x;
- float t = clamp(k0 * (x + k1 * x3), -10.0f, 10.0f);
- return 0.5f * x * (1.0f + tanh(t));
-}
-
-static inline float silu(float x, float alpha) {
- // alpha=1 gives standard SiLU; keep alpha param for parity with your API
- return x / (1.0f + exp(-alpha * x));
-}
-
template <typename T>
inline void store_vec4(device T* dst, ulong base, float4 vals) {
dst[base + 0] = T(vals.x);
diff --git a/crates/uzu/src/backends/metal/kernel/moe/finalize.metal b/crates/uzu/src/backends/metal/kernel/moe/finalize.metal
index a64abac2c..f26f8117d 100644
--- a/crates/uzu/src/backends/metal/kernel/moe/finalize.metal
+++ b/crates/uzu/src/backends/metal/kernel/moe/finalize.metal
@@ -1,29 +1,26 @@
#include <metal_stdlib>
-using namespace metal;
+#include "../definitions.metal"
-// Defaults must match Rust launcher
#define BM 32
#define BN 64
template <typename T>
-inline void moe_finalize_impl(
+VARIANTS(T, float, half, bfloat)
+KERNEL(MoeFinalize)(
device const int* tok2row, // [T*K], -1 if dropped
device const T* probs, // [T*K]
- device const T* Y_partial, // [sum_k, d_model]
- device T* Y, // [T, d_model]
- constant uint& T_count,
+ device const T* y_partial, // [sum_k, d_model]
+ device T* y, // [T, d_model]
+ constant uint& t_count,
constant uint& d_model,
- constant uint& K,
- uint lid,
- uint3 tgpig
+ constant uint& k_input,
+ const uint lid THREADS(128),
+ const uint tg_n GROUPS(d_model.div_ceil(BN)),
+ const uint tg_m GROUPS(t_count.div_ceil(BM))
) {
- if (T_count == 0u || d_model == 0u)
- return;
- const uint tile_m0 = tgpig.y * BM;
- const uint tile_n0 = tgpig.x * BN;
- if (tile_m0 >= T_count || tile_n0 >= d_model)
- return;
- const uint m_rows = min((uint)BM, T_count - tile_m0);
+ const uint tile_n0 = tg_n * BN;
+ const uint tile_m0 = tg_m * BM;
+ const uint m_rows = min((uint)BM, t_count - tile_m0);
const uint n_cols = min((uint)BN, d_model - tile_n0);
// 128 threads per TG expected
@@ -33,8 +30,8 @@ inline void moe_finalize_impl(
const uint t = tile_m0 + mi;
const uint f = tile_n0 + nj;
float acc = 0.0f;
- const uint base = t * K;
- for (uint k = 0; k < K; ++k) {
+ const uint base = t * k_input;
+ for (uint k = 0; k < k_input; ++k) {
const int row = tok2row[base + k];
if (row >= 0) {
const ulong yidx = (ulong)(uint)row * (ulong)d_model + (ulong)f;
@@ -42,7 +39,7 @@ inline void moe_finalize_impl(
if (!isfinite(prob)) {
prob = 0.0f;
}
- float val = (float)Y_partial[yidx];
+ float val = (float)y_partial[yidx];
if (!isfinite(val)) {
val = 0.0f;
}
@@ -52,27 +49,6 @@ inline void moe_finalize_impl(
if (!isfinite(acc)) {
acc = 0.0f;
}
- Y[t * d_model + f] = T(acc);
+ y[t * d_model + f] = T(acc);
}
}
-
-#define DEFINE_MOE_FINALIZE_KERNEL(SUFFIX, T) \
- [[max_total_threads_per_threadgroup(128)]] \
- kernel void moe_finalize_##SUFFIX( \
- device const int* tok2row [[buffer(0)]], \
- device const T* probs [[buffer(1)]], \
- device const T* Y_partial [[buffer(2)]], \
- device T* Y [[buffer(3)]], \
- constant uint& T_count [[buffer(4)]], \
- constant uint& d_model [[buffer(5)]], \
- constant uint& K [[buffer(6)]], \
- uint lid [[thread_index_in_threadgroup]], \
- uint3 tgpig [[threadgroup_position_in_grid]] \
- ) { \
- moe_finalize_impl< \
- T>(tok2row, probs, Y_partial, Y, T_count, d_model, K, lid, tgpig); \
- }
-
-DEFINE_MOE_FINALIZE_KERNEL(f16, half)
-DEFINE_MOE_FINALIZE_KERNEL(bf16, bfloat)
-DEFINE_MOE_FINALIZE_KERNEL(f32, float)
diff --git a/crates/uzu/src/backends/metal/kernel/moe/finalize.rs b/crates/uzu/src/backends/metal/kernel/moe/finalize.rs
deleted file mode 100644
index 8a1d4e116..000000000
--- a/crates/uzu/src/backends/metal/kernel/moe/finalize.rs
+++ /dev/null
@@ -1,92 +0,0 @@
-use crate::backends::metal::{
- KernelDataType, MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- MTLComputePipelineState, MTLContext, MTLError, MTLSize, ProtocolObject, Retained,
- metal_extensions::ComputeEncoderSetValue,
-};
-
-// ---- Finalize Kernel ----
-
-#[derive(Debug, thiserror::Error)]
-pub enum MoeFinalizeError {
- #[error("Metal error: {0}")]
- MetalError(#[from] MTLError),
-}
-
-pub struct MoeFinalizeKernel {
- pipeline_f16: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- pipeline_bf16: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- pipeline_f32: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-#[derive(Debug)]
-pub struct MoeFinalizeArguments<'a> {
- pub tok2row_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [T*K] i32
- pub probs_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [T*K] f16/bf16
- pub y_partial_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [sum_k, d_model] f16
- pub y_out_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [T, d_model] f16
- pub t: usize,
- pub d_model: usize,
- pub k: usize,
-}
-
-impl MoeFinalizeKernel {
- pub fn new(ctx: &MTLContext) -> Result<Self, MoeFinalizeError> {
- let pipeline_f16 =
- ctx.compute_pipeline_state("moe_finalize_f16", None)?;
- let pipeline_bf16 =
- ctx.compute_pipeline_state("moe_finalize_bf16", None)?;
- let pipeline_f32 =
- ctx.compute_pipeline_state("moe_finalize_f32", None)?;
- Ok(Self {
- pipeline_f16,
- pipeline_bf16,
- pipeline_f32,
- })
- }
-
- pub fn encode(
- &self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- args: MoeFinalizeArguments,
- dtype: KernelDataType,
- ) -> Result<(), MoeFinalizeError> {
- let encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- match dtype {
- KernelDataType::Float16 => {
- encoder.set_compute_pipeline_state(&self.pipeline_f16);
- },
- KernelDataType::BFloat16 => {
- encoder.set_compute_pipeline_state(&self.pipeline_bf16);
- },
- KernelDataType::Float32 => {
- encoder.set_compute_pipeline_state(&self.pipeline_f32);
- },
- }
- encoder.set_buffer(Some(args.tok2row_buffer), 0, 0);
- encoder.set_buffer(Some(args.probs_buffer), 0, 1);
- encoder.set_buffer(Some(args.y_partial_buffer), 0, 2);
- encoder.set_buffer(Some(args.y_out_buffer), 0, 3);
- let t_u = args.t as u32;
- let dm_u = args.d_model as u32;
- let k_u = args.k as u32;
- encoder.set_value(&t_u, 4);
- encoder.set_value(&dm_u, 5);
- encoder.set_value(&k_u, 6);
-
- // Launch tiles over (N tiles, M tiles)
- const BM: usize = 32;
- const BN: usize = 64;
- let num_tiles_n = (args.d_model + BN - 1) / BN;
- let num_tiles_m = (args.t + BM - 1) / BM;
- let threads_per_threadgroup = MTLSize::new(128, 1, 1); // BM * BN * 32 = 1 * 4 * 32
- if num_tiles_m > 0 && num_tiles_n > 0 {
- let threadgroups = MTLSize::new(num_tiles_n, num_tiles_m, 1);
- encoder
- .dispatch_threadgroups(threadgroups, threads_per_threadgroup);
- }
- encoder.end_encoding();
- Ok(())
- }
-}
diff --git a/crates/uzu/src/backends/metal/kernel/moe/gather.metal b/crates/uzu/src/backends/metal/kernel/moe/gather.metal
new file mode 100644
index 000000000..1ef96ec33
--- /dev/null
+++ b/crates/uzu/src/backends/metal/kernel/moe/gather.metal
@@ -0,0 +1,112 @@
+#include <metal_stdlib>
+#include "../definitions.metal"
+
+// Tiling config for 2D gather
+#define ROWS_PER_TG 8
+#define COL_VECS 32u // 256 threads = 8×32
+#define ELEMS_PER_THREAD 8u // each thread copies 8 elements (2× vec4)
+#define THREADS_PER_THREADGROUP 256
+
+template <typename T>
+VARIANTS(T, float, half, bfloat)
+KERNEL(MoeGatherXPerm2D)(
+ device const T* x,
+ device const int* bucketed_ids,
+ device T* x_perm,
+ device const uint* sumk_buf,
+ constant uint& d_model,
+ constant uint& t,
+ constant uint& k,
+ const uint tgid_x GROUPS((t * k).div_ceil(ROWS_PER_TG)),
+ const uint lid THREADS(THREADS_PER_THREADGROUP)
+) {
+ const uint total_rows = sumk_buf[0];
+ if (total_rows == 0u || d_model == 0u) {
+ return;
+ }
+
+ // Thread organization: lid = row_in_tg * COL_VECS + col_vec
+ const uint row_in_tg = lid / COL_VECS;
+ const uint col_vec = lid % COL_VECS;
+
+ // Global row assignment
+ const uint base_row = tgid_x * ROWS_PER_TG;
+ const uint row = base_row + row_in_tg;
+ if (row >= total_rows) {
+ return;
+ }
+
+ int token = bucketed_ids[row];
+ if (token < 0) {
+ return;
+ }
+
+ const device T* src = x + (ulong)(uint)token * (ulong)d_model;
+ device T* dst = x_perm + (ulong)row * (ulong)d_model;
+
+ // Each thread copies ELEMS_PER_THREAD elements
+ // Thread i processes: col_vec*8, col_vec*8+256, col_vec*8+512, ...
+ uint col = col_vec * ELEMS_PER_THREAD;
+ const uint stride = COL_VECS * ELEMS_PER_THREAD;
+
+ // Vectorized copy: 8 elements = 2× vec4
+ using Vec4 = typename metal::vec<T, 4>;
+ while (col + 7u < d_model) {
+ Vec4 v0 = *(reinterpret_cast<const device Vec4*>(src + col));
+ Vec4 v1 = *(reinterpret_cast<const device Vec4*>(src + col + 4u));
+ *(reinterpret_cast<device Vec4*>(dst + col)) = v0;
+ *(reinterpret_cast<device Vec4*>(dst + col + 4u)) = v1;
+ col += stride;
+ }
+
+ // Scalar remainder
+ if (col < d_model) {
+ const uint limit = min(d_model, col + ELEMS_PER_THREAD);
+ for (uint tail = col; tail < limit; ++tail) {
+ dst[tail] = src[tail];
+ }
+ }
+}
+
+template <typename T>
+VARIANTS(T, float, half, bfloat)
+KERNEL(MoeGatherXPerm1D)(
+ device const T* x,
+ device const int* bucketed_ids,
+ device T* x_perm,
+ device const uint* sumk_buf,
+ constant uint& d_model,
+ constant uint& t,
+ constant uint& k,
+ const uint tgid_x GROUPS((t * k).div_ceil(THREADS_PER_THREADGROUP)),
+ const uint lid THREADS(THREADS_PER_THREADGROUP)
+) {
+ const uint total_rows = sumk_buf[0];
+ if (total_rows == 0u || d_model == 0u)
+ return;
+
+ const uint num_vec4 = d_model / 4u;
+
+ // Each thread processes multiple rows sequentially
+ for (uint row = lid; row < total_rows; row += THREADS_PER_THREADGROUP) {
+ int token = bucketed_ids[row];
+ if (token < 0)
+ continue;
+
+ const device T* src = x + (ulong)(uint)token * (ulong)d_model;
+ device T* dst = x_perm + (ulong)row * (ulong)d_model;
+
+ // Vectorized copy: 4 elements at a time
+ using Vec4 = typename metal::vec<T, 4>;
+ for (uint vec_idx = 0u; vec_idx < num_vec4; vec_idx++) {
+ uint col = vec_idx * 4u;
+ Vec4 v = *(reinterpret_cast<const device Vec4*>(src + col));
+ *(reinterpret_cast<device Vec4*>(dst + col)) = v;
+ }
+
+ // Scalar remainder
+ for (uint col = num_vec4 * 4u; col < d_model; col++) {
+ dst[col] = src[col];
+ }
+ }
+}
\ No newline at end of file
diff --git a/crates/uzu/src/backends/metal/kernel/moe/gather.rs b/crates/uzu/src/backends/metal/kernel/moe/gather.rs
index 21381547c..6196ec3e5 100644
--- a/crates/uzu/src/backends/metal/kernel/moe/gather.rs
+++ b/crates/uzu/src/backends/metal/kernel/moe/gather.rs
@@ -1,22 +1,18 @@
-use crate::backends::metal::{
- KernelDataType, MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- MTLComputePipelineState, MTLContext, MTLError, MTLSize, ProtocolObject, Retained,
- metal_extensions::ComputeEncoderSetValue,
-};
-
-// ---- Gather Permuted Activations Kernel ----
-
-#[derive(Debug, thiserror::Error)]
-pub enum MoeGatherError {
- #[error("Metal error: {0}")]
- MetalError(#[from] MTLError),
-}
+use objc2::Message;
-pub struct MoeGatherKernel {
- pipeline_f16: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- pipeline_f32: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- pipeline_bf16: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
+use crate::{
+ DataType,
+ backends::{
+ common::kernel::{MoeGatherXPerm1DKernel, MoeGatherXPerm2DKernel},
+ metal::{
+ KernelDataType, MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
+ MTLContext, MTLError, ProtocolObject, Retained,
+ kernel::dsl::{
+ MoeGatherXPerm1DMetalKernel, MoeGatherXPerm2DMetalKernel,
+ },
+ },
+ },
+};
#[derive(Debug)]
pub struct MoeGatherArguments<'a> {
@@ -29,68 +25,62 @@ pub struct MoeGatherArguments<'a> {
pub d_model: usize,
}
-impl MoeGatherKernel {
- pub fn new(ctx: &MTLContext) -> Result<Self, MoeGatherError> {
- let pipeline_f16 =
- ctx.compute_pipeline_state("moe_gather_x_perm_f16", None)?;
- let pipeline_f32 =
- ctx.compute_pipeline_state("moe_gather_x_perm_f32", None)?;
- let pipeline_bf16 =
- ctx.compute_pipeline_state("moe_gather_x_perm_bf16_2d", None)?;
+pub struct MoeGatherKernels {
+ bf16: MoeGatherXPerm2DMetalKernel,
+ f16: MoeGatherXPerm1DMetalKernel,
+ f32: MoeGatherXPerm1DMetalKernel,
+}
+
+impl MoeGatherKernels {
+ pub fn new(ctx: &MTLContext) -> Result<Self, MTLError> {
Ok(Self {
- pipeline_f16,
- pipeline_f32,
- pipeline_bf16,
+ bf16: MoeGatherXPerm2DMetalKernel::new(ctx, DataType::BF16)?,
+ f16: MoeGatherXPerm1DMetalKernel::new(ctx, DataType::F16)?,
+ f32: MoeGatherXPerm1DMetalKernel::new(ctx, DataType::F32)?,
})
}
pub fn encode(
&self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
dtype: KernelDataType,
- args: MoeGatherArguments,
- ) -> Result<(), MoeGatherError> {
+ args: &MoeGatherArguments,
+ ) {
let encoder = command_buffer
.new_compute_command_encoder()
.expect("Failed to create compute command encoder");
match dtype {
- KernelDataType::Float16 => {
- encoder.set_compute_pipeline_state(&self.pipeline_f16);
- },
- KernelDataType::Float32 => {
- encoder.set_compute_pipeline_state(&self.pipeline_f32);
- },
- KernelDataType::BFloat16 => {
- encoder.set_compute_pipeline_state(&self.pipeline_bf16);
- },
+ KernelDataType::Float32 => self.f32.encode(
+ &args.x_buffer.retain(),
+ &args.bucketed_ids_buffer.retain(),
+ &args.x_perm_buffer.retain(),
+ &args.sumk_buffer.retain(),
+ args.d_model as u32,
+ args.t as u32,
+ args.k as u32,
+ &encoder,
+ ),
+ KernelDataType::Float16 => self.f16.encode(
+ &args.x_buffer.retain(),
+ &args.bucketed_ids_buffer.retain(),
+ &args.x_perm_buffer.retain(),
+ &args.sumk_buffer.retain(),
+ args.d_model as u32,
+ args.t as u32,
+ args.k as u32,
+ &encoder,
+ ),
+ KernelDataType::BFloat16 => self.bf16.encode(
+ &args.x_buffer.retain(),
+ &args.bucketed_ids_buffer.retain(),
+ &args.x_perm_buffer.retain(),
+ &args.sumk_buffer.retain(),
+ args.d_model as u32,
+ args.t as u32,
+ args.k as u32,
+ &encoder,
+ ),
}
-
- encoder.set_buffer(Some(args.x_buffer), 0, 0);
- encoder.set_buffer(Some(args.bucketed_ids_buffer), 0, 1);
- encoder.set_buffer(Some(args.x_perm_buffer), 0, 2);
- encoder.set_buffer(Some(args.sumk_buffer), 0, 3);
- let d_model_u32 = args.d_model as u32;
- encoder.set_value(&d_model_u32, 4);
-
- let max_rows = args.t * args.k;
- if max_rows == 0 {
- encoder.end_encoding();
- return Ok(());
- }
- let threads_per_threadgroup = MTLSize::new(256, 1, 1);
- let threadgroups = match dtype {
- KernelDataType::BFloat16 => {
- const BF16_ROWS_PER_TG: usize = 8;
- MTLSize::new(
- (max_rows + BF16_ROWS_PER_TG - 1) / BF16_ROWS_PER_TG,
- 1,
- 1,
- )
- },
- _ => MTLSize::new((max_rows + 255) / 256, 1, 1),
- };
- encoder.dispatch_threadgroups(threadgroups, threads_per_threadgroup);
encoder.end_encoding();
- Ok(())
}
}
diff --git a/crates/uzu/src/backends/metal/kernel/moe/mod.rs b/crates/uzu/src/backends/metal/kernel/moe/mod.rs
index 3eff065a6..9300ddfeb 100644
--- a/crates/uzu/src/backends/metal/kernel/moe/mod.rs
+++ b/crates/uzu/src/backends/metal/kernel/moe/mod.rs
@@ -1,27 +1,25 @@
use crate::backends::metal::KernelDataType;
// Submodules
-mod counts_offsets_fused;
mod experts;
-mod finalize;
+mod experts_single;
+mod experts_two_pass_decode;
mod gather;
mod router_topk;
mod scatter;
-mod tiles;
+pub mod tiles_map;
+pub mod tiles_pass_a;
// Re-export public items from submodules
-pub use counts_offsets_fused::{
- MoeCountsOffsetsFusedArguments, MoeCountsOffsetsFusedError,
- MoeCountsOffsetsFusedKernel,
-};
pub use experts::{
- MoeExpertsArguments, MoeExpertsError, MoeExpertsSingleDecodeArguments,
- MoeExpertsSingleDecodeKernel, MoeExpertsTwoPassArguments,
- MoeExpertsTwoPassDecodeKernel, MoeExpertsTwoPassPrefillKernel,
- MoeScatterError,
+ MoeExpertsArguments, MoeExpertsError, MoeExpertsTwoPassArguments,
+ MoeExpertsTwoPassPrefillKernel, MoeScatterError,
+};
+pub use experts_single::{
+ MoeExpertsSingleDecodeArguments, MoeExpertsSingleDecodeKernels,
};
-pub use finalize::{MoeFinalizeArguments, MoeFinalizeError, MoeFinalizeKernel};
-pub use gather::{MoeGatherArguments, MoeGatherError, MoeGatherKernel};
+pub use experts_two_pass_decode::MoeExpertsTwoPassDecodeKernels;
+pub use gather::{MoeGatherArguments, MoeGatherKernels};
pub use router_topk::{
MoeRouterTopKArguments, MoeRouterTopKError, MoeRouterTopKKernel,
};
@@ -29,12 +27,10 @@ pub use scatter::{
MoeBlockBasesArguments, MoeScatterArguments, MoeScatterKernels,
MoeScatterWithMapArguments,
};
-pub use tiles::{
+pub use tiles_pass_a::{
MoePassARowMapArguments, MoePassATileBuildArguments,
MoePassATileCountsArguments, MoePassATileDispatchArguments,
- MoePassATileKernel, MoePassATileScanArguments, MoeTileCountsArguments,
- MoeTileDispatchArguments, MoeTileError, MoeTileMapBuildArguments,
- MoeTileMapKernel, MoeTileScanArguments,
+ MoePassATileKernels, MoePassATileScanArguments,
};
// Common utility functions
diff --git a/crates/uzu/src/backends/metal/kernel/moe/moe_commons.h b/crates/uzu/src/backends/metal/kernel/moe/moe_commons.h
new file mode 100644
index 000000000..cec3d51ab
--- /dev/null
+++ b/crates/uzu/src/backends/metal/kernel/moe/moe_commons.h
@@ -0,0 +1,25 @@
+#ifndef __MOE_COMMONS_H__
+#define __MOE_COMMONS_H__
+
+#include <metal_stdlib>
+
+static inline float gelu_approx(float x) {
+ if (x > 10.0f) {
+ return x;
+ }
+ if (x < -10.0f) {
+ return 0.0f;
+ }
+
+ const float GELU_APPROX_K0 = 0.7978845608f; // sqrt(2/pi)
+ const float GELU_APPROX_K1 = 0.044715f;
+ float t = metal::clamp(GELU_APPROX_K0 * (x + GELU_APPROX_K1 * x * x * x), -10.0f, 10.0f);
+ return 0.5f * x * (1.0f + metal::tanh(t));
+}
+
+static inline float silu(float x, float alpha) {
+ // alpha=1 gives standard SiLU; keep alpha param for parity with your API
+ return x / (1.0f + metal::exp(-alpha * x));
+}
+
+#endif // __MOE_COMMONS_H__
\ No newline at end of file
diff --git a/crates/uzu/src/backends/metal/kernel/moe/prepare.metal b/crates/uzu/src/backends/metal/kernel/moe/prepare.metal
deleted file mode 100644
index b32f844c6..000000000
--- a/crates/uzu/src/backends/metal/kernel/moe/prepare.metal
+++ /dev/null
@@ -1,155 +0,0 @@
-#include <metal_stdlib>
-using namespace metal;
-
-// Tiling config for 2D gather
-constant uint ROWS_PER_TG = 8u;
-constant uint COL_VECS = 32u; // 256 threads = 8×32
-constant uint ELEMS_PER_THREAD = 8u; // each thread copies 8 elements (2× vec4)
-
-// ========== 2D Tiled Gather Template ==========
-template <typename T>
-void moe_gather_x_perm_2d_impl(
- device const T* X,
- device const int* bucketed_ids,
- device T* x_perm,
- device const uint* sumk_buf,
- uint d_model,
- uint3 tgid,
- uint lid
-) {
- const uint total_rows = sumk_buf[0];
- if (total_rows == 0u || d_model == 0u)
- return;
-
- // Thread organization: lid = row_in_tg * COL_VECS + col_vec
- const uint row_in_tg = lid / COL_VECS;
- const uint col_vec = lid % COL_VECS;
-
- // Global row assignment
- const uint base_row = tgid.x * ROWS_PER_TG;
- const uint row = base_row + row_in_tg;
-
- if (row >= total_rows)
- return;
-
- int token = bucketed_ids[row];
- if (token < 0)
- return;
-
- const device T* src = X + (ulong)(uint)token * (ulong)d_model;
- device T* dst = x_perm + (ulong)row * (ulong)d_model;
-
- // Each thread copies ELEMS_PER_THREAD elements
- // Thread i processes: col_vec*8, col_vec*8+256, col_vec*8+512, ...
- uint col = col_vec * ELEMS_PER_THREAD;
- const uint stride = COL_VECS * ELEMS_PER_THREAD;
-
- // Vectorized copy: 8 elements = 2× vec4
- using Vec4 = typename metal::vec<T, 4>;
- while (col + 7u < d_model) {
- Vec4 v0 = *(reinterpret_cast<const device Vec4*>(src + col));
- Vec4 v1 = *(reinterpret_cast<const device Vec4*>(src + col + 4u));
- *(reinterpret_cast<device Vec4*>(dst + col)) = v0;
- *(reinterpret_cast<device Vec4*>(dst + col + 4u)) = v1;
- col += stride;
- }
-
- // Scalar remainder
- if (col < d_model) {
- const uint limit = min(d_model, col + ELEMS_PER_THREAD);
- for (uint tail = col; tail < limit; ++tail) {
- dst[tail] = src[tail];
- }
- }
-}
-
-// ========== 1D Gather Template ==========
-template <typename T>
-void moe_gather_x_perm_1d_impl(
- device const T* X,
- device const int* bucketed_ids,
- device T* x_perm,
- device const uint* sumk_buf,
- uint d_model,
- uint lid,
- uint threads_x
-) {
- const uint total_rows = sumk_buf[0];
- if (total_rows == 0u || d_model == 0u)
- return;
-
- const uint num_vec4 = d_model / 4u;
-
- // Each thread processes multiple rows sequentially
- for (uint row = lid; row < total_rows; row += threads_x) {
- int token = bucketed_ids[row];
- if (token < 0)
- continue;
-
- const device T* src = X + (ulong)(uint)token * (ulong)d_model;
- device T* dst = x_perm + (ulong)row * (ulong)d_model;
-
- // Vectorized copy: 4 elements at a time
- using Vec4 = typename metal::vec<T, 4>;
- for (uint vec_idx = 0u; vec_idx < num_vec4; vec_idx++) {
- uint col = vec_idx * 4u;
- Vec4 v = *(reinterpret_cast<const device Vec4*>(src + col));
- *(reinterpret_cast<device Vec4*>(dst + col)) = v;
- }
-
- // Scalar remainder
- for (uint col = num_vec4 * 4u; col < d_model; col++) {
- dst[col] = src[col];
- }
- }
-}
-
-// ========== Kernel Entry Points ==========
-
-// 2D Tiled kernels
-#define GATHER_2D_KERNEL(DTYPE, SUFFIX) \
- [[max_total_threads_per_threadgroup(256)]] \
- kernel void moe_gather_x_perm_##SUFFIX##_2d( \
- device const DTYPE* X [[buffer(0)]], \
- device const int* bucketed_ids [[buffer(1)]], \
- device DTYPE* x_perm [[buffer(2)]], \
- device const uint* sumk_buf [[buffer(3)]], \
- constant uint& d_model [[buffer(4)]], \
- uint3 tgid [[threadgroup_position_in_grid]], \
- uint lid [[thread_index_in_threadgroup]] \
- ) { \
- moe_gather_x_perm_2d_impl< \
- DTYPE>(X, bucketed_ids, x_perm, sumk_buf, d_model, tgid, lid); \
- }
-
-// 1D kernels
-#define GATHER_1D_KERNEL(DTYPE, SUFFIX) \
- [[max_total_threads_per_threadgroup(256)]] \
- kernel void moe_gather_x_perm_##SUFFIX( \
- device const DTYPE* X [[buffer(0)]], \
- device const int* bucketed_ids [[buffer(1)]], \
- device DTYPE* x_perm [[buffer(2)]], \
- device const uint* sumk_buf [[buffer(3)]], \
- constant uint& d_model [[buffer(4)]], \
- uint lid [[thread_index_in_threadgroup]], \
- uint3 threads_per_threadgroup [[threads_per_threadgroup]] \
- ) { \
- moe_gather_x_perm_1d_impl<DTYPE>( \
- X, \
- bucketed_ids, \
- x_perm, \
- sumk_buf, \
- d_model, \
- lid, \
- threads_per_threadgroup.x \
- ); \
- }
-
-// Instantiate for all types
-GATHER_2D_KERNEL(bfloat, bf16)
-GATHER_2D_KERNEL(half, f16)
-GATHER_2D_KERNEL(float, f32)
-
-GATHER_1D_KERNEL(bfloat, bf16)
-GATHER_1D_KERNEL(half, f16)
-GATHER_1D_KERNEL(float, f32)
diff --git a/crates/uzu/src/backends/metal/kernel/moe/router_topk.rs b/crates/uzu/src/backends/metal/kernel/moe/router_topk.rs
index 1eb88474b..ffdb5113c 100644
--- a/crates/uzu/src/backends/metal/kernel/moe/router_topk.rs
+++ b/crates/uzu/src/backends/metal/kernel/moe/router_topk.rs
@@ -1,6 +1,7 @@
use crate::backends::metal::{
- KernelDataType, MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- MTLComputePipelineState, MTLContext, MTLError, MTLSize, ProtocolObject, Retained,
+ KernelDataType, MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
+ MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext, MTLError,
+ MTLSize, ProtocolObject, Retained,
metal_extensions::ComputeEncoderSetValue,
};
@@ -58,7 +59,7 @@ impl MoeRouterTopKKernel {
pub fn encode(
&self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
dtype: KernelDataType,
args: MoeRouterTopKArguments,
) -> Result<(), MoeRouterTopKError> {
diff --git a/crates/uzu/src/backends/metal/kernel/moe/scatter.rs b/crates/uzu/src/backends/metal/kernel/moe/scatter.rs
index 471cbba9e..2c7509217 100644
--- a/crates/uzu/src/backends/metal/kernel/moe/scatter.rs
+++ b/crates/uzu/src/backends/metal/kernel/moe/scatter.rs
@@ -1,7 +1,7 @@
use crate::backends::metal::{
- KernelDataType, MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- MTLComputePipelineState, MTLContext, MTLSize, ProtocolObject, Retained,
- metal_extensions::ComputeEncoderSetValue,
+ KernelDataType, MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
+ MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext, MTLSize,
+ ProtocolObject, Retained, metal_extensions::ComputeEncoderSetValue,
};
// ---- Scatter Buckets Kernels ----
@@ -16,11 +16,15 @@ pub struct MoeScatterKernels {
pipeline_bases: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
pipeline_scatter_f16: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
pipeline_scatter_f32: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- pipeline_scatter_bf16: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ pipeline_scatter_bf16:
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
// map variants
- pipeline_scatter_map_f16: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- pipeline_scatter_map_f32: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- pipeline_scatter_map_bf16: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ pipeline_scatter_map_f16:
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ pipeline_scatter_map_f32:
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ pipeline_scatter_map_bf16:
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
}
#[derive(Debug)]
@@ -85,7 +89,7 @@ impl MoeScatterKernels {
pub fn encode_block_bases(
&self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
args: MoeBlockBasesArguments,
) -> Result<(), MoeScatterError> {
let compute_encoder = command_buffer
@@ -117,7 +121,7 @@ impl MoeScatterKernels {
pub fn encode_scatter(
&self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
args: MoeScatterArguments,
dtype: KernelDataType,
) -> Result<(), MoeScatterError> {
@@ -168,7 +172,7 @@ impl MoeScatterKernels {
pub fn encode_scatter_with_map(
&self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
args: MoeScatterWithMapArguments,
dtype: KernelDataType,
) -> Result<(), MoeScatterError> {
diff --git a/crates/uzu/src/backends/metal/kernel/moe/tiles.rs b/crates/uzu/src/backends/metal/kernel/moe/tiles.rs
deleted file mode 100644
index b3bb27195..000000000
--- a/crates/uzu/src/backends/metal/kernel/moe/tiles.rs
+++ /dev/null
@@ -1,333 +0,0 @@
-use crate::backends::metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- MTLComputePipelineState, MTLContext, MTLError, MTLSize, ProtocolObject, Retained,
- metal_extensions::ComputeEncoderSetValue,
-};
-
-// ---- Tile Kernels ----
-
-#[derive(Debug, thiserror::Error)]
-pub enum MoeTileError {
- #[error("Metal error: {0}")]
- MetalError(#[from] MTLError),
-}
-
-/// Arguments for tile counts encoder
-pub struct MoeTileCountsArguments<'a> {
- pub offsets_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
- pub tile_counts_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [E]
- pub e: usize,
-}
-
-/// Arguments for tile scan encoder
-pub struct MoeTileScanArguments<'a> {
- pub tile_counts_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [E]
- pub tile_offsets_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
- pub total_tiles_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [>=2]
- pub e: usize,
-}
-
-#[derive(Debug)]
-pub struct MoeTileMapBuildArguments<'a> {
- pub expert_offsets: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
- pub tile_offsets: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
- pub tile_counts: &'a ProtocolObject<dyn MTLBuffer>, // [E]
- pub tile_map: &'a ProtocolObject<dyn MTLBuffer>, // [total_tiles * 3]
- pub e: usize,
-}
-
-#[derive(Debug)]
-pub struct MoeTileDispatchArguments<'a> {
- pub total_tiles: &'a ProtocolObject<dyn MTLBuffer>, // [>=1]
- pub dispatch_args: &'a ProtocolObject<dyn MTLBuffer>, // [3]
- pub num_tiles_x: u32, // x dimension for indirect dispatch
-}
-
-pub struct MoeTileMapKernel {
- counts_pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- scan_pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- build_pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- dispatch_pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-#[derive(Debug)]
-pub struct MoePassATileCountsArguments<'a> {
- pub expert_offsets: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
- pub tile_counts: &'a ProtocolObject<dyn MTLBuffer>, // [E]
- pub e: usize,
- pub h_blocks: u32,
-}
-
-#[derive(Debug)]
-pub struct MoePassATileScanArguments<'a> {
- pub tile_counts: &'a ProtocolObject<dyn MTLBuffer>, // [E]
- pub tile_offsets: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
- pub total_tiles: &'a ProtocolObject<dyn MTLBuffer>, // [>=1]
- pub e: usize,
-}
-
-#[derive(Debug)]
-pub struct MoePassARowMapArguments<'a> {
- pub expert_offsets: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
- pub row_expert_map: &'a ProtocolObject<dyn MTLBuffer>, // [total_rows]
- pub total_rows: usize,
- pub e: usize,
-}
-
-#[derive(Debug)]
-pub struct MoePassATileBuildArguments<'a> {
- pub expert_offsets: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
- pub tile_offsets: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
- pub row_expert_map: &'a ProtocolObject<dyn MTLBuffer>, // [total_rows]
- pub tile_map: &'a ProtocolObject<dyn MTLBuffer>, // [total_tiles * 3]
- pub total_rows: usize,
- pub h_blocks: u32,
-}
-
-#[derive(Debug)]
-pub struct MoePassATileDispatchArguments<'a> {
- pub total_tiles: &'a ProtocolObject<dyn MTLBuffer>, // [>=1]
- pub dispatch_args: &'a ProtocolObject<dyn MTLBuffer>, // [3]
- pub num_tiles_y: u32,
-}
-
-pub struct MoePassATileKernel {
- counts_pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- scan_pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- row_map_pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- build_pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- dispatch_pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-impl MoePassATileKernel {
- pub fn new(ctx: &MTLContext) -> Result<Self, MoeTileError> {
- Ok(Self {
- counts_pipeline: ctx
- .compute_pipeline_state("moe_pass_a_tile_counts", None)?,
- scan_pipeline: ctx
- .compute_pipeline_state("moe_pass_a_tile_scan", None)?,
- row_map_pipeline: ctx
- .compute_pipeline_state("moe_pass_a_build_row_map", None)?,
- build_pipeline: ctx
- .compute_pipeline_state("moe_pass_a_build_tile_map", None)?,
- dispatch_pipeline: ctx.compute_pipeline_state(
- "moe_pass_a_write_dispatch_args",
- None,
- )?,
- })
- }
-
- pub fn encode_counts(
- &self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- args: &MoePassATileCountsArguments,
- ) -> Result<(), MoeTileError> {
- let encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- encoder.set_compute_pipeline_state(&self.counts_pipeline);
- encoder.set_buffer(Some(args.expert_offsets), 0, 0);
- encoder.set_buffer(Some(args.tile_counts), 0, 1);
- let e_u32 = args.e as u32;
- encoder.set_value(&e_u32, 2);
- encoder.set_value(&args.h_blocks, 3);
- encoder.dispatch_threadgroups(
- MTLSize::new((args.e + 255) / 256, 1, 1),
- MTLSize::new(256, 1, 1),
- );
- encoder.end_encoding();
- Ok(())
- }
-
- pub fn encode_scan(
- &self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- args: &MoePassATileScanArguments,
- ) -> Result<(), MoeTileError> {
- let encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- encoder.set_compute_pipeline_state(&self.scan_pipeline);
- encoder.set_buffer(Some(args.tile_counts), 0, 0);
- encoder.set_buffer(Some(args.tile_offsets), 0, 1);
- encoder.set_buffer(Some(args.total_tiles), 0, 2);
- let e_u32 = args.e as u32;
- encoder.set_value(&e_u32, 3);
- encoder.set_threadgroup_memory_length(1024 * size_of::<u32>() , 0);
- encoder.dispatch_threadgroups(
- MTLSize::new(1, 1, 1),
- MTLSize::new(1024, 1, 1),
- );
- encoder.end_encoding();
- Ok(())
- }
-
- pub fn encode_row_map(
- &self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- args: &MoePassARowMapArguments,
- ) -> Result<(), MoeTileError> {
- let encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- encoder.set_compute_pipeline_state(&self.row_map_pipeline);
- encoder.set_buffer(Some(args.expert_offsets), 0, 0);
- encoder.set_buffer(Some(args.row_expert_map), 0, 1);
- let total_rows_u32 = args.total_rows as u32;
- let e_u32 = args.e as u32;
- encoder.set_value(&total_rows_u32, 2);
- encoder.set_value(&e_u32, 3);
- encoder.dispatch_threadgroups(
- MTLSize::new((args.total_rows + 255) / 256, 1, 1),
- MTLSize::new(256, 1, 1),
- );
- encoder.end_encoding();
- Ok(())
- }
-
- pub fn encode_build_map(
- &self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- args: &MoePassATileBuildArguments,
- ) -> Result<(), MoeTileError> {
- let encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- encoder.set_compute_pipeline_state(&self.build_pipeline);
- encoder.set_buffer(Some(args.expert_offsets), 0, 0);
- encoder.set_buffer(Some(args.tile_offsets), 0, 1);
- encoder.set_buffer(Some(args.row_expert_map), 0, 2);
- encoder.set_buffer(Some(args.tile_map), 0, 3);
- let total_rows_u32 = args.total_rows as u32;
- encoder.set_value(&total_rows_u32, 4);
- encoder.set_value(&args.h_blocks, 5);
- let total_tiles_linear =
- (total_rows_u32 as u64).saturating_mul(args.h_blocks as u64);
- encoder.dispatch_threadgroups(
- MTLSize::new(((total_tiles_linear + 255) / 256) as usize, 1, 1),
- MTLSize::new(256, 1, 1),
- );
- encoder.end_encoding();
- Ok(())
- }
-
- pub fn encode_dispatch_args(
- &self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- args: &MoePassATileDispatchArguments,
- ) -> Result<(), MoeTileError> {
- let encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- encoder.set_compute_pipeline_state(&self.dispatch_pipeline);
- encoder.set_buffer(Some(args.total_tiles), 0, 0);
- encoder.set_buffer(Some(args.dispatch_args), 0, 1);
- encoder.set_value(&args.num_tiles_y, 2);
- encoder.dispatch_threadgroups(
- MTLSize::new(1, 1, 1),
- MTLSize::new(1, 1, 1),
- );
- encoder.end_encoding();
- Ok(())
- }
-}
-
-impl MoeTileMapKernel {
- pub fn new(ctx: &MTLContext) -> Result<Self, MoeTileError> {
- Ok(Self {
- counts_pipeline: ctx
- .compute_pipeline_state("moe_tile_counts", None)?,
- scan_pipeline: ctx.compute_pipeline_state("moe_tile_scan", None)?,
- build_pipeline: ctx
- .compute_pipeline_state("moe_build_tile_map", None)?,
- dispatch_pipeline: ctx
- .compute_pipeline_state("moe_write_dispatch_args", None)?,
- })
- }
-
- pub fn encode_counts(
- &self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- args: &MoeTileCountsArguments,
- ) -> Result<(), MoeTileError> {
- let encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- encoder.set_compute_pipeline_state(&self.counts_pipeline);
- encoder.set_buffer(Some(args.offsets_buffer), 0, 0);
- encoder.set_buffer(Some(args.tile_counts_buffer), 0, 1);
- let e_u32 = args.e as u32;
- encoder.set_value(&e_u32, 2);
- encoder.dispatch_threadgroups(
- MTLSize::new((args.e + 255) / 256, 1, 1),
- MTLSize::new(256, 1, 1),
- );
- encoder.end_encoding();
- Ok(())
- }
-
- pub fn encode_scan(
- &self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- args: &MoeTileScanArguments,
- ) -> Result<(), MoeTileError> {
- let encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- encoder.set_compute_pipeline_state(&self.scan_pipeline);
- encoder.set_buffer(Some(args.tile_counts_buffer), 0, 0);
- encoder.set_buffer(Some(args.tile_offsets_buffer), 0, 1);
- encoder.set_buffer(Some(args.total_tiles_buffer), 0, 2);
- let e_u32 = args.e as u32;
- encoder.set_value(&e_u32, 3);
- encoder.dispatch_threadgroups(
- MTLSize::new(1, 1, 1),
- MTLSize::new(256, 1, 1),
- );
- encoder.end_encoding();
- Ok(())
- }
-
- pub fn encode_build_map(
- &self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- args: &MoeTileMapBuildArguments,
- ) -> Result<(), MoeTileError> {
- let encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- encoder.set_compute_pipeline_state(&self.build_pipeline);
- encoder.set_buffer(Some(args.expert_offsets), 0, 0);
- encoder.set_buffer(Some(args.tile_offsets), 0, 1);
- encoder.set_buffer(Some(args.tile_counts), 0, 2);
- encoder.set_buffer(Some(args.tile_map), 0, 3);
- let e_u32 = args.e as u32;
- encoder.set_value(&e_u32, 4);
- encoder.dispatch_threadgroups(
- MTLSize::new((args.e + 255) / 256, 1, 1),
- MTLSize::new(256, 1, 1),
- );
- encoder.end_encoding();
- Ok(())
- }
-
- pub fn encode_dispatch_args(
- &self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- args: &MoeTileDispatchArguments,
- ) -> Result<(), MoeTileError> {
- let encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- encoder.set_compute_pipeline_state(&self.dispatch_pipeline);
- encoder.set_buffer(Some(args.total_tiles), 0, 0);
- encoder.set_buffer(Some(args.dispatch_args), 0, 1);
- encoder.set_value(&args.num_tiles_x, 2);
- encoder.dispatch_threadgroups(
- MTLSize::new(1, 1, 1),
- MTLSize::new(1, 1, 1),
- );
- encoder.end_encoding();
- Ok(())
- }
-}
diff --git a/crates/uzu/src/backends/metal/kernel/moe/tiles.metal b/crates/uzu/src/backends/metal/kernel/moe/tiles_map.metal
similarity index 54%
rename from crates/uzu/src/backends/metal/kernel/moe/tiles.metal
rename to crates/uzu/src/backends/metal/kernel/moe/tiles_map.metal
index fccd11b52..5474a6e52 100644
--- a/crates/uzu/src/backends/metal/kernel/moe/tiles.metal
+++ b/crates/uzu/src/backends/metal/kernel/moe/tiles_map.metal
@@ -1,19 +1,16 @@
#include <metal_stdlib>
#include <metal_atomic>
-using namespace metal;
+#include "../definitions.metal"
#define BM 16
// Compute per-expert tile counts: tiles_e = ceil((seg_len)/BM)
-[[max_total_threads_per_threadgroup(256)]]
-kernel void moe_tile_counts(
- device const uint* offsets [[buffer(0)]], // [E+1]
- device uint* tile_counts [[buffer(1)]], // [E]
- constant uint& E [[buffer(2)]],
- uint gid [[thread_position_in_grid]]
+KERNEL(MoeTileCounts)(
+ device const uint* offsets, // [e+1]
+ device uint* tile_counts, // [e]
+ constant uint& e,
+ const uint gid AXIS(e, 256)
) {
- if (gid >= E)
- return;
const uint start = offsets[gid];
const uint end = offsets[gid + 1];
const uint seg_len = (end > start) ? (end - start) : 0u;
@@ -23,32 +20,31 @@ kernel void moe_tile_counts(
// Single-threadgroup exclusive scan over tile_counts -> tile_row_offsets, also
// writes total_tiles
-[[max_total_threads_per_threadgroup(256)]]
-kernel void moe_tile_scan(
- device const uint* tile_counts [[buffer(0)]], // [E]
- device uint* tile_row_offsets [[buffer(1)]], // [E+1]
- device uint* total_tiles_buf [[buffer(2)]], // [>=2]
- constant uint& E [[buffer(3)]],
- uint tid [[thread_index_in_threadgroup]]
+KERNEL(MoeTileScan)(
+ device const uint* tile_counts, // [e]
+ device uint* tile_row_offsets, // [e+1]
+ device uint* total_tiles_buf, // [>=2]
+ constant uint& e,
+ threadgroup uint scratch[1024], // supports up to E<=1023
+ uint tid THREADS(256)
) {
- threadgroup uint scratch[1024]; // supports up to E<=1023
- if (tid < E) {
+ if (tid < e) {
scratch[tid] = tile_counts[tid];
}
threadgroup_barrier(mem_flags::mem_threadgroup);
// Inclusive scan in-place over scratch
// Simple O(log E) scan; E is small
- for (uint offset = 1u; offset < E; offset <<= 1u) {
+ for (uint offset = 1u; offset < e; offset <<= 1u) {
uint val = 0u;
- if (tid < E) {
+ if (tid < e) {
val = scratch[tid];
if (tid >= offset) {
val += scratch[tid - offset];
}
}
threadgroup_barrier(mem_flags::mem_threadgroup);
- if (tid < E) {
+ if (tid < e) {
scratch[tid] = val;
}
threadgroup_barrier(mem_flags::mem_threadgroup);
@@ -57,14 +53,14 @@ kernel void moe_tile_scan(
if (tid == 0u) {
tile_row_offsets[0] = 0u;
}
- if (tid < E) {
+ if (tid < e) {
// exclusive: shift right by one
const uint prefix = (tid == 0u) ? 0u : scratch[tid - 1u];
tile_row_offsets[tid] = prefix;
}
if (tid == 0u) {
- tile_row_offsets[E] = (E == 0u) ? 0u : scratch[E - 1u];
- total_tiles_buf[0] = tile_row_offsets[E];
+ tile_row_offsets[e] = (e == 0u) ? 0u : scratch[e - 1u];
+ total_tiles_buf[0] = tile_row_offsets[e];
total_tiles_buf[1] = 0u;
total_tiles_buf[2] = 0u;
total_tiles_buf[3] = 0u;
@@ -77,17 +73,14 @@ kernel void moe_tile_scan(
// Build flattened tile_map of length total_tiles; each entry is 3 uints:
// [3*i+0]=expert_idx, [3*i+1]=seg_start, [3*i+2]=tile_m0
-[[max_total_threads_per_threadgroup(256)]]
-kernel void moe_build_tile_map(
- device const uint* offsets [[buffer(0)]], // [E+1]
- device const uint* tile_row_offsets [[buffer(1)]], // [E+1]
- device const uint* tile_counts [[buffer(2)]], // [E]
- device uint* tile_map [[buffer(3)]], // [3*total_tiles]
- constant uint& E [[buffer(4)]],
- uint gid [[thread_position_in_grid]]
+KERNEL(MoeBuildTileMap)(
+ device const uint* offsets, // [e+1]
+ device const uint* tile_row_offsets, // [e+1]
+ device const uint* tile_counts, // [e]
+ device uint* tile_map, // [3*total_tiles]
+ constant uint& e,
+ uint gid AXIS(e, 256)
) {
- if (gid >= E)
- return;
const uint start = offsets[gid];
const uint tiles = tile_counts[gid];
const uint base = tile_row_offsets[gid];
@@ -101,18 +94,14 @@ kernel void moe_build_tile_map(
// Write MTLDispatchThreadgroupsIndirectArguments {x, y, z} where:
// x = num_tiles_n (computed on CPU and passed in), y = total_tiles, z = 1
-[[max_total_threads_per_threadgroup(1)]]
-kernel void moe_write_dispatch_args(
- device const uint* total_tiles_buf
- [[buffer(0)]], // [>=1], total_tiles_buf[0] = total_rows
- device uint* dispatch_args [[buffer(1)]], // [3] u32
- constant uint& num_tiles_n [[buffer(2)]], // x dimension
- uint gid [[thread_position_in_grid]]
+KERNEL(MoeWriteDispatchArgs)(
+ device const uint* total_tiles_buf, // [>=1], total_tiles_buf[0] = total_rows
+ device uint* dispatch_args, // [3] u32
+ constant uint& num_tiles_n, // x dimension
+ uint gid AXIS(1, 1)
) {
- if (gid > 0u)
- return;
const uint total_rows = total_tiles_buf[0];
dispatch_args[0] = num_tiles_n; // x
dispatch_args[1] = total_rows; // y
dispatch_args[2] = 1u; // z
-}
+}
\ No newline at end of file
diff --git a/crates/uzu/src/backends/metal/kernel/moe/tiles_map.rs b/crates/uzu/src/backends/metal/kernel/moe/tiles_map.rs
new file mode 100644
index 000000000..22bbd7b18
--- /dev/null
+++ b/crates/uzu/src/backends/metal/kernel/moe/tiles_map.rs
@@ -0,0 +1,134 @@
+use crate::backends::{
+ common::kernel::{
+ MoeBuildTileMapKernel, MoeTileCountsKernel, MoeTileScanKernel,
+ MoeWriteDispatchArgsKernel,
+ },
+ metal::{
+ MTLContext, MTLError,
+ kernel::dsl::{
+ MoeBuildTileMapMetalKernel, MoeTileCountsMetalKernel,
+ MoeTileScanMetalKernel, MoeWriteDispatchArgsMetalKernel,
+ },
+ },
+};
+use metal::{MTLBuffer, MTLCommandBuffer, MTLCommandEncoder};
+use objc2::__framework_prelude::Retained;
+use objc2::{__framework_prelude::ProtocolObject, Message};
+
+pub struct MoeTileMapKernels {
+ counts: MoeTileCountsMetalKernel,
+ scan: MoeTileScanMetalKernel,
+ build: MoeBuildTileMapMetalKernel,
+ dispatch: MoeWriteDispatchArgsMetalKernel,
+}
+
+impl MoeTileMapKernels {
+ pub fn new(ctx: &MTLContext) -> Result<Self, MTLError> {
+ Ok(Self {
+ counts: MoeTileCountsMetalKernel::new(ctx)?,
+ scan: MoeTileScanMetalKernel::new(ctx)?,
+ build: MoeBuildTileMapMetalKernel::new(ctx)?,
+ dispatch: MoeWriteDispatchArgsMetalKernel::new(ctx)?,
+ })
+ }
+
+ pub fn encode_counts(
+ &self,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
+ args: &MoeTileCountsArguments,
+ ) {
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+ self.counts.encode(
+ &args.offsets_buffer.retain(),
+ &args.tile_counts_buffer.retain(),
+ args.e as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
+ }
+
+ pub fn encode_scan(
+ &self,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
+ args: &MoeTileScanArguments,
+ ) {
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+ self.scan.encode(
+ &args.tile_counts_buffer.retain(),
+ &args.tile_offsets_buffer.retain(),
+ &args.total_tiles_buffer.retain(),
+ args.e as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
+ }
+
+ pub fn encode_build_map(
+ &self,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
+ args: &MoeTileMapBuildArguments,
+ ) {
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+ self.build.encode(
+ &args.expert_offsets.retain(),
+ &args.tile_offsets.retain(),
+ &args.tile_counts.retain(),
+ &args.tile_map.retain(),
+ args.e as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
+ }
+
+ pub fn encode_dispatch_args(
+ &self,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
+ args: &MoeTileDispatchArguments,
+ ) {
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+ self.dispatch.encode(
+ &args.total_tiles.retain(),
+ &args.dispatch_args.retain(),
+ args.num_tiles_x,
+ &encoder,
+ );
+ encoder.end_encoding();
+ }
+}
+
+pub struct MoeTileCountsArguments<'a> {
+ pub offsets_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
+ pub tile_counts_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [E]
+ pub e: usize,
+}
+
+pub struct MoeTileScanArguments<'a> {
+ pub tile_counts_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [E]
+ pub tile_offsets_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
+ pub total_tiles_buffer: &'a ProtocolObject<dyn MTLBuffer>, // [>=2]
+ pub e: usize,
+}
+
+#[derive(Debug)]
+pub struct MoeTileMapBuildArguments<'a> {
+ pub expert_offsets: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
+ pub tile_offsets: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
+ pub tile_counts: &'a ProtocolObject<dyn MTLBuffer>, // [E]
+ pub tile_map: &'a ProtocolObject<dyn MTLBuffer>, // [total_tiles * 3]
+ pub e: usize,
+}
+
+#[derive(Debug)]
+pub struct MoeTileDispatchArguments<'a> {
+ pub total_tiles: &'a ProtocolObject<dyn MTLBuffer>, // [>=1]
+ pub dispatch_args: &'a ProtocolObject<dyn MTLBuffer>, // [3]
+ pub num_tiles_x: u32, // x dimension for indirect dispatch
+}
diff --git a/crates/uzu/src/backends/metal/kernel/moe/tiles_pass_a.metal b/crates/uzu/src/backends/metal/kernel/moe/tiles_pass_a.metal
new file mode 100644
index 000000000..ec1522135
--- /dev/null
+++ b/crates/uzu/src/backends/metal/kernel/moe/tiles_pass_a.metal
@@ -0,0 +1,129 @@
+#include <metal_stdlib>
+#include "../definitions.metal"
+
+// === Helper kernels for indirect dispatch of Pass A ===
+
+// Count tiles per expert: tiles = (num_rows > 0) ? num_rows * h_blocks : 0
+KERNEL(MoePassATileCounts)(
+ device const uint* expert_offsets, // [e+1]
+ device uint* tile_counts, // [e]
+ constant uint& e,
+ constant uint& h_blocks,
+ const uint tid AXIS(e, 256)
+) {
+ const uint start = expert_offsets[tid];
+ const uint end = expert_offsets[tid + 1];
+ const uint num_rows = end - start;
+ tile_counts[tid] = (num_rows > 0) ? (num_rows * h_blocks) : 0;
+}
+
+// Exclusive scan of tile_counts to get tile_offsets and total_tiles
+KERNEL(MoePassATileScan)(
+ device const uint* tile_counts, // [e]
+ device uint* tile_offsets, // [e+1]
+ device uint* total_tiles, // [1]
+ constant uint& e,
+ threadgroup uint scratch[1024],
+ uint gid GROUPS(1),
+ uint lid THREADS(1024)
+) {
+ // Simple single-threadgroup scan (works for E <= 1024)
+ const uint idx = lid;
+
+ // Load into threadgroup memory
+ if (idx < e) {
+ scratch[idx] = tile_counts[idx];
+ } else {
+ scratch[idx] = 0;
+ }
+ threadgroup_barrier(mem_flags::mem_threadgroup);
+
+ // Kogge-Stone scan
+ uint val = scratch[idx];
+ for (uint offset = 1; offset < 1024; offset *= 2) {
+ uint temp = 0;
+ if (idx >= offset && idx < e) {
+ temp = scratch[idx - offset];
+ }
+ threadgroup_barrier(mem_flags::mem_threadgroup);
+ if (idx >= offset && idx < e) {
+ val += temp;
+ scratch[idx] = val;
+ }
+ threadgroup_barrier(mem_flags::mem_threadgroup);
+ }
+
+ // Write exclusive scan (shift right by 1)
+ if (idx == 0) {
+ tile_offsets[0] = 0;
+ }
+ if (idx < e) {
+ tile_offsets[idx + 1] = scratch[idx];
+ if (idx == e - 1) {
+ total_tiles[0] = scratch[idx];
+ }
+ }
+}
+
+// Build row→expert map: one thread per routed row
+KERNEL(MoePassABuildRowMap)(
+ device const uint* expert_offsets, // [e+1]
+ device uint* row_expert_map, // [total_rows]
+ constant uint& total_rows,
+ constant uint& e,
+ uint tid AXIS(total_rows, 256)
+) {
+ uint left = 0u;
+ uint right = e;
+ const uint row = tid;
+
+ while (left + 1u < right) {
+ const uint mid = (left + right) >> 1;
+ if (row < expert_offsets[mid]) {
+ right = mid;
+ } else {
+ left = mid;
+ }
+ }
+
+ row_expert_map[row] = left;
+}
+
+// Build tile map entries from row→expert map
+KERNEL(MoePassABuildTileMap)(
+ device const uint* expert_offsets, // [E+1]
+ device const uint* tile_offsets, // [E+1]
+ device const uint* row_expert_map, // [total_rows]
+ device uint* tile_map, // [total_tiles * 3]
+ constant uint& total_rows,
+ constant uint& h_blocks,
+ uint tid AXIS(total_rows.saturating_mul(h_blocks), 256)
+) {
+ const uint row_idx = tid / h_blocks;
+ const uint h_block = tid % h_blocks;
+ if (row_idx >= total_rows) {
+ return;
+ }
+
+ const uint expert_idx = row_expert_map[row_idx];
+ const uint row_start = expert_offsets[expert_idx];
+ const uint row_in_expert = row_idx - row_start;
+ const uint tile_base =
+ tile_offsets[expert_idx] + row_in_expert * h_blocks + h_block;
+
+ tile_map[tile_base * 3 + 0] = h_block;
+ tile_map[tile_base * 3 + 1] = expert_idx;
+ tile_map[tile_base * 3 + 2] = row_in_expert;
+}
+
+// Write dispatch args for indirect dispatch (reusable from tiled version)
+KERNEL(MoePassAWriteDispatchArgs)(
+ device const uint* total_tiles, // [1]
+ device uint* dispatch_args, // [3] - MTLDispatchThreadgroupsIndirectArguments
+ constant uint& num_tiles_y, // usually 1 for Pass A
+ uint tid AXIS(1, 1)
+) {
+ dispatch_args[0] = total_tiles[0]; // x dimension = total tiles
+ dispatch_args[1] = num_tiles_y; // y dimension
+ dispatch_args[2] = 1; // z dimension
+}
diff --git a/crates/uzu/src/backends/metal/kernel/moe/tiles_pass_a.rs b/crates/uzu/src/backends/metal/kernel/moe/tiles_pass_a.rs
new file mode 100644
index 000000000..8023d37d4
--- /dev/null
+++ b/crates/uzu/src/backends/metal/kernel/moe/tiles_pass_a.rs
@@ -0,0 +1,170 @@
+use objc2::Message;
+
+use crate::backends::{
+ common::kernel::{
+ MoePassABuildRowMapKernel, MoePassABuildTileMapKernel,
+ MoePassATileCountsKernel, MoePassATileScanKernel,
+ MoePassAWriteDispatchArgsKernel,
+ },
+ metal::{
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLContext, MTLError,
+ ProtocolObject,
+ kernel::dsl::{
+ MoePassABuildRowMapMetalKernel, MoePassABuildTileMapMetalKernel,
+ MoePassATileCountsMetalKernel, MoePassATileScanMetalKernel,
+ MoePassAWriteDispatchArgsMetalKernel,
+ },
+ },
+};
+
+#[derive(Debug)]
+pub struct MoePassATileCountsArguments<'a> {
+ pub expert_offsets: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
+ pub tile_counts: &'a ProtocolObject<dyn MTLBuffer>, // [E]
+ pub e: usize,
+ pub h_blocks: u32,
+}
+
+#[derive(Debug)]
+pub struct MoePassATileScanArguments<'a> {
+ pub tile_counts: &'a ProtocolObject<dyn MTLBuffer>, // [E]
+ pub tile_offsets: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
+ pub total_tiles: &'a ProtocolObject<dyn MTLBuffer>, // [>=1]
+ pub e: usize,
+}
+
+#[derive(Debug)]
+pub struct MoePassARowMapArguments<'a> {
+ pub expert_offsets: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
+ pub row_expert_map: &'a ProtocolObject<dyn MTLBuffer>, // [total_rows]
+ pub total_rows: usize,
+ pub e: usize,
+}
+
+#[derive(Debug)]
+pub struct MoePassATileBuildArguments<'a> {
+ pub expert_offsets: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
+ pub tile_offsets: &'a ProtocolObject<dyn MTLBuffer>, // [E+1]
+ pub row_expert_map: &'a ProtocolObject<dyn MTLBuffer>, // [total_rows]
+ pub tile_map: &'a ProtocolObject<dyn MTLBuffer>, // [total_tiles * 3]
+ pub total_rows: usize,
+ pub h_blocks: u32,
+}
+
+#[derive(Debug)]
+pub struct MoePassATileDispatchArguments<'a> {
+ pub total_tiles: &'a ProtocolObject<dyn MTLBuffer>, // [>=1]
+ pub dispatch_args: &'a ProtocolObject<dyn MTLBuffer>, // [3]
+ pub num_tiles_y: u32,
+}
+
+pub struct MoePassATileKernels {
+ counts: MoePassATileCountsMetalKernel,
+ scan: MoePassATileScanMetalKernel,
+ row_map: MoePassABuildRowMapMetalKernel,
+ build_map: MoePassABuildTileMapMetalKernel,
+ dispatch: MoePassAWriteDispatchArgsMetalKernel,
+}
+
+impl MoePassATileKernels {
+ pub fn new(ctx: &MTLContext) -> Result<Self, MTLError> {
+ Ok(Self {
+ counts: MoePassATileCountsMetalKernel::new(ctx)?,
+ scan: MoePassATileScanMetalKernel::new(ctx)?,
+ row_map: MoePassABuildRowMapMetalKernel::new(ctx)?,
+ build_map: MoePassABuildTileMapMetalKernel::new(ctx)?,
+ dispatch: MoePassAWriteDispatchArgsMetalKernel::new(ctx)?,
+ })
+ }
+
+ pub fn encode_counts(
+ &self,
+ command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ args: &MoePassATileCountsArguments,
+ ) {
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+ self.counts.encode(
+ &args.expert_offsets.retain(),
+ &args.tile_counts.retain(),
+ args.e as u32,
+ args.h_blocks,
+ &encoder,
+ );
+ encoder.end_encoding();
+ }
+
+ pub fn encode_scan(
+ &self,
+ command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ args: &MoePassATileScanArguments,
+ ) {
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+ self.scan.encode(
+ &args.tile_counts.retain(),
+ &args.tile_offsets.retain(),
+ &args.total_tiles.retain(),
+ args.e as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
+ }
+
+ pub fn encode_row_map(
+ &self,
+ command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ args: &MoePassARowMapArguments,
+ ) {
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+ self.row_map.encode(
+ &args.expert_offsets.retain(),
+ &args.row_expert_map.retain(),
+ args.total_rows as u32,
+ args.e as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
+ }
+
+ pub fn encode_build_map(
+ &self,
+ command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ args: &MoePassATileBuildArguments,
+ ) {
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+ self.build_map.encode(
+ &args.expert_offsets.retain(),
+ &args.tile_offsets.retain(),
+ &args.row_expert_map.retain(),
+ &args.tile_map.retain(),
+ args.total_rows as u32,
+ args.h_blocks,
+ &encoder,
+ );
+ encoder.end_encoding();
+ }
+
+ pub fn encode_dispatch_args(
+ &self,
+ command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ args: &MoePassATileDispatchArguments,
+ ) {
+ let encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute command encoder");
+ self.dispatch.encode(
+ &args.total_tiles.retain(),
+ &args.dispatch_args.retain(),
+ args.num_tiles_y,
+ &encoder,
+ );
+ encoder.end_encoding();
+ }
+}
diff --git a/crates/uzu/src/backends/metal/kernel/pooling/mod.rs b/crates/uzu/src/backends/metal/kernel/pooling/mod.rs
deleted file mode 100644
index f0252a5b5..000000000
--- a/crates/uzu/src/backends/metal/kernel/pooling/mod.rs
+++ /dev/null
@@ -1,92 +0,0 @@
-use crate::{
- DataType,
- backends::metal::{
- ComputeEncoderSetValue, MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState,
- MTLContext, MTLError, MTLSize, ProtocolObject, Retained,
- },
-};
-
-pub struct PoolingKernel {
- cls_pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- mean_pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-impl PoolingKernel {
- fn kernel_name_for_type(
- data_type: DataType,
- pooling_type: &str,
- ) -> Result<String, MTLError> {
- let suffix = match data_type {
- DataType::F16 => "f16",
- DataType::F32 => "f32",
- DataType::BF16 => "bf16",
- other => {
- return Err(MTLError::Generic(format!(
- "Unsupported dtype for pooling: {:?}",
- other
- )));
- },
- };
- Ok(format!("pool_{}_{}", pooling_type, suffix))
- }
-
- pub fn new(
- context: &MTLContext,
- data_type: DataType,
- ) -> Result<Self, MTLError> {
- let cls_fn_name = Self::kernel_name_for_type(data_type, "cls")?;
- let mean_fn_name = Self::kernel_name_for_type(data_type, "mean")?;
-
- let cls_pipeline =
- context.compute_pipeline_state(&cls_fn_name, None)?;
- let mean_pipeline =
- context.compute_pipeline_state(&mean_fn_name, None)?;
-
- Ok(Self {
- cls_pipeline,
- mean_pipeline,
- })
- }
-
- pub fn encode_cls(
- &self,
- encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- input_buffer: &ProtocolObject<dyn MTLBuffer>,
- output_buffer: &ProtocolObject<dyn MTLBuffer>,
- batch_size: i32,
- seq_len: i32,
- hidden_dim: i32,
- ) -> Result<(), MTLError> {
- encoder.set_compute_pipeline_state(&self.cls_pipeline);
- encoder.set_buffer(Some(input_buffer), 0, 0);
- encoder.set_buffer(Some(output_buffer), 0, 1);
- encoder.set_value(&seq_len, 2);
- encoder.set_value(&hidden_dim, 3);
-
- let threads_per_tg = MTLSize::new(16, 16, 1);
- let grid = MTLSize::new(hidden_dim as usize, batch_size as usize, 1);
- encoder.dispatch_threads(grid, threads_per_tg);
- Ok(())
- }
-
- pub fn encode_mean(
- &self,
- encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- input_buffer: &ProtocolObject<dyn MTLBuffer>,
- output_buffer: &ProtocolObject<dyn MTLBuffer>,
- batch_size: i32,
- seq_len: i32,
- hidden_dim: i32,
- ) -> Result<(), MTLError> {
- encoder.set_compute_pipeline_state(&self.mean_pipeline);
- encoder.set_buffer(Some(input_buffer), 0, 0);
- encoder.set_buffer(Some(output_buffer), 0, 1);
- encoder.set_value(&seq_len, 2);
- encoder.set_value(&hidden_dim, 3);
-
- let threads_per_tg = MTLSize::new(16, 16, 1);
- let grid = MTLSize::new(hidden_dim as usize, batch_size as usize, 1);
- encoder.dispatch_threads(grid, threads_per_tg);
- Ok(())
- }
-}
diff --git a/crates/uzu/src/backends/metal/kernel/pooling/pooling.metal b/crates/uzu/src/backends/metal/kernel/pooling/pooling.metal
index 2ec92926c..134c7915a 100644
--- a/crates/uzu/src/backends/metal/kernel/pooling/pooling.metal
+++ b/crates/uzu/src/backends/metal/kernel/pooling/pooling.metal
@@ -1,24 +1,19 @@
#include <metal_stdlib>
#include "../definitions.metal"
-using namespace metal;
// CLS pooling: Extract first token [batch, seq_len, hidden_dim] → [batch,
// hidden_dim]
template <typename T>
-[[kernel, max_total_threads_per_threadgroup(256)]]
-void pool_cls(
- const device T* input [[buffer(0)]],
- device T* output [[buffer(1)]],
- constant int& seq_len [[buffer(2)]],
- constant int& hidden_dim [[buffer(3)]],
- uint2 tid [[thread_position_in_grid]]
-) // x: hidden_dim, y: batch
-{
- int batch_idx = tid.y;
- int dim_idx = tid.x;
- if (dim_idx >= hidden_dim)
- return;
-
+VARIANTS(T, float, half, bfloat)
+KERNEL(PoolingCls) (
+ const device T* input,
+ device T* output,
+ constant uint& seq_len,
+ constant uint& hidden_dim,
+ constant uint& batch_size,
+ uint dim_idx AXIS(hidden_dim, 16),
+ uint batch_idx AXIS(batch_size, 16)
+) {
output[batch_idx * hidden_dim + dim_idx] =
input[batch_idx * seq_len * hidden_dim + dim_idx];
}
@@ -26,73 +21,21 @@ void pool_cls(
// Mean pooling: Average across sequence [batch, seq_len, hidden_dim] → [batch,
// hidden_dim]
template <typename T>
-[[kernel, max_total_threads_per_threadgroup(256)]]
-void pool_mean(
- const device T* input [[buffer(0)]],
- device T* output [[buffer(1)]],
- constant int& seq_len [[buffer(2)]],
- constant int& hidden_dim [[buffer(3)]],
- uint2 tid [[thread_position_in_grid]]
+VARIANTS(T, float, half, bfloat)
+KERNEL(PoolingMean) (
+ const device T* input,
+ device T* output,
+ constant uint& seq_len,
+ constant uint& hidden_dim,
+ constant uint& batch_size,
+ uint dim_idx AXIS(hidden_dim, 16),
+ uint batch_idx AXIS(batch_size, 16)
) {
- int batch_idx = tid.y;
- int dim_idx = tid.x;
- if (dim_idx >= hidden_dim)
- return;
-
float sum = 0.0f;
- for (int seq_idx = 0; seq_idx < seq_len; seq_idx++) {
+ for (uint seq_idx = 0; seq_idx < seq_len; seq_idx++) {
sum += float(
input[batch_idx * seq_len * hidden_dim + seq_idx * hidden_dim + dim_idx]
);
}
output[batch_idx * hidden_dim + dim_idx] = T(sum / float(seq_len));
}
-
-// Explicit instantiations for f16, f32, bf16
-template [[host_name("pool_cls_f16")]] [[kernel]] void pool_cls<half>(
- const device half* input [[buffer(0)]],
- device half* output [[buffer(1)]],
- constant int& seq_len [[buffer(2)]],
- constant int& hidden_dim [[buffer(3)]],
- uint2 tid [[thread_position_in_grid]]
-);
-
-template [[host_name("pool_cls_f32")]] [[kernel]] void pool_cls<float>(
- const device float* input [[buffer(0)]],
- device float* output [[buffer(1)]],
- constant int& seq_len [[buffer(2)]],
- constant int& hidden_dim [[buffer(3)]],
- uint2 tid [[thread_position_in_grid]]
-);
-
-template [[host_name("pool_cls_bf16")]] [[kernel]] void pool_cls<bfloat>(
- const device bfloat* input [[buffer(0)]],
- device bfloat* output [[buffer(1)]],
- constant int& seq_len [[buffer(2)]],
- constant int& hidden_dim [[buffer(3)]],
- uint2 tid [[thread_position_in_grid]]
-);
-
-template [[host_name("pool_mean_f16")]] [[kernel]] void pool_mean<half>(
- const device half* input [[buffer(0)]],
- device half* output [[buffer(1)]],
- constant int& seq_len [[buffer(2)]],
- constant int& hidden_dim [[buffer(3)]],
- uint2 tid [[thread_position_in_grid]]
-);
-
-template [[host_name("pool_mean_f32")]] [[kernel]] void pool_mean<float>(
- const device float* input [[buffer(0)]],
- device float* output [[buffer(1)]],
- constant int& seq_len [[buffer(2)]],
- constant int& hidden_dim [[buffer(3)]],
- uint2 tid [[thread_position_in_grid]]
-);
-
-template [[host_name("pool_mean_bf16")]] [[kernel]] void pool_mean<bfloat>(
- const device bfloat* input [[buffer(0)]],
- device bfloat* output [[buffer(1)]],
- constant int& seq_len [[buffer(2)]],
- constant int& hidden_dim [[buffer(3)]],
- uint2 tid [[thread_position_in_grid]]
-);
diff --git a/crates/uzu/src/backends/metal/kernel/quant_matmul/mod.rs b/crates/uzu/src/backends/metal/kernel/quant_matmul/mod.rs
index b63bad3f6..a34d37522 100644
--- a/crates/uzu/src/backends/metal/kernel/quant_matmul/mod.rs
+++ b/crates/uzu/src/backends/metal/kernel/quant_matmul/mod.rs
@@ -3,9 +3,9 @@ use std::{collections::HashMap, ptr::NonNull};
use crate::{
DataType,
backends::metal::{
- MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext, MTLError,
- MTLFunctionConstantValues, MTLSize, ProtocolObject, Retained,
- metal_extensions::ComputeEncoderSetValue,
+ MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState,
+ MTLContext, MTLError, MTLFunctionConstantValues, MTLSize,
+ ProtocolObject, Retained, metal_extensions::ComputeEncoderSetValue,
},
config::QuantizationMode,
};
@@ -37,7 +37,10 @@ pub enum QuantizedMatmulError {
}
pub struct QuantizedMatmulKernel {
- pipelines: HashMap<KernelKind, (Retained<ProtocolObject<dyn MTLComputePipelineState>>, u64, u64)>,
+ pipelines: HashMap<
+ KernelKind,
+ (Retained<ProtocolObject<dyn MTLComputePipelineState>>, u64, u64),
+ >,
output_dim: usize,
weights_transposed: bool,
}
diff --git a/crates/uzu/src/backends/metal/kernel/quant_matmul/quant_matmul.metal b/crates/uzu/src/backends/metal/kernel/quant_matmul/quant_matmul.metal
index 1f11b9008..f4be25cb6 100644
--- a/crates/uzu/src/backends/metal/kernel/quant_matmul/quant_matmul.metal
+++ b/crates/uzu/src/backends/metal/kernel/quant_matmul/quant_matmul.metal
@@ -4196,6 +4196,20 @@ qmm_transposed<bfloat, 128, 4, true, 128, 32, 64>(
/// MLP Fused QMV - Quantized Matrix-Vector with Fused Activation
///////////////////////////////////////////////////////////////////////////////
+template <typename U>
+static inline U extract_zp(
+ const device uint8_t* zps,
+ int row_offset,
+ int zp_stride,
+ int bits,
+ bool high_nibble
+) {
+ uint8_t zp_byte = zps[row_offset * zp_stride];
+ return bits == 4
+ ? static_cast<U>(high_nibble ? (zp_byte >> 4) : (zp_byte & 0x0F))
+ : static_cast<U>(zp_byte);
+};
+
// MLP fused QMV kernel for decode path (M=1).
// Computes paired up and gate projections, then applies: out = up *
// activation(gate) Weight layout: [up_weights (hidden_dim rows), gate_weights
@@ -4381,26 +4395,15 @@ void qmv_mlp_fused_impl(
sum
);
} else {
- auto extract_zp = [&](const device uint8_t* zps, int row_offset) -> U {
- uint8_t zp_byte = zps[row_offset * zp_stride];
- if (bits == 4) {
- return static_cast<U>(
- high_nibble ? (zp_byte >> 4) : (zp_byte & 0x0F)
- );
- } else {
- return static_cast<U>(zp_byte);
- }
- };
-
- U zp_up0 = extract_zp(zps_up, 0);
- U zp_up1 = extract_zp(zps_up, 1);
- U zp_up2 = extract_zp(zps_up, 2);
- U zp_up3 = extract_zp(zps_up, 3);
-
- U zp_gate0 = extract_zp(zps_gate, 0);
- U zp_gate1 = extract_zp(zps_gate, 1);
- U zp_gate2 = extract_zp(zps_gate, 2);
- U zp_gate3 = extract_zp(zps_gate, 3);
+ U zp_up0 = extract_zp<U>(zps_up, 0, zp_stride, bits, high_nibble);
+ U zp_up1 = extract_zp<U>(zps_up, 1, zp_stride, bits, high_nibble);
+ U zp_up2 = extract_zp<U>(zps_up, 2, zp_stride, bits, high_nibble);
+ U zp_up3 = extract_zp<U>(zps_up, 3, zp_stride, bits, high_nibble);
+
+ U zp_gate0 = extract_zp<U>(zps_gate, 0, zp_stride, bits, high_nibble);
+ U zp_gate1 = extract_zp<U>(zps_gate, 1, zp_stride, bits, high_nibble);
+ U zp_gate2 = extract_zp<U>(zps_gate, 2, zp_stride, bits, high_nibble);
+ U zp_gate3 = extract_zp<U>(zps_gate, 3, zp_stride, bits, high_nibble);
result_up[0] += qdot_zero_point<U, values_per_thread, bits>(
wl_up0,
diff --git a/crates/uzu/src/backends/metal/kernel/rms_norm/mod.rs b/crates/uzu/src/backends/metal/kernel/rms_norm/mod.rs
index 090c39f64..19e9b5c8d 100644
--- a/crates/uzu/src/backends/metal/kernel/rms_norm/mod.rs
+++ b/crates/uzu/src/backends/metal/kernel/rms_norm/mod.rs
@@ -1,8 +1,9 @@
use crate::{
DataType,
backends::metal::{
- ComputeEncoderSetValue, MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState,
- MTLContext, MTLError, MTLSize, ProtocolObject, Retained,
+ ComputeEncoderSetValue, MTLBuffer, MTLComputeCommandEncoder,
+ MTLComputePipelineState, MTLContext, MTLError, MTLSize, ProtocolObject,
+ Retained,
},
};
diff --git a/crates/uzu/src/backends/metal/kernel/rope/mod.rs b/crates/uzu/src/backends/metal/kernel/rope/mod.rs
index 7be3ef0fc..2fadd633f 100644
--- a/crates/uzu/src/backends/metal/kernel/rope/mod.rs
+++ b/crates/uzu/src/backends/metal/kernel/rope/mod.rs
@@ -1,8 +1,9 @@
use thiserror::Error;
use crate::backends::metal::{
- ComputeEncoderSetValue, KernelDataType, MTLBuffer, MTLComputeCommandEncoder,
- MTLComputePipelineState, MTLContext, MTLError, MTLSize, ProtocolObject, Retained,
+ ComputeEncoderSetValue, KernelDataType, MTLBuffer,
+ MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext, MTLError,
+ MTLSize, ProtocolObject, Retained,
};
pub struct RopeKernel {
diff --git a/crates/uzu/src/backends/metal/kernel/sampling/argmax.metal b/crates/uzu/src/backends/metal/kernel/sampling/argmax.metal
index af6db77a3..2de31b445 100644
--- a/crates/uzu/src/backends/metal/kernel/sampling/argmax.metal
+++ b/crates/uzu/src/backends/metal/kernel/sampling/argmax.metal
@@ -155,7 +155,9 @@ static ArgmaxPair threadgroup_raking_argmax(
// Single-pass argmax
-SPECIALIZE(T, float, half, bfloat) KERNEL(ArgmaxSingle)(
+template <typename T>
+VARIANTS(T, float, half, bfloat)
+KERNEL(ArgmaxSingle)(
const device T* logits_data,
device uint* final_tokens,
constant uint& batch_size,
@@ -195,7 +197,9 @@ SPECIALIZE(T, float, half, bfloat) KERNEL(ArgmaxSingle)(
// Two-pass argmax
-SPECIALIZE(T, float, half, bfloat) KERNEL(ArgmaxMain)(
+template <typename T>
+VARIANTS(T, float, half, bfloat)
+KERNEL(ArgmaxMain)(
const device T* logits_data,
device ArgmaxPair* partial_results,
constant uint& batch_size,
diff --git a/crates/uzu/src/backends/metal/kernel/sampling/bitmask.metal b/crates/uzu/src/backends/metal/kernel/sampling/bitmask.metal
index b4ee1d1a1..0b1b017d0 100644
--- a/crates/uzu/src/backends/metal/kernel/sampling/bitmask.metal
+++ b/crates/uzu/src/backends/metal/kernel/sampling/bitmask.metal
@@ -5,7 +5,9 @@
#define GRAIN_SIZE 64
#define BITS_IN_U32 32
-SPECIALIZE(T, float, half, bfloat) KERNEL(Bitmask) (
+template <typename T>
+VARIANTS(T, float, half, bfloat)
+KERNEL(Bitmask) (
device const T* logits,
device const uint32_t* bitmask,
constant uint& bitmask_offset,
diff --git a/crates/uzu/src/backends/metal/kernel/sampling/gumbel.metal b/crates/uzu/src/backends/metal/kernel/sampling/gumbel.metal
index 4ef098927..f298730dd 100644
--- a/crates/uzu/src/backends/metal/kernel/sampling/gumbel.metal
+++ b/crates/uzu/src/backends/metal/kernel/sampling/gumbel.metal
@@ -6,7 +6,9 @@
#define GRAIN_SIZE 64
#define WORDS_PER_OFFSET 4
-SPECIALIZE(T, float, half, bfloat) KERNEL(Gumbel) (
+template <typename T>
+VARIANTS(T, float, half, bfloat)
+KERNEL(Gumbel) (
device const T* logits,
device const uint64_t* batch_seeds,
device T* processed_logits,
diff --git a/crates/uzu/src/backends/metal/kernel/sampling/min_p.metal b/crates/uzu/src/backends/metal/kernel/sampling/min_p.metal
index 6c845a38b..aca8878fb 100644
--- a/crates/uzu/src/backends/metal/kernel/sampling/min_p.metal
+++ b/crates/uzu/src/backends/metal/kernel/sampling/min_p.metal
@@ -4,7 +4,9 @@
#define BLOCK_SIZE 1024
#define BLOCK_SIZE_IN_SIMDS (BLOCK_SIZE / 32)
-SPECIALIZE(T, float, half, bfloat) KERNEL(MinP) (
+template <typename T>
+VARIANTS(T, float, half, bfloat)
+KERNEL(MinP) (
device const T* logits,
device T* processed_logits,
threadgroup float shared_reduce_buffer[BLOCK_SIZE_IN_SIMDS],
diff --git a/crates/uzu/src/backends/metal/kernel/sampling/temperature.metal b/crates/uzu/src/backends/metal/kernel/sampling/temperature.metal
index a462c4655..fcb47eee1 100644
--- a/crates/uzu/src/backends/metal/kernel/sampling/temperature.metal
+++ b/crates/uzu/src/backends/metal/kernel/sampling/temperature.metal
@@ -4,7 +4,9 @@
#define BLOCK_SIZE 1024
#define GRAIN_SIZE 64
-SPECIALIZE(T, float, half, bfloat) KERNEL(Temperature) (
+template <typename T>
+VARIANTS(T, float, half, bfloat)
+KERNEL(Temperature) (
device const T* logits,
device T* processed_logits,
constant uint& batch_size,
diff --git a/crates/uzu/src/backends/metal/kernel/sampling/top_k.metal b/crates/uzu/src/backends/metal/kernel/sampling/top_k.metal
index 9b5c856af..bfefe0380 100644
--- a/crates/uzu/src/backends/metal/kernel/sampling/top_k.metal
+++ b/crates/uzu/src/backends/metal/kernel/sampling/top_k.metal
@@ -5,7 +5,9 @@
#define BLOCK_SIZE_IN_SIMDS (BLOCK_SIZE / 32)
#define MAX_ITERS 16
-SPECIALIZE(T, float, half, bfloat) KERNEL(TopK) (
+template <typename T>
+VARIANTS(T, float, half, bfloat)
+KERNEL(TopK) (
device const T* logits,
device T* processed_logits,
threadgroup float shared_reduce_buffer[BLOCK_SIZE_IN_SIMDS],
diff --git a/crates/uzu/src/backends/metal/kernel/sampling/top_p.metal b/crates/uzu/src/backends/metal/kernel/sampling/top_p.metal
index cb0b8654b..b1c3bc481 100644
--- a/crates/uzu/src/backends/metal/kernel/sampling/top_p.metal
+++ b/crates/uzu/src/backends/metal/kernel/sampling/top_p.metal
@@ -6,7 +6,9 @@
#define MAX_ITERS 16
-SPECIALIZE(T, float, half, bfloat) KERNEL(TopP) (
+template <typename T>
+VARIANTS(T, float, half, bfloat)
+KERNEL(TopP) (
device const T* logits,
device T* processed_logits,
threadgroup float shared_reduce_buffer[BLOCK_SIZE_IN_SIMDS],
diff --git a/crates/uzu/src/backends/metal/kernel/short_conv/mod.rs b/crates/uzu/src/backends/metal/kernel/short_conv/mod.rs
index 3e9dd2ed5..3ba1ee509 100644
--- a/crates/uzu/src/backends/metal/kernel/short_conv/mod.rs
+++ b/crates/uzu/src/backends/metal/kernel/short_conv/mod.rs
@@ -1,9 +1,9 @@
use std::ptr::NonNull;
use crate::backends::metal::{
- ComputeEncoderSetValue, KernelDataType, MTLBuffer, MTLComputeCommandEncoder,
- MTLComputePipelineState, MTLContext, MTLDataType, MTLError, MTLFunctionConstantValues,
- MTLSize, ProtocolObject, Retained,
+ ComputeEncoderSetValue, KernelDataType, MTLBuffer,
+ MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext, MTLDataType,
+ MTLError, MTLFunctionConstantValues, MTLSize, ProtocolObject, Retained,
};
#[derive(Debug, thiserror::Error)]
@@ -12,12 +12,13 @@ pub enum ShortConvKernelError {
MetalError(#[from] MTLError),
}
-
fn fn_suffix(dt: KernelDataType) -> &'static str {
dt.function_name_suffix()
}
-fn make_function_constants(has_bias: bool) -> Retained<MTLFunctionConstantValues> {
+fn make_function_constants(
+ has_bias: bool
+) -> Retained<MTLFunctionConstantValues> {
let function_constants = MTLFunctionConstantValues::new();
function_constants.set_constant_value_type_at_index(
NonNull::from(&has_bias).cast(),
@@ -29,10 +30,14 @@ fn make_function_constants(has_bias: bool) -> Retained<MTLFunctionConstantValues
pub struct ShortConvKernel {
pack_pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- prefill_pipeline_no_bias: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- prefill_pipeline_with_bias: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- decode_pipeline_no_bias: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- decode_pipeline_with_bias: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ prefill_pipeline_no_bias:
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ prefill_pipeline_with_bias:
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ decode_pipeline_no_bias:
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ decode_pipeline_with_bias:
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
}
pub struct ShortConvPackArguments<'a> {
diff --git a/crates/uzu/src/backends/metal/kernel/sigmoid/mod.rs b/crates/uzu/src/backends/metal/kernel/sigmoid/mod.rs
deleted file mode 100644
index 110c0f9d4..000000000
--- a/crates/uzu/src/backends/metal/kernel/sigmoid/mod.rs
+++ /dev/null
@@ -1,56 +0,0 @@
-use crate::{
- DataType,
- backends::metal::{
- ComputeEncoderSetValue, MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState,
- MTLContext, MTLError, MTLSize, ProtocolObject, Retained,
- },
-};
-
-pub struct SigmoidKernel {
- pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-impl SigmoidKernel {
- fn kernel_name_for_type(
- data_type: DataType
- ) -> Result<&'static str, MTLError> {
- match data_type {
- DataType::F16 => Ok("apply_sigmoid_f16"),
- DataType::F32 => Ok("apply_sigmoid_f32"),
- DataType::BF16 => Ok("apply_sigmoid_bf16"),
- other => Err(MTLError::Generic(format!(
- "Unsupported dtype for sigmoid: {:?}",
- other
- ))),
- }
- }
-
- pub fn new(
- context: &MTLContext,
- data_type: DataType,
- ) -> Result<Self, MTLError> {
- let fn_name = Self::kernel_name_for_type(data_type)?;
- let pipeline = context.compute_pipeline_state(fn_name, None)?;
- Ok(Self {
- pipeline,
- })
- }
-
- pub fn encode(
- &self,
- encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- input_buffer: &ProtocolObject<dyn MTLBuffer>,
- output_buffer: &ProtocolObject<dyn MTLBuffer>,
- total_elements: i32,
- ) -> Result<(), MTLError> {
- encoder.set_compute_pipeline_state(&self.pipeline);
- encoder.set_buffer(Some(input_buffer), 0, 0);
- encoder.set_buffer(Some(output_buffer), 0, 1);
- encoder.set_value(&total_elements, 2);
-
- let threads_per_tg = MTLSize::new(256, 1, 1);
- let grid = MTLSize::new(total_elements as usize, 1, 1);
- encoder.dispatch_threads(grid, threads_per_tg);
- Ok(())
- }
-}
diff --git a/crates/uzu/src/backends/metal/kernel/sigmoid/sigmoid.metal b/crates/uzu/src/backends/metal/kernel/sigmoid/sigmoid.metal
index a76de23c0..890c41e8d 100644
--- a/crates/uzu/src/backends/metal/kernel/sigmoid/sigmoid.metal
+++ b/crates/uzu/src/backends/metal/kernel/sigmoid/sigmoid.metal
@@ -1,42 +1,16 @@
#include <metal_stdlib>
#include "../definitions.metal"
-using namespace metal;
-// Apply sigmoid element-wise to logits
template <typename T>
-[[kernel, max_total_threads_per_threadgroup(256)]]
-void apply_sigmoid(
- const device T* input [[buffer(0)]],
- device T* output [[buffer(1)]],
- constant int& total_elements [[buffer(2)]],
- uint tid [[thread_position_in_grid]]
+VARIANTS(T, float, half, bfloat)
+KERNEL(Sigmoid) (
+ const device T* input,
+ device T* output,
+ const constant uint& total_elements,
+ uint tid AXIS(total_elements, 256)
) {
if (tid >= total_elements)
return;
float x = float(input[tid]);
output[tid] = T(1.0f / (1.0f + exp(-x)));
-}
-
-// Explicit instantiations
-template [[host_name("apply_sigmoid_f16")]] [[kernel]] void apply_sigmoid<half>(
- const device half* input [[buffer(0)]],
- device half* output [[buffer(1)]],
- constant int& total_elements [[buffer(2)]],
- uint tid [[thread_position_in_grid]]
-);
-
-template [[host_name("apply_sigmoid_f32")]] [[kernel]] void apply_sigmoid<
- float>(
- const device float* input [[buffer(0)]],
- device float* output [[buffer(1)]],
- constant int& total_elements [[buffer(2)]],
- uint tid [[thread_position_in_grid]]
-);
-
-template [[host_name("apply_sigmoid_bf16")]] [[kernel]] void apply_sigmoid<
- bfloat>(
- const device bfloat* input [[buffer(0)]],
- device bfloat* output [[buffer(1)]],
- constant int& total_elements [[buffer(2)]],
- uint tid [[thread_position_in_grid]]
-);
+}
\ No newline at end of file
diff --git a/crates/uzu/src/backends/metal/kernel/ssm/conv1d_scan.metal b/crates/uzu/src/backends/metal/kernel/ssm/conv1d_scan.metal
index 2e1bf340a..35739e2d5 100644
--- a/crates/uzu/src/backends/metal/kernel/ssm/conv1d_scan.metal
+++ b/crates/uzu/src/backends/metal/kernel/ssm/conv1d_scan.metal
@@ -1,5 +1,6 @@
#include <metal_stdlib>
#include "../definitions.metal"
+#include "ssm_common.h"
using namespace metal;
@@ -12,14 +13,6 @@ constant int ACTIVATION_GELU = 2;
constant uint CONV_SCAN_THREADS = 32u;
-template <typename T>
-inline T apply_silu(T x) {
- float xf = float(x);
- float y = 1.0f / (1.0f + fast::exp(-fabs(xf)));
- float out = (xf < 0.0f) ? (1.0f - y) * xf : y * xf;
- return static_cast<T>(out);
-}
-
template <typename T>
inline T apply_gelu(T x) {
float xf = float(x);
diff --git a/crates/uzu/src/backends/metal/kernel/ssm/conv1d_scan.rs b/crates/uzu/src/backends/metal/kernel/ssm/conv1d_scan.rs
index 2247643fa..36e1c486d 100644
--- a/crates/uzu/src/backends/metal/kernel/ssm/conv1d_scan.rs
+++ b/crates/uzu/src/backends/metal/kernel/ssm/conv1d_scan.rs
@@ -3,8 +3,9 @@ use std::ptr::NonNull;
use super::{SSMKernelError, fn_suffix};
use crate::{
backends::metal::{
- KernelDataType, MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext,
- MTLDataType, MTLFunctionConstantValues, MTLSize, ProtocolObject, Retained,
+ KernelDataType, MTLBuffer, MTLComputeCommandEncoder,
+ MTLComputePipelineState, MTLContext, MTLDataType,
+ MTLFunctionConstantValues, MTLSize, ProtocolObject, Retained,
metal_extensions::ComputeEncoderSetValue,
},
config::Activation,
@@ -52,8 +53,10 @@ pub struct Conv1dScanKernel {
pipeline_no_bias: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
pipeline_with_bias: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
pack_pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- decode_pipeline_no_bias: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- decode_pipeline_with_bias: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ decode_pipeline_no_bias:
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
+ decode_pipeline_with_bias:
+ Retained<ProtocolObject<dyn MTLComputePipelineState>>,
}
pub struct Conv1dScanArguments<'a> {
diff --git a/crates/uzu/src/backends/metal/kernel/ssm/mod.rs b/crates/uzu/src/backends/metal/kernel/ssm/mod.rs
index baf58a079..fe660542b 100644
--- a/crates/uzu/src/backends/metal/kernel/ssm/mod.rs
+++ b/crates/uzu/src/backends/metal/kernel/ssm/mod.rs
@@ -13,13 +13,9 @@ fn fn_suffix(dt: KernelDataType) -> &'static str {
}
pub mod conv1d_scan;
-pub mod split_inproj;
pub mod ssd_prefill;
-pub mod ssd_update;
pub use conv1d_scan::{
Conv1dPackArguments, Conv1dScanArguments, Conv1dScanKernel,
};
-pub use split_inproj::{SplitInProjArguments, SplitInProjKernel};
-pub use ssd_prefill::{SSDPrefillArguments, SSDPrefillKernel, SSDPrefillMode};
-pub use ssd_update::{SSDUpdateArguments, SSDUpdateKernel};
+pub use ssd_prefill::{SSDPrefillArguments, SSDPrefillKernels, SSDPrefillMode};
diff --git a/crates/uzu/src/backends/metal/kernel/ssm/split_inproj.metal b/crates/uzu/src/backends/metal/kernel/ssm/split_inproj.metal
index 87562ff04..a4949bb9e 100644
--- a/crates/uzu/src/backends/metal/kernel/ssm/split_inproj.metal
+++ b/crates/uzu/src/backends/metal/kernel/ssm/split_inproj.metal
@@ -1,30 +1,22 @@
#include <metal_stdlib>
#include "../definitions.metal"
-using namespace metal;
-
template <typename T>
-[[kernel, max_total_threads_per_threadgroup(256)]]
-void ssm_split_inproj_kernel(
- device const T* input [[buffer(0)]],
- device T* conv_out [[buffer(1)]],
- device T* z_out [[buffer(2)]],
- device T* dt_out [[buffer(3)]],
- device const T* z_bias [[buffer(4)]],
- constant const int& total_dim [[buffer(5)]],
- constant const int& conv_dim [[buffer(6)]],
- constant const int& inner_dim [[buffer(7)]],
- constant const int& num_heads [[buffer(8)]],
- uint2 grid_idx [[thread_position_in_grid]],
- uint2 grid_size [[threads_per_grid]]
+VARIANTS(T, float, half, bfloat)
+KERNEL(SplitInProj)(
+ device const T* input,
+ device T* conv_out,
+ device T* z_out,
+ device T* dt_out,
+ device const T* z_bias,
+ constant const uint& suffix_length,
+ constant const uint& total_dim,
+ constant const uint& conv_dim,
+ constant const uint& inner_dim,
+ constant const uint& num_heads,
+ const uint row AXIS(suffix_length, 16),
+ const uint col AXIS(total_dim, 16)
) {
- const int row = static_cast<int>(grid_idx.x);
- const int col = static_cast<int>(grid_idx.y);
-
- if (row >= grid_size.x || col >= grid_size.y || col >= total_dim) {
- return;
- }
-
const int input_idx = row * total_dim + col;
if (col < conv_dim) {
const int dst = row * conv_dim + col;
@@ -37,26 +29,4 @@ void ssm_split_inproj_kernel(
const int dst = row * num_heads + (col - conv_dim - inner_dim);
dt_out[dst] = input[input_idx];
}
-}
-
-#define instantiate_ssm_split_inproj_kernel(type_name, type) \
- template [[host_name("ssm_split_inproj_kernel_" #type_name)]] \
- kernel void ssm_split_inproj_kernel<type>( \
- device const type* input [[buffer(0)]], \
- device type* conv_out [[buffer(1)]], \
- device type* z_out [[buffer(2)]], \
- device type* dt_out [[buffer(3)]], \
- device const type* z_bias [[buffer(4)]], \
- constant const int& total_dim [[buffer(5)]], \
- constant const int& conv_dim [[buffer(6)]], \
- constant const int& inner_dim [[buffer(7)]], \
- constant const int& num_heads [[buffer(8)]], \
- uint2 grid_idx [[thread_position_in_grid]], \
- uint2 grid_size [[threads_per_grid]] \
- );
-
-instantiate_ssm_split_inproj_kernel(float, float);
-instantiate_ssm_split_inproj_kernel(bfloat, bfloat);
-instantiate_ssm_split_inproj_kernel(half, half);
-
-#undef instantiate_ssm_split_inproj_kernel
+}
\ No newline at end of file
diff --git a/crates/uzu/src/backends/metal/kernel/ssm/split_inproj.rs b/crates/uzu/src/backends/metal/kernel/ssm/split_inproj.rs
deleted file mode 100644
index 4afb04c78..000000000
--- a/crates/uzu/src/backends/metal/kernel/ssm/split_inproj.rs
+++ /dev/null
@@ -1,79 +0,0 @@
-use super::{SSMKernelError, fn_suffix};
-use crate::backends::metal::{
- KernelDataType, MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext,
- MTLSize, ProtocolObject, Retained,
- metal_extensions::ComputeEncoderSetValue,
-};
-
-pub struct SplitInProjKernel {
- pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-pub struct SplitInProjArguments<'a> {
- pub input: &'a ProtocolObject<dyn MTLBuffer>, // buffer(0) [suffix, total_dim]
- pub conv_out: &'a ProtocolObject<dyn MTLBuffer>, // buffer(1) [suffix, conv_dim]
- pub z_out: &'a ProtocolObject<dyn MTLBuffer>, // buffer(2) [suffix, inner_dim]
- pub dt_out: &'a ProtocolObject<dyn MTLBuffer>, // buffer(3) [suffix, num_heads]
- pub z_bias: &'a ProtocolObject<dyn MTLBuffer>, // buffer(4) [inner_dim]
- pub total_dim: usize,
- pub conv_dim: usize,
- pub inner_dim: usize,
- pub num_heads: usize,
- pub suffix_length: usize,
-}
-
-impl SplitInProjKernel {
- pub fn new(
- context: &MTLContext,
- data_type: KernelDataType,
- ) -> Result<Self, SSMKernelError> {
- let fn_name =
- format!("ssm_split_inproj_kernel_{}", fn_suffix(data_type));
- let pipeline = context
- .compute_pipeline_state(&fn_name, None)
- .map_err(SSMKernelError::MetalError)?;
- Ok(Self {
- pipeline,
- })
- }
-
- pub fn encode(
- &self,
- compute_encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- args: SplitInProjArguments,
- ) -> Result<(), SSMKernelError> {
- compute_encoder.set_compute_pipeline_state(&self.pipeline);
- compute_encoder.set_buffer(Some(args.input), 0, 0);
- compute_encoder.set_buffer(Some(args.conv_out), 0, 1);
- compute_encoder.set_buffer(Some(args.z_out), 0, 2);
- compute_encoder.set_buffer(Some(args.dt_out), 0, 3);
- compute_encoder.set_buffer(Some(args.z_bias), 0, 4);
-
- let total_dim = args.total_dim as i32;
- let conv_dim = args.conv_dim as i32;
- let inner_dim = args.inner_dim as i32;
- let num_heads = args.num_heads as i32;
- let suffix = args.suffix_length;
- let cols = args.total_dim;
-
- compute_encoder.set_value(&total_dim, 5);
- compute_encoder.set_value(&conv_dim, 6);
- compute_encoder.set_value(&inner_dim, 7);
- compute_encoder.set_value(&num_heads, 8);
-
- let threads_per_threadgroup = MTLSize {
- width: 16,
- height: 16,
- depth: 1,
- };
- let total_threads = MTLSize {
- width: suffix,
- height: cols,
- depth: 1,
- };
-
- compute_encoder
- .dispatch_threads(total_threads, threads_per_threadgroup);
- Ok(())
- }
-}
diff --git a/crates/uzu/src/backends/metal/kernel/ssm/ssd_prefill.metal b/crates/uzu/src/backends/metal/kernel/ssm/ssd_prefill.metal
index a56e0e974..a1966002e 100644
--- a/crates/uzu/src/backends/metal/kernel/ssm/ssd_prefill.metal
+++ b/crates/uzu/src/backends/metal/kernel/ssm/ssd_prefill.metal
@@ -1,127 +1,99 @@
#include <metal_stdlib>
#include <metal_simdgroup>
#include "../definitions.metal"
+#include "ssm_common.h"
using namespace metal;
constant ushort SSM_PREFILL_MAX_STATE = 256;
-struct SILU {
- template <typename T>
- T operator()(T x) const {
- float xf = float(x);
- float y = 1.0f / (1.0f + fast::exp(-fabs(xf)));
- float out = (xf < 0.0f) ? (1.0f - y) * xf : y * xf;
- return static_cast<T>(out);
- }
-};
-
-template <typename T>
-inline T softplus(T x) {
- float xf = float(x);
- if (xf > 20.0f) {
- return x;
- }
- return static_cast<T>(log(1.0f + fast::exp(xf)));
-}
-
template <typename T>
-[[kernel, max_total_threads_per_threadgroup(32)]]
-void ssd_prefill_kernel_64(
- device const T* x [[buffer(0)]], // (suffix, h, dh)
- device const T* dt_raw [[buffer(1)]], // (suffix, h) - raw dt values
- device const T* B [[buffer(2)]], // (suffix, g, n)
- device const T* C [[buffer(3)]], // (suffix, g, n)
- device const T* D [[buffer(4)]], // (h)
- device const T* z [[buffer(5)]], // (suffix, h, dh)
- device T* state [[buffer(6)]], // (h, dh, n)
- device T* y [[buffer(7)]], // (suffix, h, dh)
- constant const size_t& suffix_len [[buffer(8)]],
- constant const int& group_size [[buffer(9)]],
- constant const int& state_size [[buffer(10)]],
- constant const size_t* x_strides [[buffer(11)]],
- constant const size_t* dt_strides [[buffer(12)]],
- constant const size_t* cb_strides [[buffer(13)]],
- constant const size_t* state_strides [[buffer(14)]],
- constant const uint& num_heads [[buffer(15)]],
- constant const uint& head_dim [[buffer(16)]],
- uint3 tg_pos [[threadgroup_position_in_grid]],
- ushort lane [[thread_index_in_threadgroup]]
+VARIANTS(T, float, half, bfloat)
+KERNEL(SSDPrefill64)(
+ device const T* x, // (suffix, h, dh)
+ device const T* dt_raw, // (suffix, h) - raw dt values
+ device const T* b, // (suffix, g, n)
+ device const T* c, // (suffix, g, n)
+ device const T* d, // (h)
+ device const T* z, // (suffix, h, dh)
+ device T* state, // (h, dh, n)
+ device T* y, // (suffix, h, dh)
+ constant const uint& suffix_len,
+ constant const int& group_size,
+ constant const int& state_size,
+ constant const uint* x_strides,
+ constant const uint* dt_strides,
+ constant const uint* cb_strides,
+ constant const uint* state_strides,
+ constant const uint& num_heads,
+ constant const uint& head_dim,
+ const uint pair_idx GROUPS(num_heads * head_dim),
+ const uint lane_idx THREADS(32)
) {
const uint simd_width = 32;
const int state_dim = state_size;
-
- const uint total_pairs = num_heads * head_dim;
- const uint pair_idx = tg_pos.x;
- if (pair_idx >= total_pairs) {
- return;
- }
-
- const uint lane_idx = uint(lane);
const uint h_idx = pair_idx / head_dim;
const uint dh_idx = pair_idx % head_dim;
const uint safe_group = uint(max(group_size, 1));
const uint group_idx = h_idx / safe_group;
- const size_t x_token_stride = x_strides[0];
- const size_t x_head_stride = x_strides[1];
- const size_t x_dim_stride = x_strides[2];
- const size_t dt_token_stride = dt_strides[0];
- const size_t dt_head_stride = dt_strides[1];
- const size_t cb_token_stride = cb_strides[0];
- const size_t cb_group_stride = cb_strides[1];
- const size_t cb_state_stride = cb_strides[2];
- const size_t state_head_stride = state_strides[0];
- const size_t state_dim_stride = state_strides[1];
- const size_t state_inner_stride = state_strides[2];
-
- const size_t x_base =
- size_t(h_idx) * x_head_stride + size_t(dh_idx) * x_dim_stride;
- const size_t dt_base = size_t(h_idx) * dt_head_stride;
- const size_t state_base =
- size_t(h_idx) * state_head_stride + size_t(dh_idx) * state_dim_stride;
- const size_t cb_group_base = size_t(group_idx) * cb_group_stride;
- const float d_scalar = float(D[h_idx]);
+ const uint x_token_stride = x_strides[0];
+ const uint x_head_stride = x_strides[1];
+ const uint x_dim_stride = x_strides[2];
+ const uint dt_token_stride = dt_strides[0];
+ const uint dt_head_stride = dt_strides[1];
+ const uint cb_token_stride = cb_strides[0];
+ const uint cb_group_stride = cb_strides[1];
+ const uint cb_state_stride = cb_strides[2];
+ const uint state_head_stride = state_strides[0];
+ const uint state_dim_stride = state_strides[1];
+ const uint state_inner_stride = state_strides[2];
+
+ const uint x_base = h_idx * x_head_stride + dh_idx * x_dim_stride;
+ const uint dt_base = h_idx * dt_head_stride;
+ const uint state_base = h_idx * state_head_stride + dh_idx * state_dim_stride;
+ const uint cb_group_base = group_idx * cb_group_stride;
+ const float d_scalar = float(d[h_idx]);
const uint idx0 = lane_idx;
const uint idx1 = lane_idx + simd_width;
float state0 = 0.0f;
float state1 = 0.0f;
- size_t state_idx0 = 0;
- size_t state_idx1 = 0;
- state_idx0 = state_base + size_t(idx0) * state_inner_stride;
+ uint state_idx0 = 0;
+ uint state_idx1 = 0;
+ state_idx0 = state_base + idx0 * state_inner_stride;
state0 = float(state[state_idx0]);
- state_idx1 = state_base + size_t(idx1) * state_inner_stride;
+ state_idx1 = state_base + idx1 * state_inner_stride;
state1 = float(state[state_idx1]);
- size_t cb_idx0 = 0;
- size_t cb_idx1 = 0;
- cb_idx0 = cb_group_base + size_t(idx0) * cb_state_stride;
- cb_idx1 = cb_group_base + size_t(idx1) * cb_state_stride;
+ uint cb_idx0 = 0;
+ uint cb_idx1 = 0;
+ cb_idx0 = cb_group_base + idx0 * cb_state_stride;
+ cb_idx1 = cb_group_base + idx1 * cb_state_stride;
- for (size_t token = 0; token < suffix_len; ++token) {
- const size_t x_idx = token * x_token_stride + x_base;
- const size_t dt_idx = token * dt_token_stride + dt_base;
+ for (uint token = 0; token < suffix_len; ++token) {
+ const uint x_idx = token * x_token_stride + x_base;
+ const uint dt_idx = token * dt_token_stride + dt_base;
const float x_val = float(x[x_idx]);
const float decay_val = fast::exp(-float(softplus(float(dt_raw[dt_idx]))));
- const float gate = float(SILU{}(z[x_idx]));
+ const float gate = float(apply_silu(z[x_idx]));
const float skip = d_scalar * x_val;
const float dt_scaled_input = x_val;
float contrib = 0.0f;
const float new_state0 =
- decay_val * state0 + dt_scaled_input * float(B[cb_idx0]);
+ decay_val * state0 + dt_scaled_input * float(b[cb_idx0]);
state0 = new_state0;
- contrib += new_state0 * float(C[cb_idx0]);
+ contrib += new_state0 * float(c[cb_idx0]);
cb_idx0 += cb_token_stride;
const float new_state1 =
- decay_val * state1 + dt_scaled_input * float(B[cb_idx1]);
+ decay_val * state1 + dt_scaled_input * float(b[cb_idx1]);
state1 = new_state1;
- contrib += new_state1 * float(C[cb_idx1]);
+ contrib += new_state1 * float(c[cb_idx1]);
cb_idx1 += cb_token_stride;
float dot = simd_sum(contrib);
@@ -134,93 +106,53 @@ void ssd_prefill_kernel_64(
state[state_idx1] = static_cast<T>(state1);
}
-#define instantiate_ssd_prefill_kernel_64(type_name, type) \
- template [[host_name("ssd_prefill_kernel_64_" #type_name)]] \
- kernel void ssd_prefill_kernel_64<type>( \
- device const type* x [[buffer(0)]], \
- device const type* dt_raw [[buffer(1)]], \
- device const type* B [[buffer(2)]], \
- device const type* C [[buffer(3)]], \
- device const type* D [[buffer(4)]], \
- device const type* z [[buffer(5)]], \
- device type* state [[buffer(6)]], \
- device type* y [[buffer(7)]], \
- constant const size_t& suffix_len [[buffer(8)]], \
- constant const int& group_size [[buffer(9)]], \
- constant const int& state_size [[buffer(10)]], \
- constant const size_t* x_strides [[buffer(11)]], \
- constant const size_t* dt_strides [[buffer(12)]], \
- constant const size_t* cb_strides [[buffer(13)]], \
- constant const size_t* state_strides [[buffer(14)]], \
- constant const uint& num_heads [[buffer(15)]], \
- constant const uint& head_dim [[buffer(16)]], \
- uint3 tg_pos [[threadgroup_position_in_grid]], \
- ushort lane [[thread_index_in_threadgroup]] \
- );
-
-instantiate_ssd_prefill_kernel_64(float, float);
-instantiate_ssd_prefill_kernel_64(bfloat, bfloat);
-instantiate_ssd_prefill_kernel_64(half, half);
-
-#undef instantiate_ssd_prefill_kernel_64
-
template <typename T>
-[[kernel, max_total_threads_per_threadgroup(32)]]
-void ssd_prefill_kernel(
- device const T* x [[buffer(0)]], // (suffix, h, dh)
- device const T* dt_raw [[buffer(1)]], // (suffix, h) - raw dt values
- device const T* B [[buffer(2)]], // (suffix, g, n)
- device const T* C [[buffer(3)]], // (suffix, g, n)
- device const T* D [[buffer(4)]], // (h)
- device const T* z [[buffer(5)]], // (suffix, h, dh)
- device T* state [[buffer(6)]], // (h, dh, n)
- device T* y [[buffer(7)]], // (suffix, h, dh)
- constant const size_t& suffix_len [[buffer(8)]],
- constant const int& group_size [[buffer(9)]],
- constant const int& state_size [[buffer(10)]],
- constant const size_t* x_strides [[buffer(11)]],
- constant const size_t* dt_strides [[buffer(12)]],
- constant const size_t* cb_strides [[buffer(13)]],
- constant const size_t* state_strides [[buffer(14)]],
- constant const uint& num_heads [[buffer(15)]],
- constant const uint& head_dim [[buffer(16)]],
- uint3 tg_pos [[threadgroup_position_in_grid]],
- ushort lane [[thread_index_in_threadgroup]]
+VARIANTS(T, float, half, bfloat)
+KERNEL(SSDPrefill)(
+ device const T* x, // (suffix, h, dh)
+ device const T* dt_raw, // (suffix, h) - raw dt values
+ device const T* b, // (suffix, g, n)
+ device const T* c, // (suffix, g, n)
+ device const T* d, // (h)
+ device const T* z, // (suffix, h, dh)
+ device T* state, // (h, dh, n)
+ device T* y, // (suffix, h, dh)
+ constant const uint& suffix_len,
+ constant const int& group_size,
+ constant const int& state_size,
+ constant const uint* x_strides,
+ constant const uint* dt_strides,
+ constant const uint* cb_strides,
+ constant const uint* state_strides,
+ constant const uint& num_heads,
+ constant const uint& head_dim,
+ const uint pair_idx GROUPS(num_heads * head_dim),
+ const uint lane_idx THREADS(32)
) {
const uint simd_width = 32;
const int state_dim = state_size;
-
- const uint total_pairs = num_heads * head_dim;
- const uint pair_idx = tg_pos.x;
- if (pair_idx >= total_pairs) {
- return;
- }
-
- const uint lane_idx = uint(lane);
const uint h_idx = pair_idx / head_dim;
const uint dh_idx = pair_idx % head_dim;
const uint safe_group = uint(max(group_size, 1));
const uint group_idx = h_idx / safe_group;
- const size_t x_token_stride = x_strides[0];
- const size_t x_head_stride = x_strides[1];
- const size_t x_dim_stride = x_strides[2];
- const size_t dt_token_stride = dt_strides[0];
- const size_t dt_head_stride = dt_strides[1];
- const size_t cb_token_stride = cb_strides[0];
- const size_t cb_group_stride = cb_strides[1];
- const size_t cb_state_stride = cb_strides[2];
- const size_t state_head_stride = state_strides[0];
- const size_t state_dim_stride = state_strides[1];
- const size_t state_inner_stride = state_strides[2];
-
- const size_t x_base =
- size_t(h_idx) * x_head_stride + size_t(dh_idx) * x_dim_stride;
- const size_t dt_base = size_t(h_idx) * dt_head_stride;
- const size_t state_base =
- size_t(h_idx) * state_head_stride + size_t(dh_idx) * state_dim_stride;
- const size_t cb_group_base = size_t(group_idx) * cb_group_stride;
- const float d_scalar = float(D[h_idx]);
+ const uint x_token_stride = x_strides[0];
+ const uint x_head_stride = x_strides[1];
+ const uint x_dim_stride = x_strides[2];
+ const uint dt_token_stride = dt_strides[0];
+ const uint dt_head_stride = dt_strides[1];
+ const uint cb_token_stride = cb_strides[0];
+ const uint cb_group_stride = cb_strides[1];
+ const uint cb_state_stride = cb_strides[2];
+ const uint state_head_stride = state_strides[0];
+ const uint state_dim_stride = state_strides[1];
+ const uint state_inner_stride = state_strides[2];
+
+ const uint x_base = h_idx * x_head_stride + dh_idx * x_dim_stride;
+ const uint dt_base = h_idx * dt_head_stride;
+ const uint state_base = h_idx * state_head_stride + dh_idx * state_dim_stride;
+ const uint cb_group_base = group_idx * cb_group_stride;
+ const float d_scalar = float(d[h_idx]);
const int max_chunks = SSM_PREFILL_MAX_STATE / simd_width;
const int chunk_count = (state_dim + int(simd_width) - 1) / int(simd_width);
@@ -233,17 +165,17 @@ void ssd_prefill_kernel(
#pragma unroll
for (int chunk = 0; chunk < chunk_count; ++chunk) {
const int idx = chunk * int(simd_width) + int(lane_idx);
- const size_t state_idx = state_base + size_t(idx) * state_inner_stride;
+ const uint state_idx = state_base + idx * state_inner_stride;
lane_states[chunk] = float(state[state_idx]);
}
- for (size_t token = 0; token < suffix_len; ++token) {
- const size_t x_idx = token * x_token_stride + x_base;
- const size_t dt_idx = token * dt_token_stride + dt_base;
+ for (uint token = 0; token < suffix_len; ++token) {
+ const uint x_idx = token * x_token_stride + x_base;
+ const uint dt_idx = token * dt_token_stride + dt_base;
const float x_val = float(x[x_idx]);
const float decay_val = fast::exp(-float(softplus(float(dt_raw[dt_idx]))));
- const float gate = float(SILU{}(z[x_idx]));
+ const float gate = float(apply_silu(z[x_idx]));
const float skip = d_scalar * x_val;
const float dt_scaled_input = x_val;
@@ -251,13 +183,12 @@ void ssd_prefill_kernel(
#pragma unroll
for (int chunk = 0; chunk < chunk_count; ++chunk) {
const int idx = chunk * int(simd_width) + int(lane_idx);
- const size_t cb_idx = cb_group_base + size_t(idx) * cb_state_stride +
+ const uint cb_idx = cb_group_base + uint(idx) * cb_state_stride +
token * cb_token_stride;
-
const float new_state =
- decay_val * lane_states[chunk] + dt_scaled_input * float(B[cb_idx]);
+ decay_val * lane_states[chunk] + dt_scaled_input * float(b[cb_idx]);
lane_states[chunk] = new_state;
- contrib_sum += new_state * float(C[cb_idx]);
+ contrib_sum += new_state * float(c[cb_idx]);
}
float dot = simd_sum(contrib_sum);
@@ -269,141 +200,78 @@ void ssd_prefill_kernel(
#pragma unroll
for (int chunk = 0; chunk < chunk_count; ++chunk) {
const int idx = chunk * int(simd_width) + int(lane_idx);
- const size_t state_idx = state_base + size_t(idx) * state_inner_stride;
+ const uint state_idx = state_base + uint(idx) * state_inner_stride;
state[state_idx] = static_cast<T>(lane_states[chunk]);
}
}
-#define instantiate_ssd_prefill_kernel(type_name, type) \
- template [[host_name("ssd_prefill_kernel_" #type_name)]] \
- kernel void ssd_prefill_kernel<type>( \
- device const type* x [[buffer(0)]], \
- device const type* dt_raw [[buffer(1)]], \
- device const type* B [[buffer(2)]], \
- device const type* C [[buffer(3)]], \
- device const type* D [[buffer(4)]], \
- device const type* z [[buffer(5)]], \
- device type* state [[buffer(6)]], \
- device type* y [[buffer(7)]], \
- constant const size_t& suffix_len [[buffer(8)]], \
- constant const int& group_size [[buffer(9)]], \
- constant const int& state_size [[buffer(10)]], \
- constant const size_t* x_strides [[buffer(11)]], \
- constant const size_t* dt_strides [[buffer(12)]], \
- constant const size_t* cb_strides [[buffer(13)]], \
- constant const size_t* state_strides [[buffer(14)]], \
- constant const uint& num_heads [[buffer(15)]], \
- constant const uint& head_dim [[buffer(16)]], \
- uint3 tg_pos [[threadgroup_position_in_grid]], \
- ushort lane [[thread_index_in_threadgroup]] \
- );
-
-instantiate_ssd_prefill_kernel(float, float);
-instantiate_ssd_prefill_kernel(bfloat, bfloat);
-instantiate_ssd_prefill_kernel(half, half);
-
-#undef instantiate_ssd_prefill_kernel
-
template <typename T>
-[[kernel, max_total_threads_per_threadgroup(1024)]]
-void ssd_prefill_kernel_sequential(
- device const T* x [[buffer(0)]], // (suffix, h, dh)
- device const T* dt_raw [[buffer(1)]], // (suffix, h) - raw dt values
- device const T* B [[buffer(2)]], // (suffix, g, n)
- device const T* C [[buffer(3)]], // (suffix, g, n)
- device const T* D [[buffer(4)]], // (h)
- device const T* z [[buffer(5)]], // (suffix, h, dh)
- device T* state [[buffer(6)]], // (h, dh, n)
- device T* y [[buffer(7)]], // (suffix, h, dh)
- constant const size_t& suffix_len [[buffer(8)]],
- constant const int& group_size [[buffer(9)]],
- constant const int& state_size [[buffer(10)]],
- constant const size_t* x_strides [[buffer(11)]],
- constant const size_t* dt_strides [[buffer(12)]],
- constant const size_t* cb_strides [[buffer(13)]],
- constant const size_t* state_strides [[buffer(14)]],
- uint3 tid [[thread_position_in_grid]],
- uint3 grid_dim [[threads_per_grid]]
+VARIANTS(T, float, half, bfloat)
+KERNEL(SSDPrefillSequential)(
+ device const T* x, // (suffix, h, dh)
+ device const T* dt_raw, // (suffix, h) - raw dt values
+ device const T* b, // (suffix, g, n)
+ device const T* c, // (suffix, g, n)
+ device const T* d, // (h)
+ device const T* z, // (suffix, h, dh)
+ device T* state, // (h, dh, n)
+ device T* y, // (suffix, h, dh)
+ constant const uint& suffix_len,
+ constant const int& group_size,
+ constant const int& state_size,
+ constant const uint* x_strides,
+ constant const uint* dt_strides,
+ constant const uint* cb_strides,
+ constant const uint* state_strides,
+ constant const uint& channels,
+ constant const uint& head_dim,
+ const uint h_idx AXIS(channels, 32),
+ const uint dh_idx AXIS(head_dim, 32)
) {
- const uint h_idx = tid.x;
- const uint dh_idx = tid.y;
- if (h_idx >= grid_dim.x || dh_idx >= grid_dim.y) {
- return;
- }
-
const uint safe_group = uint(max(group_size, 1));
const uint group_idx = h_idx / safe_group;
device T* state_row = state + size_t(h_idx) * state_strides[0] +
size_t(dh_idx) * state_strides[1];
- for (size_t token = 0; token < suffix_len; ++token) {
- const size_t x_idx = token * x_strides[0] + size_t(h_idx) * x_strides[1] +
- size_t(dh_idx) * x_strides[2];
- const size_t dt_idx = token * dt_strides[0] + size_t(h_idx) * dt_strides[1];
- const size_t cb_base = token * cb_strides[0] + group_idx * cb_strides[1];
+ for (uint token = 0; token < suffix_len; ++token) {
+ const uint x_idx = token * x_strides[0] + h_idx * x_strides[1] + dh_idx * x_strides[2];
+ const uint dt_idx = token * dt_strides[0] + h_idx * dt_strides[1];
+ const uint cb_base = token * cb_strides[0] + group_idx * cb_strides[1];
const T this_x = x[x_idx];
const T dt_raw_val = dt_raw[dt_idx];
const T this_dt = softplus(dt_raw_val);
const T this_decay = static_cast<T>(fast::exp(-float(this_dt)));
- const T this_D = D[h_idx];
- const T this_z = SILU{}(z[x_idx]);
+ const T this_D = d[h_idx];
+ const T this_z = apply_silu(z[x_idx]);
const T dt_scaled_input = this_x;
T acc = T(0);
int s = 0;
const int vec_bound = (state_size / 4) * 4;
for (; s < vec_bound; s += 4) {
- const size_t state_idx = s * state_strides[2];
- const size_t cb_idx = cb_base + s * cb_strides[2];
+ const uint state_idx = s * state_strides[2];
+ const uint cb_idx = cb_base + s * cb_strides[2];
auto prev_state =
*reinterpret_cast<device vec<T, 4>*>(state_row + state_idx);
- auto b_vec = *reinterpret_cast<device const vec<T, 4>*>(B + cb_idx);
- auto c_vec = *reinterpret_cast<device const vec<T, 4>*>(C + cb_idx);
+ auto b_vec = *reinterpret_cast<device const vec<T, 4>*>(b + cb_idx);
+ auto c_vec = *reinterpret_cast<device const vec<T, 4>*>(c + cb_idx);
vec<T, 4> new_state = prev_state * this_decay + b_vec * dt_scaled_input;
*reinterpret_cast<device vec<T, 4>*>(state_row + state_idx) = new_state;
vec<T, 4> prod = new_state * c_vec;
acc += prod.x + prod.y + prod.z + prod.w;
}
for (; s < state_size; ++s) {
- const size_t state_idx = s * state_strides[2];
+ const uint state_idx = s * state_strides[2];
const T prev_state = state_row[state_idx];
- const size_t cb_idx = cb_base + s * cb_strides[2];
- const T new_state = prev_state * this_decay + B[cb_idx] * dt_scaled_input;
+ const uint cb_idx = cb_base + s * cb_strides[2];
+ const T new_state = prev_state * this_decay + b[cb_idx] * dt_scaled_input;
state_row[state_idx] = new_state;
- acc += new_state * C[cb_idx];
+ acc += new_state * c[cb_idx];
}
acc += this_D * this_x;
acc *= this_z;
y[x_idx] = acc;
}
-}
-
-#define instantiate_ssd_prefill_kernel_sequential(type_name, type) \
- template [[host_name("ssd_prefill_kernel_sequential_" #type_name)]] \
- kernel void ssd_prefill_kernel_sequential<type>( \
- device const type* x [[buffer(0)]], \
- device const type* dt_raw [[buffer(1)]], \
- device const type* B [[buffer(2)]], \
- device const type* C [[buffer(3)]], \
- device const type* D [[buffer(4)]], \
- device const type* z [[buffer(5)]], \
- device type* state [[buffer(6)]], \
- device type* y [[buffer(7)]], \
- constant const size_t& suffix_len [[buffer(8)]], \
- constant const int& group_size [[buffer(9)]], \
- constant const int& state_size [[buffer(10)]], \
- constant const size_t* x_strides [[buffer(11)]], \
- constant const size_t* dt_strides [[buffer(12)]], \
- constant const size_t* cb_strides [[buffer(13)]], \
- constant const size_t* state_strides [[buffer(14)]], \
- uint3 tid [[thread_position_in_grid]], \
- uint3 grid_dim [[threads_per_grid]] \
- );
-
-instantiate_ssd_prefill_kernel_sequential(float, float);
-instantiate_ssd_prefill_kernel_sequential(bfloat, bfloat);
-instantiate_ssd_prefill_kernel_sequential(half, half);
-
-#undef instantiate_ssd_prefill_kernel_sequential
+}
\ No newline at end of file
diff --git a/crates/uzu/src/backends/metal/kernel/ssm/ssd_prefill.rs b/crates/uzu/src/backends/metal/kernel/ssm/ssd_prefill.rs
index e8a82884f..ec0e9dd57 100644
--- a/crates/uzu/src/backends/metal/kernel/ssm/ssd_prefill.rs
+++ b/crates/uzu/src/backends/metal/kernel/ssm/ssd_prefill.rs
@@ -1,11 +1,18 @@
-use super::{SSMKernelError, fn_suffix};
-use crate::backends::metal::{
- KernelDataType, MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext,
- MTLSize, ProtocolObject, Retained,
- metal_extensions::ComputeEncoderSetValue,
+use crate::backends::{
+ common::kernel::{
+ SSDPrefill64Kernel as _, SSDPrefillKernel as _,
+ SSDPrefillSequentialKernel as _,
+ },
+ metal::{
+ KernelDataType, MTLBuffer, MTLComputeCommandEncoder, MTLContext,
+ MTLError, ProtocolObject,
+ kernel::dsl::{
+ SSDPrefill64MetalKernel, SSDPrefillMetalKernel,
+ SSDPrefillSequentialMetalKernel,
+ },
+ },
};
-
-const SSD_PREFILL_SINGLE_THREADS: usize = 32;
+use objc2::Message;
#[derive(Copy, Clone, Debug, PartialEq, Eq)]
pub enum SSDPrefillMode {
@@ -13,12 +20,6 @@ pub enum SSDPrefillMode {
SinglePass,
}
-pub struct SSDPrefillKernel {
- sequential: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- single_pass: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
- single_pass_64: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
pub struct SSDPrefillArguments<'a> {
pub x: &'a ProtocolObject<dyn MTLBuffer>,
pub dt: &'a ProtocolObject<dyn MTLBuffer>, // raw dt values
@@ -39,30 +40,26 @@ pub struct SSDPrefillArguments<'a> {
pub head_dim: usize,
}
-impl SSDPrefillKernel {
+pub struct SSDPrefillKernels {
+ single: SSDPrefillMetalKernel,
+ single_64: SSDPrefill64MetalKernel,
+ sequential: SSDPrefillSequentialMetalKernel,
+}
+
+impl SSDPrefillKernels {
pub fn new(
context: &MTLContext,
data_type: KernelDataType,
- ) -> Result<Self, SSMKernelError> {
- let sequential_name =
- format!("ssd_prefill_kernel_sequential_{}", fn_suffix(data_type));
- let single_name =
- format!("ssd_prefill_kernel_{}", fn_suffix(data_type));
- let single_64_name =
- format!("ssd_prefill_kernel_64_{}", fn_suffix(data_type));
- let sequential = context
- .compute_pipeline_state(&sequential_name, None)
- .map_err(SSMKernelError::MetalError)?;
- let single_pass = context
- .compute_pipeline_state(&single_name, None)
- .map_err(SSMKernelError::MetalError)?;
- let single_pass_64 = context
- .compute_pipeline_state(&single_64_name, None)
- .map_err(SSMKernelError::MetalError)?;
+ ) -> Result<Self, MTLError> {
+ let single = SSDPrefillMetalKernel::new(context, data_type.into())?;
+ let single_64 =
+ SSDPrefill64MetalKernel::new(context, data_type.into())?;
+ let sequential =
+ SSDPrefillSequentialMetalKernel::new(context, data_type.into())?;
Ok(Self {
+ single,
+ single_64,
sequential,
- single_pass,
- single_pass_64,
})
}
@@ -71,95 +68,120 @@ impl SSDPrefillKernel {
compute_encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
args: SSDPrefillArguments,
mode: SSDPrefillMode,
- ) -> Result<(), SSMKernelError> {
- match mode {
- SSDPrefillMode::Sequential => {
- self.encode_sequential(compute_encoder, &args)
- },
- SSDPrefillMode::SinglePass => {
- self.encode_single(compute_encoder, &args)
- },
- }
- }
-
- fn encode_sequential(
- &self,
- compute_encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- args: &SSDPrefillArguments,
- ) -> Result<(), SSMKernelError> {
- if args.channels == 0 || args.head_dim == 0 || args.suffix_len == 0 {
- return Ok(());
- }
- compute_encoder.set_compute_pipeline_state(&self.sequential);
- self.bind_common_buffers(compute_encoder, args);
- let threads_per_threadgroup = MTLSize {
- width: 32,
- height: 32,
- depth: 1,
- };
- let total_threads = MTLSize {
- width: args.channels,
- height: args.head_dim,
- depth: 1,
- };
- compute_encoder
- .dispatch_threads(total_threads, threads_per_threadgroup);
- Ok(())
- }
-
- fn encode_single(
- &self,
- compute_encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- args: &SSDPrefillArguments,
- ) -> Result<(), SSMKernelError> {
- if args.channels == 0 || args.head_dim == 0 || args.suffix_len == 0 {
- return Ok(());
- }
- if args.state_size == 64 {
- compute_encoder.set_compute_pipeline_state(&self.single_pass_64);
- } else {
- compute_encoder.set_compute_pipeline_state(&self.single_pass);
- }
- self.bind_common_buffers(compute_encoder, args);
- let channels = args.channels as u32;
- let head_dim = args.head_dim as u32;
- compute_encoder.set_value(&channels, 15);
- compute_encoder.set_value(&head_dim, 16);
- let threads_per_threadgroup = MTLSize {
- width: SSD_PREFILL_SINGLE_THREADS,
- height: 1,
- depth: 1,
- };
- let pair_count = args.channels * args.head_dim;
- let total_threads = MTLSize {
- width: pair_count * SSD_PREFILL_SINGLE_THREADS,
- height: 1,
- depth: 1,
- };
- compute_encoder
- .dispatch_threads(total_threads, threads_per_threadgroup);
- Ok(())
- }
-
- fn bind_common_buffers(
- &self,
- compute_encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- args: &SSDPrefillArguments,
) {
- compute_encoder.set_buffer(Some(args.x), 0, 0);
- compute_encoder.set_buffer(Some(args.dt), 0, 1);
- compute_encoder.set_buffer(Some(args.b), 0, 2);
- compute_encoder.set_buffer(Some(args.c), 0, 3);
- compute_encoder.set_buffer(Some(args.d), 0, 4);
- compute_encoder.set_buffer(Some(args.z), 0, 5);
- compute_encoder.set_buffer(Some(args.state), 0, 6);
- compute_encoder.set_buffer(Some(args.y), 0, 7);
- compute_encoder.set_value(&args.suffix_len, 8);
- compute_encoder.set_value(&args.group_size, 9);
- compute_encoder.set_value(&args.state_size, 10);
- compute_encoder.set_value(&args.x_strides, 11);
- compute_encoder.set_value(&args.dt_strides, 12);
- compute_encoder.set_value(&args.cb_strides, 13);
- compute_encoder.set_value(&args.state_strides, 14);
+ if mode == SSDPrefillMode::SinglePass {
+ if args.state_size == 64 {
+ self.single_64.encode(
+ &args.x.retain(),
+ &args.dt.retain(),
+ &args.b.retain(),
+ &args.c.retain(),
+ &args.d.retain(),
+ &args.z.retain(),
+ &args.state.retain(),
+ &args.y.retain(),
+ args.suffix_len as u32,
+ args.group_size,
+ args.state_size,
+ args.x_strides
+ .iter()
+ .map(|x| *x as u32)
+ .collect::<Vec<_>>()
+ .as_slice(),
+ args.dt_strides
+ .iter()
+ .map(|x| *x as u32)
+ .collect::<Vec<_>>()
+ .as_slice(),
+ args.cb_strides
+ .iter()
+ .map(|x| *x as u32)
+ .collect::<Vec<_>>()
+ .as_slice(),
+ args.state_strides
+ .iter()
+ .map(|x| *x as u32)
+ .collect::<Vec<_>>()
+ .as_slice(),
+ args.channels as u32,
+ args.head_dim as u32,
+ compute_encoder,
+ )
+ } else {
+ self.single.encode(
+ &args.x.retain(),
+ &args.dt.retain(),
+ &args.b.retain(),
+ &args.c.retain(),
+ &args.d.retain(),
+ &args.z.retain(),
+ &args.state.retain(),
+ &args.y.retain(),
+ args.suffix_len as u32,
+ args.group_size,
+ args.state_size,
+ args.x_strides
+ .iter()
+ .map(|x| *x as u32)
+ .collect::<Vec<_>>()
+ .as_slice(),
+ args.dt_strides
+ .iter()
+ .map(|x| *x as u32)
+ .collect::<Vec<_>>()
+ .as_slice(),
+ args.cb_strides
+ .iter()
+ .map(|x| *x as u32)
+ .collect::<Vec<_>>()
+ .as_slice(),
+ args.state_strides
+ .iter()
+ .map(|x| *x as u32)
+ .collect::<Vec<_>>()
+ .as_slice(),
+ args.channels as u32,
+ args.head_dim as u32,
+ compute_encoder,
+ )
+ }
+ } else if mode == SSDPrefillMode::Sequential {
+ self.sequential.encode(
+ &args.x.retain(),
+ &args.dt.retain(),
+ &args.b.retain(),
+ &args.c.retain(),
+ &args.d.retain(),
+ &args.z.retain(),
+ &args.state.retain(),
+ &args.y.retain(),
+ args.suffix_len as u32,
+ args.group_size,
+ args.state_size,
+ args.x_strides
+ .iter()
+ .map(|x| *x as u32)
+ .collect::<Vec<_>>()
+ .as_slice(),
+ args.dt_strides
+ .iter()
+ .map(|x| *x as u32)
+ .collect::<Vec<_>>()
+ .as_slice(),
+ args.cb_strides
+ .iter()
+ .map(|x| *x as u32)
+ .collect::<Vec<_>>()
+ .as_slice(),
+ args.state_strides
+ .iter()
+ .map(|x| *x as u32)
+ .collect::<Vec<_>>()
+ .as_slice(),
+ args.channels as u32,
+ args.head_dim as u32,
+ compute_encoder,
+ )
+ }
}
}
diff --git a/crates/uzu/src/backends/metal/kernel/ssm/ssd_update.metal b/crates/uzu/src/backends/metal/kernel/ssm/ssd_update.metal
index 10ce2519b..43eeed1ab 100644
--- a/crates/uzu/src/backends/metal/kernel/ssm/ssd_update.metal
+++ b/crates/uzu/src/backends/metal/kernel/ssm/ssd_update.metal
@@ -1,116 +1,64 @@
#include <metal_stdlib>
#include "../definitions.metal"
+#include "ssm_common.h"
using namespace metal;
-struct SILU {
- template <typename T>
- T operator()(T x) {
- float xf = float(x);
- float y = 1.0f / (1.0f + fast::exp(-fabs(xf)));
- float out = (xf < 0.0f) ? (1.0f - y) * xf : y * xf;
- return static_cast<T>(out);
- }
-};
-
-template <typename T>
-inline T softplus(T x) {
- float xf = float(x);
- if (xf > 20.0f) {
- return x;
- }
- return static_cast<T>(log(1.0f + fast::exp(xf)));
-}
-
template <typename T>
-[[kernel, max_total_threads_per_threadgroup(1024)]]
-void ssd_update_kernel(
+VARIANTS(T, float, half, bfloat)
+KERNEL(SSDUpdate)(
// Input
- device const T* x [[buffer(0)]], // (b, h, dh)
- device const T* dt_raw [[buffer(1)]], // (b, h)
- device const T* B [[buffer(2)]], // (b, g, n)
- device const T* C [[buffer(3)]], // (b, g, n)
- device const T* D [[buffer(4)]], // (h)
- device const T* z [[buffer(5)]], // (b, d)
- device const T* state [[buffer(6)]], // (b, h, dh, n)
- device T* y [[buffer(7)]],
- device T* next_state [[buffer(8)]],
+ device const T* x, // (b, h, dh)
+ device const T* dt_raw, // (b, h)
+ device const T* b, // (b, g, n)
+ device const T* c, // (b, g, n)
+ device const T* d, // (h)
+ device const T* z, // (b, d)
+ device const T* state, // (b, h, dh, n)
+ device T* y,
+ device T* next_state,
// Parameters
- constant const int& group_size [[buffer(9)]], // h = group_size * g
- constant const int& state_size [[buffer(10)]],
+ constant const uint& group_size, // h = group_size * g
+ constant const uint& state_size,
// Strides
- constant const size_t* x_strides [[buffer(11)]],
- constant const size_t* dt_strides [[buffer(12)]],
- constant const size_t* CB_strides [[buffer(13)]],
- constant const size_t* state_strides [[buffer(14)]],
+ constant const uint x_strides[3],
+ constant const uint dt_strides[2],
+ constant const uint cb_strides[3],
+ constant const uint state_strides[4],
+ // Threads count
+ constant const uint& b_size,
+ constant const uint& h_size,
+ constant const uint& dh_size,
// Grid
- uint3 grid_idx [[thread_position_in_grid]],
- uint3 grid_size [[threads_per_grid]]
+ const uint b_idx AXIS(b_size, 32),
+ const uint h_idx AXIS(h_size, 32),
+ const uint dh_idx AXIS(dh_size, 1)
) {
- const int b_idx = grid_idx.x;
- const int h_idx = grid_idx.y;
- const int dh_idx = grid_idx.z;
-
- const int CB_start_idx = static_cast<int>(
- b_idx * CB_strides[0] + (h_idx / group_size) * CB_strides[1]
- );
- const int x_idx = static_cast<int>(
- b_idx * x_strides[0] + h_idx * x_strides[1] + dh_idx * x_strides[2]
- );
- const int dt_idx =
- static_cast<int>(b_idx * dt_strides[0] + h_idx * dt_strides[1]);
- const int state_start_idx = static_cast<int>(
- b_idx * state_strides[0] + h_idx * state_strides[1] +
- dh_idx * state_strides[2]
- );
+ const uint cb_start_idx = b_idx * cb_strides[0] + (h_idx / group_size) * cb_strides[1];
+ const uint x_idx = b_idx * x_strides[0] + h_idx * x_strides[1] + dh_idx * x_strides[2];
+ const uint dt_idx = b_idx * dt_strides[0] + h_idx * dt_strides[1];
+ const uint state_start_idx = b_idx * state_strides[0] + h_idx * state_strides[1] + dh_idx * state_strides[2];
// load data
T this_x = x[x_idx];
T dt_raw_val = dt_raw[dt_idx];
T this_dt = softplus(dt_raw_val);
T this_decay = static_cast<T>(fast::exp(-float(this_dt)));
- T this_D = D[h_idx];
+ T this_d = d[h_idx];
T this_z = z[x_idx];
- this_z = SILU{}(this_z);
+ this_z = apply_silu(this_z);
T dt_scaled_input = this_x;
T temp = T(0);
#pragma unroll
- for (int i = 0; i < state_size; ++i) {
- int CB_idx = CB_start_idx + i;
- int state_idx = state_start_idx + i;
- T this_new_state =
- state[state_idx] * this_decay + B[CB_idx] * dt_scaled_input;
+ for (uint i = 0; i < state_size; ++i) {
+ uint cb_idx = cb_start_idx + i;
+ uint state_idx = state_start_idx + i;
+ T this_new_state = state[state_idx] * this_decay + b[cb_idx] * dt_scaled_input;
next_state[state_idx] = this_new_state;
- temp = temp + this_new_state * C[CB_idx];
+ temp = temp + this_new_state * c[cb_idx];
}
- temp = temp + this_D * this_x;
+ temp = temp + this_d * this_x;
temp = temp * this_z;
y[x_idx] = temp;
-}
-
-#define instantiate_ssd_update_kernel(type_name, type) \
- template [[host_name("ssd_update_kernel_" #type_name)]] \
- kernel void ssd_update_kernel<type>( \
- device const type* x [[buffer(0)]], \
- device const type* dt_raw [[buffer(1)]], \
- device const type* B [[buffer(2)]], \
- device const type* C [[buffer(3)]], \
- device const type* D [[buffer(4)]], \
- device const type* z [[buffer(5)]], \
- device const type* state [[buffer(6)]], \
- device type* y [[buffer(7)]], \
- device type* next_state [[buffer(8)]], \
- constant const int& group_size [[buffer(9)]], \
- constant const int& state_size [[buffer(10)]], \
- constant const size_t* x_strides [[buffer(11)]], \
- constant const size_t* dt_strides [[buffer(12)]], \
- constant const size_t* CB_strides [[buffer(13)]], \
- constant const size_t* state_strides [[buffer(14)]], \
- uint3 grid_idx [[thread_position_in_grid]], \
- uint3 grid_size [[threads_per_grid]] \
- );
-
-instantiate_ssd_update_kernel(float, float);
-instantiate_ssd_update_kernel(bfloat, bfloat);
-instantiate_ssd_update_kernel(half, half);
\ No newline at end of file
+}
\ No newline at end of file
diff --git a/crates/uzu/src/backends/metal/kernel/ssm/ssd_update.rs b/crates/uzu/src/backends/metal/kernel/ssm/ssd_update.rs
deleted file mode 100644
index cfa5cd0cf..000000000
--- a/crates/uzu/src/backends/metal/kernel/ssm/ssd_update.rs
+++ /dev/null
@@ -1,85 +0,0 @@
-use super::{SSMKernelError, fn_suffix};
-use crate::backends::metal::{
- KernelDataType, MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext,
- MTLSize, ProtocolObject, Retained,
- metal_extensions::ComputeEncoderSetValue,
-};
-
-pub struct SSDUpdateKernel {
- pipeline: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-pub struct SSDUpdateArguments<'a> {
- pub x: &'a ProtocolObject<dyn MTLBuffer>, // buffer(0) (b, h, dh)
- pub dt: &'a ProtocolObject<dyn MTLBuffer>, // buffer(1) (b, h) - raw dt values
- pub b: &'a ProtocolObject<dyn MTLBuffer>, // buffer(2) (b, g, n)
- pub c: &'a ProtocolObject<dyn MTLBuffer>, // buffer(3) (b, g, n)
- pub d: &'a ProtocolObject<dyn MTLBuffer>, // buffer(4) (h)
- pub z: &'a ProtocolObject<dyn MTLBuffer>, // buffer(5) (b, d)
- pub state: &'a ProtocolObject<dyn MTLBuffer>, // buffer(6) (b, h, dh, n)
- pub y: &'a ProtocolObject<dyn MTLBuffer>, // buffer(7)
- pub next_state: &'a ProtocolObject<dyn MTLBuffer>, // buffer(8)
- pub group_size: i32, // buffer(9)
- pub state_size: i32, // buffer(10)
- pub x_strides: [usize; 3], // buffer(11)
- pub dt_strides: [usize; 2], // buffer(12)
- pub cb_strides: [usize; 3], // buffer(13)
- pub state_strides: [usize; 4], // buffer(14)
- pub b_size: usize,
- pub h_size: usize,
- pub dh_size: usize,
-}
-
-impl SSDUpdateKernel {
- pub fn new(
- context: &MTLContext,
- data_type: KernelDataType,
- ) -> Result<Self, SSMKernelError> {
- let fn_name = format!("ssd_update_kernel_{}", fn_suffix(data_type));
- let pipeline = context
- .compute_pipeline_state(&fn_name, None)
- .map_err(SSMKernelError::MetalError)?;
- Ok(Self {
- pipeline,
- })
- }
-
- pub fn encode(
- &self,
- compute_encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- args: SSDUpdateArguments,
- ) -> Result<(), SSMKernelError> {
- compute_encoder.set_compute_pipeline_state(&self.pipeline);
-
- compute_encoder.set_buffer(Some(args.x), 0, 0);
- compute_encoder.set_buffer(Some(args.dt), 0, 1);
- compute_encoder.set_buffer(Some(args.b), 0, 2);
- compute_encoder.set_buffer(Some(args.c), 0, 3);
- compute_encoder.set_buffer(Some(args.d), 0, 4);
- compute_encoder.set_buffer(Some(args.z), 0, 5);
- compute_encoder.set_buffer(Some(args.state), 0, 6);
- compute_encoder.set_buffer(Some(args.y), 0, 7);
- compute_encoder.set_buffer(Some(args.next_state), 0, 8);
-
- compute_encoder.set_value(&args.group_size, 9);
- compute_encoder.set_value(&args.state_size, 10);
- compute_encoder.set_value(&args.x_strides, 11);
- compute_encoder.set_value(&args.dt_strides, 12);
- compute_encoder.set_value(&args.cb_strides, 13);
- compute_encoder.set_value(&args.state_strides, 14);
-
- let threads_per_threadgroup = MTLSize {
- width: 32,
- height: 32,
- depth: 1,
- };
- let total_threads = MTLSize {
- width: args.b_size,
- height: args.h_size,
- depth: args.dh_size,
- };
- compute_encoder
- .dispatch_threads(total_threads, threads_per_threadgroup);
- Ok(())
- }
-}
diff --git a/crates/uzu/src/backends/metal/kernel/ssm/ssm_common.h b/crates/uzu/src/backends/metal/kernel/ssm/ssm_common.h
new file mode 100644
index 000000000..d96d207a9
--- /dev/null
+++ b/crates/uzu/src/backends/metal/kernel/ssm/ssm_common.h
@@ -0,0 +1,16 @@
+template <typename T>
+inline T apply_silu(T x) {
+ float xf = float(x);
+ float y = 1.0f / (1.0f + fast::exp(-fabs(xf)));
+ float out = (xf < 0.0f) ? (1.0f - y) * xf : y * xf;
+ return static_cast<T>(out);
+}
+
+template <typename T>
+inline T softplus(T x) {
+ float xf = float(x);
+ if (xf > 20.0f) {
+ return x;
+ }
+ return static_cast<T>(log(1.0f + fast::exp(xf)));
+}
\ No newline at end of file
diff --git a/crates/uzu/src/backends/metal/kernel/tensor_add_bias/mod.rs b/crates/uzu/src/backends/metal/kernel/tensor_add_bias/mod.rs
deleted file mode 100644
index 42fb9d130..000000000
--- a/crates/uzu/src/backends/metal/kernel/tensor_add_bias/mod.rs
+++ /dev/null
@@ -1,82 +0,0 @@
-use metal::MTLCommandEncoderExt;
-
-use crate::backends::metal::{
- KernelDataType, MTLBuffer, MTLCommandBuffer, MTLCommandEncoder,
- MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext, MTLError,
- ProtocolObject, Retained,
- metal_extensions::{
- ComputeEncoderConditional, ComputeEncoderDispatch,
- ComputeEncoderSetValue,
- },
-};
-
-#[derive(Debug)]
-pub struct TensorAddBias {
- pipeline_state: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-impl TensorAddBias {
- pub fn new(
- context: &MTLContext,
- data_type: KernelDataType,
- ) -> Result<Self, MTLError> {
- let function_name =
- format!("tensorAddBias_{}", data_type.function_name_suffix());
- let pipeline_state =
- context.compute_pipeline_state(&function_name, None)?;
- Ok(Self {
- pipeline_state,
- })
- }
-
- pub fn encode_into_command_buffer(
- &self,
- input: &ProtocolObject<dyn MTLBuffer>,
- bias: &ProtocolObject<dyn MTLBuffer>,
- output: &ProtocolObject<dyn MTLBuffer>,
- num_cols: usize,
- total_len: usize,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- predicate: Option<&ProtocolObject<dyn MTLBuffer>>,
- ) {
- let encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- self.encode_with_encoder(
- input, bias, output, num_cols, total_len, &encoder, predicate,
- );
- encoder.end_encoding();
- }
-
- pub fn encode_with_encoder(
- &self,
- input: &ProtocolObject<dyn MTLBuffer>,
- bias: &ProtocolObject<dyn MTLBuffer>,
- output: &ProtocolObject<dyn MTLBuffer>,
- num_cols: usize,
- total_len: usize,
- encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- predicate: Option<&ProtocolObject<dyn MTLBuffer>>,
- ) {
- encoder.condition(
- predicate,
- || {
- encoder.set_label(Some("Tensor Add Bias"));
- encoder.set_compute_pipeline_state(&self.pipeline_state);
- encoder.set_buffer(Some(input), 0, 0);
- encoder.set_buffer(Some(bias), 0, 1);
- encoder.set_buffer(Some(output), 0, 2);
- let num_cols_i32 = num_cols as i32;
- let total_len_i32 = total_len as i32;
- encoder.set_value(&num_cols_i32, 3);
- encoder.set_value(&total_len_i32, 4);
- encoder.dispatch_1d_exactly(
- &self.pipeline_state,
- total_len,
- None,
- );
- },
- None::<fn()>,
- );
- }
-}
diff --git a/crates/uzu/src/backends/metal/kernel/tensor_add_bias/tensor_add_bias.metal b/crates/uzu/src/backends/metal/kernel/tensor_add_bias/tensor_add_bias.metal
index b1b791801..de789eb92 100644
--- a/crates/uzu/src/backends/metal/kernel/tensor_add_bias/tensor_add_bias.metal
+++ b/crates/uzu/src/backends/metal/kernel/tensor_add_bias/tensor_add_bias.metal
@@ -2,31 +2,15 @@
#include "../definitions.metal"
template <typename T>
-void tensorAddBias(
+VARIANTS(T, float, half, bfloat)
+KERNEL(TensorAddBias)(
const device T* input,
const device T* bias,
device T* output,
- constant int& numCols,
- constant int& length,
- const uint position
+ constant uint& num_cols,
+ constant uint& length,
+ const uint position AXIS(length, 32)
) {
- if (position < length) {
- int col = position % numCols;
- output[position] = input[position] + bias[col];
- }
-}
-
-#define outerArguments(T) \
- (const device T* input [[buffer(0)]], \
- const device T* bias [[buffer(1)]], \
- device T* output [[buffer(2)]], \
- constant int& numCols [[buffer(3)]], \
- constant int& length [[buffer(4)]], \
- const uint position [[thread_position_in_grid]])
-
-#define innerArguments (input, bias, output, numCols, length, position)
-
-generateKernels(32, tensorAddBias)
-
-#undef outerArguments
-#undef innerArguments
+ uint col = position % num_cols;
+ output[position] = input[position] + bias[col];
+}
\ No newline at end of file
diff --git a/crates/uzu/src/backends/metal/kernel/tensor_add_swap/mod.rs b/crates/uzu/src/backends/metal/kernel/tensor_add_swap/mod.rs
deleted file mode 100644
index 9f9632989..000000000
--- a/crates/uzu/src/backends/metal/kernel/tensor_add_swap/mod.rs
+++ /dev/null
@@ -1,78 +0,0 @@
-use std::{mem::size_of, ptr::NonNull};
-
-use objc2::msg_send;
-use objc2_foundation::NSString;
-
-use crate::backends::metal::{
- KernelDataType, MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- MTLComputePipelineState, MTLContext, MTLError, ProtocolObject, Retained,
- metal_extensions::ComputeEncoderDispatch,
-};
-
-#[derive(Debug)]
-pub struct TensorAddSwapKernel {
- pipeline_state: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-impl TensorAddSwapKernel {
- pub fn new(
- context: &MTLContext,
- data_type: KernelDataType,
- ) -> Result<Self, MTLError> {
- let function_name =
- format!("tensorAddSwap_{}", data_type.function_name_suffix());
-
- let pipeline_state = context.compute_pipeline_state(&function_name, None)?;
-
- Ok(Self {
- pipeline_state,
- })
- }
-
- pub fn encode_into_command_buffer(
- &self,
- skip_buffer: &ProtocolObject<dyn MTLBuffer>,
- main_buffer: &ProtocolObject<dyn MTLBuffer>,
- length: usize,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- ) {
- let compute_encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- self.encode_with_encoder(
- skip_buffer,
- main_buffer,
- length,
- &compute_encoder,
- );
- compute_encoder.end_encoding();
- }
-
- pub fn encode_with_encoder(
- &self,
- skip_buffer: &ProtocolObject<dyn MTLBuffer>,
- main_buffer: &ProtocolObject<dyn MTLBuffer>,
- length: usize,
- compute_encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- ) {
- unsafe {
- let label = NSString::from_str("Tensor Add-Swap");
- let _: () = msg_send![compute_encoder, setLabel: &*label];
- }
-
- compute_encoder.set_buffer(Some(skip_buffer), 0, 0);
- compute_encoder.set_buffer(Some(main_buffer), 0, 1);
-
- unsafe {
- compute_encoder.set_bytes(
- NonNull::new_unchecked(
- &(length as i32) as *const i32 as *mut std::ffi::c_void,
- ),
- size_of::<i32>(),
- 2,
- );
- }
-
- compute_encoder.dispatch_1d_exactly(&self.pipeline_state, length, None);
- }
-}
diff --git a/crates/uzu/src/backends/metal/kernel/tensor_add_swap/tensor_add_swap.metal b/crates/uzu/src/backends/metal/kernel/tensor_add_swap/tensor_add_swap.metal
index cfc551b1f..deb5f2de9 100644
--- a/crates/uzu/src/backends/metal/kernel/tensor_add_swap/tensor_add_swap.metal
+++ b/crates/uzu/src/backends/metal/kernel/tensor_add_swap/tensor_add_swap.metal
@@ -2,31 +2,17 @@
#include "../definitions.metal"
template <typename T>
-void tensorAddSwap(
- device T* skipBuffer,
- device T* mainBuffer,
- constant int& length,
- const uint position
+VARIANTS(T, float, half, bfloat)
+KERNEL(TensorAddSwap)(
+ device T* skip_buffer,
+ device T* main_buffer,
+ constant uint& length,
+ const uint position AXIS(length, 32)
) {
- if (position < length) {
- T skipValue = skipBuffer[position];
- T mainValue = mainBuffer[position];
- T result = mainValue + skipValue;
+ T skip_value = skip_buffer[position];
+ T main_value = main_buffer[position];
+ T result = main_value + skip_value;
- mainBuffer[position] = result;
- skipBuffer[position] = result;
- }
-}
-
-#define outerArguments(T) \
- (device T * skipBuffer [[buffer(0)]], \
- device T * mainBuffer [[buffer(1)]], \
- constant int& length [[buffer(2)]], \
- const uint position [[thread_position_in_grid]])
-
-#define innerArguments (skipBuffer, mainBuffer, length, position)
-
-generateKernels(32, tensorAddSwap)
-
-#undef outerArguments
-#undef innerArguments
\ No newline at end of file
+ main_buffer[position] = result;
+ skip_buffer[position] = result;
+}
\ No newline at end of file
diff --git a/crates/uzu/src/backends/metal/kernel/tensor_copy/mod.rs b/crates/uzu/src/backends/metal/kernel/tensor_copy/mod.rs
deleted file mode 100644
index fc83519c7..000000000
--- a/crates/uzu/src/backends/metal/kernel/tensor_copy/mod.rs
+++ /dev/null
@@ -1,76 +0,0 @@
-use std::{mem::size_of, ptr::NonNull};
-
-use objc2::msg_send;
-use objc2_foundation::NSString;
-
-use crate::backends::metal::{
- KernelDataType, MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLComputeCommandEncoder,
- MTLComputePipelineState, MTLContext, MTLError, ProtocolObject, Retained,
- metal_extensions::ComputeEncoderDispatch,
-};
-
-pub struct TensorCopyKernel {
- pipeline_state: Retained<ProtocolObject<dyn MTLComputePipelineState>>,
-}
-
-impl TensorCopyKernel {
- pub fn new(
- context: &MTLContext,
- data_type: KernelDataType,
- ) -> Result<Self, MTLError> {
- let function_name =
- format!("tensorCopy_{}", data_type.function_name_suffix());
-
- let pipeline_state = context.compute_pipeline_state(&function_name, None)?;
-
- Ok(Self {
- pipeline_state,
- })
- }
-
- pub fn encode_into_command_buffer(
- &self,
- source_buffer: &ProtocolObject<dyn MTLBuffer>,
- destination_buffer: &ProtocolObject<dyn MTLBuffer>,
- length: usize,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
- ) {
- let compute_encoder = command_buffer
- .new_compute_command_encoder()
- .expect("Failed to create compute command encoder");
- self.encode_with_encoder(
- source_buffer,
- destination_buffer,
- length,
- &compute_encoder,
- );
- compute_encoder.end_encoding();
- }
-
- pub fn encode_with_encoder(
- &self,
- source_buffer: &ProtocolObject<dyn MTLBuffer>,
- destination_buffer: &ProtocolObject<dyn MTLBuffer>,
- length: usize,
- compute_encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
- ) {
- unsafe {
- let label = NSString::from_str("Tensor Copy");
- let _: () = msg_send![compute_encoder, setLabel: &*label];
- }
-
- compute_encoder.set_buffer(Some(source_buffer), 0, 0);
- compute_encoder.set_buffer(Some(destination_buffer), 0, 1);
- unsafe {
- compute_encoder.set_bytes(
- NonNull::new_unchecked(
- &(length as i32) as *const i32 as *mut std::ffi::c_void,
- ),
- size_of::<i32>(),
- 2,
- );
- }
-
- compute_encoder.dispatch_1d_exactly(&self.pipeline_state, length, None);
- }
-}
diff --git a/crates/uzu/src/backends/metal/kernel/tensor_copy/tensor_copy.metal b/crates/uzu/src/backends/metal/kernel/tensor_copy/tensor_copy.metal
index a6e11cf6e..1bc45bf28 100644
--- a/crates/uzu/src/backends/metal/kernel/tensor_copy/tensor_copy.metal
+++ b/crates/uzu/src/backends/metal/kernel/tensor_copy/tensor_copy.metal
@@ -2,26 +2,12 @@
#include "../definitions.metal"
template <typename T>
-void tensorCopy(
- const device T* sourceBuffer,
- device T* destinationBuffer,
- constant int& length,
- const uint position
+VARIANTS(T, float, half, bfloat)
+KERNEL(TensorCopy)(
+ const device T* src_buffer,
+ device T* dst_buffer,
+ constant uint& length,
+ const uint position AXIS(length, 32)
) {
- if (position < length) {
- destinationBuffer[position] = sourceBuffer[position];
- }
-}
-
-#define outerArguments(T) \
- (const device T* sourceBuffer [[buffer(0)]], \
- device T* destinationBuffer [[buffer(1)]], \
- constant int& length [[buffer(2)]], \
- const uint position [[thread_position_in_grid]])
-
-#define innerArguments (sourceBuffer, destinationBuffer, length, position)
-
-generateKernels(32, tensorCopy)
-
-#undef outerArguments
-#undef innerArguments
\ No newline at end of file
+ dst_buffer[position] = src_buffer[position];
+}
\ No newline at end of file
diff --git a/crates/uzu/src/backends/metal/kernel/token_copy/mod.rs b/crates/uzu/src/backends/metal/kernel/token_copy/mod.rs
index 427de7c83..d64232315 100644
--- a/crates/uzu/src/backends/metal/kernel/token_copy/mod.rs
+++ b/crates/uzu/src/backends/metal/kernel/token_copy/mod.rs
@@ -1,6 +1,6 @@
use crate::backends::metal::{
- MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext, MTLError, MTLSize,
- ProtocolObject, Retained,
+ MTLBuffer, MTLComputeCommandEncoder, MTLComputePipelineState, MTLContext,
+ MTLError, MTLSize, ProtocolObject, Retained,
};
/// Kernel for copying sampled tokens in async pipeline.
@@ -14,8 +14,10 @@ pub struct TokenCopyKernel {
impl TokenCopyKernel {
pub fn new(context: &MTLContext) -> Result<Self, MTLError> {
- let copy_to_token_ids = context.compute_pipeline_state("copy_sampled_token", None)?;
- let copy_to_results = context.compute_pipeline_state("copy_token_to_results", None)?;
+ let copy_to_token_ids =
+ context.compute_pipeline_state("copy_sampled_token", None)?;
+ let copy_to_results =
+ context.compute_pipeline_state("copy_token_to_results", None)?;
Ok(Self {
copy_to_token_ids,
@@ -27,8 +29,8 @@ impl TokenCopyKernel {
/// For next forward pass to read the token.
pub fn encode_to_token_ids(
&self,
- sampling_output: &ProtocolObject<dyn MTLBuffer>,
- token_ids: &ProtocolObject<dyn MTLBuffer>,
+ sampling_output: &Retained<ProtocolObject<dyn MTLBuffer>>,
+ token_ids: &Retained<ProtocolObject<dyn MTLBuffer>>,
encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
) {
encoder.set_compute_pipeline_state(&self.copy_to_token_ids);
@@ -41,8 +43,8 @@ impl TokenCopyKernel {
/// For callback to read without race condition.
pub fn encode_to_results(
&self,
- sampling_output: &ProtocolObject<dyn MTLBuffer>,
- results: &ProtocolObject<dyn MTLBuffer>,
+ sampling_output: &Retained<ProtocolObject<dyn MTLBuffer>>,
+ results: &Retained<ProtocolObject<dyn MTLBuffer>>,
pass_idx: usize,
encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
) {
diff --git a/crates/uzu/src/backends/metal/language_model_generator_context.rs b/crates/uzu/src/backends/metal/language_model_generator_context.rs
index 0a38a46cd..a90bc5051 100644
--- a/crates/uzu/src/backends/metal/language_model_generator_context.rs
+++ b/crates/uzu/src/backends/metal/language_model_generator_context.rs
@@ -6,23 +6,26 @@ use std::{
rc::Rc,
};
-use crate::backends::metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandQueue, MTLDevice, MTLDeviceExt, MTLEvent,
- MTLResourceOptions, ProtocolObject, Retained,
-};
-
use super::{
- CacheLayers, Decoder, KVCacheUpdate, KernelDataType, MTLContext,
+ CacheLayers, Decoder, KVCacheUpdate, KernelDataType, MTLContext, Metal,
ModelShape,
compilation_parameters::CompilationConfig,
- encodable_block::Sampling,
forward_pass::{ScratchBuffers, SharedBuffers},
- kernel::{MaskUpdateKernel, TokenCopyKernel},
+ kernel::TokenCopyKernel,
};
use crate::{
DataType,
+ backends::{
+ common::{Context, kernel::MaskUpdateKernel},
+ metal::{
+ MTLBuffer, MTLCommandBuffer, MTLCommandQueue, MTLDeviceExt,
+ MTLEvent, ProtocolObject, Retained,
+ kernel::dsl::MaskUpdateMetalKernel,
+ },
+ },
config::{DecoderConfig, LanguageModelConfig, ModelMetadata},
- language_model::rng::DerivableSeed,
+ encodable_block::Sampling,
+ language_model::rng::PRng,
parameters::ParameterLoader,
session::{
config::DecodingConfig,
@@ -55,23 +58,20 @@ pub struct AsyncBuffers {
impl AsyncBuffers {
pub fn new(
- device: &Retained<ProtocolObject<dyn MTLDevice>>,
+ context: &MTLContext,
max_tokens: usize,
batch_size: usize,
) -> Self {
- let positions = device.new_buffer(
- max_tokens * std::mem::size_of::<i32>() ,
- MTLResourceOptions::STORAGE_MODE_SHARED,
- ).expect("Failed to create positions buffer");
- let seeds = device.new_buffer(
- max_tokens * std::mem::size_of::<u64>() ,
- MTLResourceOptions::STORAGE_MODE_SHARED,
- ).expect("Failed to create seeds buffer");
- let results = device.new_buffer(
- batch_size * std::mem::size_of::<u32>() ,
- MTLResourceOptions::STORAGE_MODE_SHARED,
- ).expect("Failed to create results buffer");
- let event = device.new_event().expect("Failed to create event");
+ let positions = context
+ .create_buffer(max_tokens * std::mem::size_of::<i32>())
+ .expect("Failed to create positions buffer");
+ let seeds = context
+ .create_buffer(max_tokens * std::mem::size_of::<u64>())
+ .expect("Failed to create seeds buffer");
+ let results = context
+ .create_buffer(batch_size * std::mem::size_of::<u32>())
+ .expect("Failed to create results buffer");
+ let event = context.device.new_event().expect("Failed to create event");
Self {
positions,
@@ -104,13 +104,14 @@ impl AsyncBuffers {
/// Prepare seeds buffer with deterministic sequence
pub fn prepare_seeds(
&self,
- seed_source: &mut DerivableSeed,
+ seed: &PRng,
+ prefix_len: usize,
tokens_to_generate: usize,
) {
let ptr = self.seeds.contents().as_ptr() as *mut u64;
for i in 0..tokens_to_generate {
unsafe {
- *ptr.add(i) = seed_source.next();
+ *ptr.add(i) = seed.derive((prefix_len + i - 1) as u64);
}
}
}
@@ -136,20 +137,20 @@ pub struct LanguageModelGeneratorContext {
pub cache_layers: Rc<RefCell<CacheLayers>>,
pub shared_buffers: Rc<RefCell<SharedBuffers>>,
- pub scratch_buffers: ScratchBuffers<Rc<MTLContext>>,
+ pub scratch_buffers: ScratchBuffers<MTLContext>,
pub model_config: LanguageModelConfig,
pub decoder_config: Rc<DecoderConfig>,
pub model_shape: ModelShape,
pub executables: Decoder,
pub kv_cache_update: Box<KVCacheUpdate>,
- pub gpu_sampler: Sampling,
- pub next_seed: DerivableSeed,
+ pub gpu_sampler: Sampling<Metal>,
+ pub seed: PRng,
/// Kernel for copying sampled tokens in async pipeline
pub token_copy: TokenCopyKernel,
/// Kernel for updating attention mask between async passes
- pub mask_update: Option<MaskUpdateKernel>,
+ pub mask_update: Option<MaskUpdateMetalKernel>,
/// Pre-allocated buffers for async generation
pub async_buffers: AsyncBuffers,
}
@@ -159,11 +160,13 @@ impl LanguageModelGeneratorContext {
model_path: &Path,
decoding_config: &DecodingConfig,
) -> Result<Self, Error> {
- let mtl_device: Retained<ProtocolObject<dyn MTLDevice>> = <dyn metal::MTLDevice>::system_default().ok_or(Error::UnableToCreateMetalContext)?;
- let mtl_command_queue =
- mtl_device.new_command_queue_with_max_command_buffer_count(1024).ok_or(Error::UnableToCreateMetalContext)?;
+ let context =
+ MTLContext::new().map_err(|_| Error::UnableToCreateMetalContext)?;
- let command_buffer = mtl_command_queue.command_buffer().expect("Failed to create command buffer").to_owned();
+ let command_buffer = context
+ .create_command_buffer()
+ .expect("Failed to create command buffer")
+ .to_owned();
let config_path = model_path.join("config.json");
if !config_path.exists() {
@@ -196,11 +199,6 @@ impl LanguageModelGeneratorContext {
let max_suffix_length: usize =
std::cmp::max(prefill_step_size, generate_suffix_length);
- let mtl_context = Rc::new(
- MTLContext::new(mtl_device, mtl_command_queue)
- .map_err(|_| Error::UnableToCreateMetalContext)?,
- );
-
let compilation_config = Rc::new(CompilationConfig::default());
let weights_path = model_path.join("model.safetensors");
if !weights_path.exists() {
@@ -208,19 +206,19 @@ impl LanguageModelGeneratorContext {
}
let weights_file = File::open(&weights_path)
.map_err(|_| Error::UnableToLoadWeights)?;
- let loader = ParameterLoader::new(&weights_file, &mtl_context)
+ let loader = ParameterLoader::new(&weights_file, context.as_ref())
.map_err(|_| Error::UnableToLoadWeights)?;
let root_loader_view = loader.tree();
let shared_buffers = Rc::new(RefCell::new(SharedBuffers::new(
- &mtl_context,
+ &context,
&decoder_config,
&model_shape,
)));
shared_buffers.borrow_mut().update_data(&root_loader_view);
let scratch_buffers = ScratchBuffers::new(
- &mtl_context,
+ context.as_ref(),
&decoder_config,
&model_shape,
max_prefix_length,
@@ -228,14 +226,14 @@ impl LanguageModelGeneratorContext {
);
let executables = Decoder::new(
- mtl_context.clone(),
+ context.clone(),
decoder_config.clone(),
&root_loader_view,
compilation_config.clone(),
);
let cache_layers = Rc::new(RefCell::new(CacheLayers::new(
- &mtl_context,
+ &context,
&model_shape,
max_prefix_length,
max_suffix_length,
@@ -245,29 +243,25 @@ impl LanguageModelGeneratorContext {
decoder_config.output_norm_config.scale_precision.into();
let kernel_data_type: KernelDataType = intermediate_data_type.into();
let kv_cache_update = Box::new(
- KVCacheUpdate::new(
- &mtl_context,
- kernel_data_type,
- max_prefix_length,
- )
- .map_err(|_| Error::UnableToCreateMetalContext)?,
+ KVCacheUpdate::new(&context, kernel_data_type, max_prefix_length)
+ .map_err(|_| Error::UnableToCreateMetalContext)?,
);
- let gpu_sampler = Sampling::new(
- &mtl_context,
- kernel_data_type,
+ let gpu_sampler = Sampling::<Metal>::new(
+ &context,
+ intermediate_data_type,
max_suffix_length,
decoder_config.vocab_size,
)
.map_err(|_| Error::UnableToCreateMetalContext)?;
- let token_copy = TokenCopyKernel::new(&mtl_context)
+ let token_copy = TokenCopyKernel::new(&context)
.map_err(|_| Error::UnableToCreateMetalContext)?;
// Create mask update kernel if model has attention layers
let mask_update = if decoder_config.has_attention_layers() {
Some(
- MaskUpdateKernel::new(&mtl_context, kernel_data_type)
+ MaskUpdateMetalKernel::new(&context, kernel_data_type.into())
.map_err(|_| Error::UnableToCreateMetalContext)?,
)
} else {
@@ -276,17 +270,13 @@ impl LanguageModelGeneratorContext {
let async_batch_size =
decoding_config.async_batch_size.resolve(model_path);
- let async_buffers = AsyncBuffers::new(
- &mtl_context.device,
- max_prefix_length,
- async_batch_size,
- );
+ let async_buffers =
+ AsyncBuffers::new(&context, max_prefix_length, async_batch_size);
- let base_seed = decoding_config.sampling_seed.resolve();
- let next_seed = DerivableSeed::new(base_seed);
+ let seed = PRng::new(decoding_config.sampling_seed.resolve());
let context = Self {
- mtl_context,
+ mtl_context: context,
command_buffer,
cache_layers,
shared_buffers,
@@ -297,7 +287,7 @@ impl LanguageModelGeneratorContext {
executables,
kv_cache_update,
gpu_sampler,
- next_seed,
+ seed,
token_copy,
mask_update,
async_buffers,
@@ -307,7 +297,11 @@ impl LanguageModelGeneratorContext {
}
pub fn reset_command_buffer(&mut self) {
- self.command_buffer =
- self.mtl_context.command_queue.command_buffer().expect("Failed to create command buffer").to_owned();
+ self.command_buffer = self
+ .mtl_context
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer")
+ .to_owned();
}
}
diff --git a/crates/uzu/src/backends/metal/media_utils.rs b/crates/uzu/src/backends/metal/media_utils.rs
index dd5b353bf..6761abc4c 100644
--- a/crates/uzu/src/backends/metal/media_utils.rs
+++ b/crates/uzu/src/backends/metal/media_utils.rs
@@ -3,8 +3,8 @@ use std::{cell::RefCell, mem, rc::Rc};
use crate::{
DataType,
backends::metal::{
- MTLCommandBuffer, MTLContext, MTLDevice, MTLDeviceExt, MTLResourceOptions, MetalArray,
- ProtocolObject, Retained,
+ MTLCommandBuffer, MTLContext, MTLDevice, MTLDeviceExt,
+ MTLResourceOptions, MetalArray, ProtocolObject, Retained,
error::MTLError,
image::{Image, PixelFormat, TextureUsage},
kernel::media_kernels::{
@@ -147,7 +147,7 @@ impl MetalImagePreprocessor {
pub unsafe fn encode_preprocessing(
&self,
- command_buffer: &ProtocolObject<dyn MTLCommandBuffer>,
+ command_buffer: &Retained<ProtocolObject<dyn MTLCommandBuffer>>,
input_image: &Image,
params: &ImagePreprocessingParams,
requirements: &ImagePreprocessingRequirements,
@@ -253,7 +253,7 @@ impl MetalImagePreprocessor {
};
self.extract_patches_kernel.encode_internal(
intermediate_texture,
- unsafe { final_output_buffer_array_ref.mtl_buffer() },
+ final_output_buffer_array_ref.buffer(),
&patch_params as *const _ as *const std::ffi::c_void,
command_buffer,
);
@@ -269,10 +269,11 @@ fn data_to_mtl_tensor<T: Copy>(
data_type: DataType,
) -> Result<MetalArray, MTLError> {
let size = mem::size_of::<T>();
- let bytes = unsafe { std::slice::from_raw_parts(data as *const T as *const u8, size) };
- let buffer = device.new_buffer_with_data(
- bytes,
- MTLResourceOptions::STORAGE_MODE_SHARED,
- ).expect("Failed to create buffer");
- unsafe { Ok(MetalArray::new(buffer, &shape, data_type)) }
+ let bytes = unsafe {
+ std::slice::from_raw_parts(data as *const T as *const u8, size)
+ };
+ let buffer = device
+ .new_buffer_with_data(bytes, MTLResourceOptions::STORAGE_MODE_SHARED)
+ .expect("Failed to create buffer");
+ unsafe { Ok(MetalArray::from_parts(buffer, 0, &shape, data_type)) }
}
diff --git a/crates/uzu/src/backends/metal/metal_extensions/command_buffer_timing.rs b/crates/uzu/src/backends/metal/metal_extensions/command_buffer_timing.rs
index 623408739..d3e2df9b3 100644
--- a/crates/uzu/src/backends/metal/metal_extensions/command_buffer_timing.rs
+++ b/crates/uzu/src/backends/metal/metal_extensions/command_buffer_timing.rs
@@ -1,4 +1,6 @@
-use crate::backends::metal::{MTLCommandBuffer, MTLCommandBufferExt, ProtocolObject};
+use crate::backends::metal::{
+ MTLCommandBuffer, MTLCommandBufferExt, ProtocolObject,
+};
/// Extension trait providing convenience timing methods for command buffers.
pub trait CommandBufferTimingExt {
diff --git a/crates/uzu/src/backends/metal/metal_extensions/compute_command_encoder_extensions_conditional.rs b/crates/uzu/src/backends/metal/metal_extensions/compute_command_encoder_extensions_conditional.rs
index e1c5a9319..e3035cb59 100644
--- a/crates/uzu/src/backends/metal/metal_extensions/compute_command_encoder_extensions_conditional.rs
+++ b/crates/uzu/src/backends/metal/metal_extensions/compute_command_encoder_extensions_conditional.rs
@@ -2,6 +2,7 @@ use objc2::msg_send;
use crate::backends::metal::{
MTLBuffer, MTLCompareFunction, MTLComputeCommandEncoder, ProtocolObject,
+ Retained,
};
/// Low-level, unsafe conditional control of Metal encoders.
@@ -24,7 +25,7 @@ trait ComputeEncoderRawConditional {
pub trait ComputeEncoderConditional {
fn condition<IfBlock, ElseBlock>(
&self,
- predicate: Option<&ProtocolObject<dyn MTLBuffer>>,
+ predicate: Option<&Retained<ProtocolObject<dyn MTLBuffer>>>,
if_block: IfBlock,
else_block: Option<ElseBlock>,
) where
@@ -69,7 +70,7 @@ where
{
fn condition<IfBlock, ElseBlock>(
&self,
- predicate: Option<&ProtocolObject<dyn MTLBuffer>>,
+ predicate: Option<&Retained<ProtocolObject<dyn MTLBuffer>>>,
if_block: IfBlock,
else_block: Option<ElseBlock>,
) where
diff --git a/crates/uzu/src/backends/metal/metal_extensions/compute_command_encoder_extensions_dispatch.rs b/crates/uzu/src/backends/metal/metal_extensions/compute_command_encoder_extensions_dispatch.rs
index 2f3c8e06e..5b4fb5d6c 100644
--- a/crates/uzu/src/backends/metal/metal_extensions/compute_command_encoder_extensions_dispatch.rs
+++ b/crates/uzu/src/backends/metal/metal_extensions/compute_command_encoder_extensions_dispatch.rs
@@ -3,8 +3,8 @@ use super::{
device_extensions_features::{DeviceFeatures, Feature},
};
use crate::backends::metal::{
- MTLCommandEncoderExt, MTLComputeCommandEncoder, MTLComputePipelineState, MTLSize,
- ProtocolObject,
+ MTLCommandEncoderExt, MTLComputeCommandEncoder, MTLComputePipelineState,
+ MTLSize, ProtocolObject,
};
/// Extensions for MTLComputeCommandEncoder to simplify dispatch operations
@@ -89,8 +89,8 @@ impl ComputeEncoderDispatch for ProtocolObject<dyn MTLComputeCommandEncoder> {
size: usize,
threadgroup_width: Option<usize>,
) {
- let tg_width =
- threadgroup_width.unwrap_or(state.thread_execution_width() as usize);
+ let tg_width = threadgroup_width
+ .unwrap_or(state.thread_execution_width() as usize);
let tg_size = MTLSize {
width: tg_width,
height: 1,
@@ -114,7 +114,8 @@ impl ComputeEncoderDispatch for ProtocolObject<dyn MTLComputeCommandEncoder> {
threadgroup_width: Option<usize>,
) {
let tg_size = MTLSize {
- width: threadgroup_width.unwrap_or(state.thread_execution_width() as usize),
+ width: threadgroup_width
+ .unwrap_or(state.thread_execution_width() as usize),
height: 1,
depth: 1,
};
@@ -151,7 +152,8 @@ impl ComputeEncoderDispatch for ProtocolObject<dyn MTLComputeCommandEncoder> {
size: MTLSize,
threadgroup_size: Option<MTLSize>,
) {
- let tg_size = threadgroup_size.unwrap_or_else(|| state.max_2d_threadgroup_size());
+ let tg_size =
+ threadgroup_size.unwrap_or_else(|| state.max_2d_threadgroup_size());
let count = MTLSize {
width: (size.width + tg_size.width - 1) / tg_size.width,
@@ -169,7 +171,8 @@ impl ComputeEncoderDispatch for ProtocolObject<dyn MTLComputeCommandEncoder> {
size: MTLSize,
threadgroup_size: Option<MTLSize>,
) {
- let tg_size = threadgroup_size.unwrap_or_else(|| state.max_2d_threadgroup_size());
+ let tg_size =
+ threadgroup_size.unwrap_or_else(|| state.max_2d_threadgroup_size());
self.set_compute_pipeline_state(state);
self.dispatch_threads(size, tg_size);
@@ -196,7 +199,8 @@ impl ComputeEncoderDispatch for ProtocolObject<dyn MTLComputeCommandEncoder> {
size: MTLSize,
threadgroup_size: Option<MTLSize>,
) {
- let tg_size = threadgroup_size.unwrap_or_else(|| state.max_2d_threadgroup_size());
+ let tg_size =
+ threadgroup_size.unwrap_or_else(|| state.max_2d_threadgroup_size());
let count = MTLSize {
width: (size.width + tg_size.width - 1) / tg_size.width,
@@ -214,7 +218,8 @@ impl ComputeEncoderDispatch for ProtocolObject<dyn MTLComputeCommandEncoder> {
size: MTLSize,
threadgroup_size: Option<MTLSize>,
) {
- let tg_size = threadgroup_size.unwrap_or_else(|| state.max_2d_threadgroup_size());
+ let tg_size =
+ threadgroup_size.unwrap_or_else(|| state.max_2d_threadgroup_size());
self.set_compute_pipeline_state(state);
self.dispatch_threads(size, tg_size);
diff --git a/crates/uzu/src/backends/metal/metal_extensions/compute_command_encoder_extensions_set_value.rs b/crates/uzu/src/backends/metal/metal_extensions/compute_command_encoder_extensions_set_value.rs
index a875bd082..a80e4b30a 100644
--- a/crates/uzu/src/backends/metal/metal_extensions/compute_command_encoder_extensions_set_value.rs
+++ b/crates/uzu/src/backends/metal/metal_extensions/compute_command_encoder_extensions_set_value.rs
@@ -5,20 +5,36 @@ use crate::backends::metal::{MTLComputeCommandEncoder, ProtocolObject};
/// Extension trait providing ergonomic `set_value` methods for compute command encoders.
pub trait ComputeEncoderSetValue {
/// Sets a value at the specified buffer index.
- fn set_value<T>(&self, value: &T, index: usize);
+ fn set_value<T>(
+ &self,
+ value: &T,
+ index: usize,
+ );
/// Sets a slice of values at the specified buffer index.
- fn set_slice<T>(&self, slice: &[T], index: usize);
+ fn set_slice<T>(
+ &self,
+ slice: &[T],
+ index: usize,
+ );
}
impl ComputeEncoderSetValue for ProtocolObject<dyn MTLComputeCommandEncoder> {
- fn set_value<T>(&self, value: &T, index: usize) {
+ fn set_value<T>(
+ &self,
+ value: &T,
+ index: usize,
+ ) {
let ptr = NonNull::new(value as *const T as *mut c_void)
.expect("value pointer should never be null");
self.set_bytes(ptr, size_of::<T>(), index);
}
- fn set_slice<T>(&self, slice: &[T], index: usize) {
+ fn set_slice<T>(
+ &self,
+ slice: &[T],
+ index: usize,
+ ) {
let ptr = NonNull::new(slice.as_ptr() as *mut c_void)
.expect("slice pointer should never be null");
self.set_bytes(ptr, size_of::<T>() * slice.len(), index);
diff --git a/crates/uzu/src/backends/metal/metal_extensions/library_extensions_pipeline.rs b/crates/uzu/src/backends/metal/metal_extensions/library_extensions_pipeline.rs
index 8c72ae1e3..3ddf8c424 100644
--- a/crates/uzu/src/backends/metal/metal_extensions/library_extensions_pipeline.rs
+++ b/crates/uzu/src/backends/metal/metal_extensions/library_extensions_pipeline.rs
@@ -2,7 +2,8 @@ use objc2::{rc::Retained, runtime::ProtocolObject};
use objc2_foundation::NSError;
use crate::backends::metal::{
- MTLComputePipelineState, MTLDeviceExt, MTLFunctionConstantValues, MTLLibrary, MTLLibraryExt,
+ MTLComputePipelineState, MTLDeviceExt, MTLFunctionConstantValues,
+ MTLLibrary, MTLLibraryExt,
error::{LibraryError, MTLError},
};
@@ -22,7 +23,8 @@ impl LibraryPipelineExtensions for ProtocolObject<dyn MTLLibrary> {
&self,
function_name: &str,
constants: Option<&MTLFunctionConstantValues>,
- ) -> Result<Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError> {
+ ) -> Result<Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError>
+ {
let function = match constants {
Some(const_values) => {
let mut error: *mut NSError = std::ptr::null_mut();
@@ -31,21 +33,21 @@ impl LibraryPipelineExtensions for ProtocolObject<dyn MTLLibrary> {
const_values,
&mut error,
)
- }
+ },
None => self.new_function_with_name(function_name),
}
.ok_or(MTLError::Library(LibraryError::FunctionCreationFailed))?;
let device = self.device();
- device
- .new_compute_pipeline_state_with_function(&function)
- .map_err(|e| {
+ device.new_compute_pipeline_state_with_function(&function).map_err(
+ |e| {
MTLError::Library(LibraryError::Custom(format!(
"Pipeline state creation failed: {}",
e
)))
- })
+ },
+ )
}
}
@@ -54,7 +56,8 @@ impl LibraryPipelineExtensions for Retained<ProtocolObject<dyn MTLLibrary>> {
&self,
function_name: &str,
constants: Option<&MTLFunctionConstantValues>,
- ) -> Result<Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError> {
+ ) -> Result<Retained<ProtocolObject<dyn MTLComputePipelineState>>, MTLError>
+ {
(**self).compute_pipeline_state(function_name, constants)
}
}
diff --git a/crates/uzu/src/backends/metal/mod.rs b/crates/uzu/src/backends/metal/mod.rs
index 5b408e671..f4c45e5ca 100644
--- a/crates/uzu/src/backends/metal/mod.rs
+++ b/crates/uzu/src/backends/metal/mod.rs
@@ -1,6 +1,8 @@
+mod allocator;
mod array;
-mod buffer_allocator;
+mod backend;
mod classifier_context;
+mod command_buffer;
pub mod compilation_parameters;
mod context;
pub mod encodable_block;
@@ -13,12 +15,8 @@ pub mod media_utils;
pub mod metal_extensions;
pub mod placement_analysis;
pub mod utils;
-
-pub use metal::prelude::*;
-
pub use array::MetalArray;
-pub use metal_extensions::ComputeEncoderSetValue;
-pub use buffer_allocator::BufferAllocator;
+pub use backend::Metal;
pub use classifier_context::ClassifierContext;
pub use context::{
DeviceArchitecture, DeviceClass, DeviceGeneration, MTLContext,
@@ -26,10 +24,14 @@ pub use context::{
pub use encodable_block::Decoder;
pub use error::MTLError;
pub use forward_pass::{CacheLayers, ForwardPassState, ModelShape};
-pub use kernel::{KVCacheUpdate, KernelDataType, RopeKernel};
+pub use kernel::{
+ KVCacheUpdate, KernelDataType, RopeKernel, dsl::MetalKernels,
+};
pub use language_model_generator_context::LanguageModelGeneratorContext;
pub use media_utils::{
ImagePreprocessingParams, ImagePreprocessingRequirements,
MetalImagePreprocessor,
};
+pub use metal::prelude::*;
+pub use metal_extensions::ComputeEncoderSetValue;
pub use placement_analysis::PlacementAnalysis;
diff --git a/crates/uzu/src/backends/mod.rs b/crates/uzu/src/backends/mod.rs
index a73513c34..d69e0ae69 100644
--- a/crates/uzu/src/backends/mod.rs
+++ b/crates/uzu/src/backends/mod.rs
@@ -1,2 +1,5 @@
+pub mod common;
+
pub mod cpu;
+#[cfg(feature = "metal")]
pub mod metal;
diff --git a/crates/uzu/src/classifier/classifier.rs b/crates/uzu/src/classifier/classifier.rs
index 8d42f7a56..094325df3 100644
--- a/crates/uzu/src/classifier/classifier.rs
+++ b/crates/uzu/src/classifier/classifier.rs
@@ -8,10 +8,12 @@ use objc2::rc::autoreleasepool;
use super::ActivationTrace;
use super::{ClassificationOutput, ClassificationStats, ClassifierContext};
use crate::{
- Array, DataType,
+ DataType,
backends::metal::{
MTLCommandBuffer,
- forward_pass::{ArrayId, EncodableBlock, EncodingParameters, ForwardPassState},
+ forward_pass::{
+ ArrayId, EncodableBlock, EncodingParameters, ForwardPassState,
+ },
},
session::types::Error,
};
@@ -246,9 +248,9 @@ impl Classifier {
let logits_array = logits_arrays[0].borrow();
let num_labels = self.context.model_config.model_config.num_labels;
- let buffer = Array::buffer(&*logits_array);
+ let buffer = logits_array.as_bytes();
- match Array::data_type(&*logits_array) {
+ match logits_array.data_type() {
DataType::F32 => {
let slice: &[f32] = bytemuck::cast_slice(buffer);
Ok(slice[..num_labels].into())
diff --git a/crates/uzu/src/data_type.rs b/crates/uzu/src/data_type.rs
new file mode 100644
index 000000000..e2943aafa
--- /dev/null
+++ b/crates/uzu/src/data_type.rs
@@ -0,0 +1,103 @@
+use bytemuck::Pod;
+use half::{bf16, f16};
+use num_traits::NumCast;
+
+use xgrammar::{DLDataType, DLDataTypeCode};
+
+#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Copy, Clone)]
+pub enum DataType {
+ // Floating point
+ BF16,
+ F16,
+ F32,
+ F64,
+ // Sub-byte integers
+ I4,
+ U4,
+ // Normal integers
+ I8,
+ U8,
+ I16,
+ U16,
+ I32,
+ U32,
+ I64,
+ U64,
+}
+
+impl DataType {
+ pub const fn size_in_bits(&self) -> usize {
+ match self {
+ DataType::I4 | DataType::U4 => 4,
+ DataType::I8 | DataType::U8 => 8,
+ DataType::I16 | DataType::U16 => 16,
+ DataType::BF16 | DataType::F16 => 16,
+ DataType::F32 | DataType::I32 | DataType::U32 => 32,
+ DataType::F64 | DataType::I64 | DataType::U64 => 64,
+ }
+ }
+
+ pub const fn size_in_bytes(&self) -> usize {
+ self.size_in_bits().div_ceil(8)
+ }
+
+ const fn dl_data_type_code(self) -> DLDataTypeCode {
+ match self {
+ DataType::BF16 => DLDataTypeCode::kDLBfloat,
+ DataType::F16 | DataType::F32 | DataType::F64 => {
+ DLDataTypeCode::kDLFloat
+ },
+ DataType::I4
+ | DataType::I8
+ | DataType::I16
+ | DataType::I32
+ | DataType::I64 => DLDataTypeCode::kDLInt,
+ DataType::U4
+ | DataType::U8
+ | DataType::U16
+ | DataType::U32
+ | DataType::U64 => DLDataTypeCode::kDLUInt,
+ }
+ }
+}
+
+impl From<DataType> for DLDataType {
+ fn from(data_type: DataType) -> Self {
+ Self {
+ code: data_type.dl_data_type_code() as u8,
+ bits: data_type.size_in_bits() as u8,
+ lanes: 1,
+ }
+ }
+}
+
+pub trait ArrayElement: NumCast + Pod {
+ fn data_type() -> DataType;
+}
+
+macro_rules! impl_array_element {
+ ($($type:ty => $variant:ident),+ $(,)?) => {
+ $(
+ impl ArrayElement for $type {
+ fn data_type() -> DataType {
+ DataType::$variant
+ }
+ }
+ )+
+ };
+}
+
+impl_array_element! {
+ f16 => F16,
+ bf16 => BF16,
+ f32 => F32,
+ f64 => F64,
+ i8 => I8,
+ u8 => U8,
+ i16 => I16,
+ u16 => U16,
+ i32 => I32,
+ u32 => U32,
+ i64 => I64,
+ u64 => U64,
+}
diff --git a/crates/uzu/src/device/array.rs b/crates/uzu/src/device/array.rs
deleted file mode 100644
index 48b45a053..000000000
--- a/crates/uzu/src/device/array.rs
+++ /dev/null
@@ -1,142 +0,0 @@
-use bytemuck;
-use ndarray::{ArrayView, ArrayViewMut, IxDyn};
-use thiserror::Error;
-
-use crate::{ArrayElement, DataType};
-
-#[derive(Error, Debug)]
-pub enum ArrayConversionError {
- #[error("Invalid data type, expected {expected:?}, actual {actual:?}")]
- InvalidDataType {
- expected: DataType,
- actual: DataType,
- },
- #[error("Array of shape {0:?} is not a scalar")]
- NotAScalar(Box<[usize]>),
-}
-
-/// Calculate the number of bytes required to store an array with a given shape and DataType.
-/// Takes into account alignment of packed sub-byte data types.
-pub fn array_size_in_bytes(
- shape: &[usize],
- data_type: DataType,
-) -> usize {
- if shape.is_empty() {
- return data_type.size_in_bytes();
- }
- let last_dim = *shape.last().unwrap();
- let bits_per_row = last_dim * data_type.size_in_bits();
- let padded_bytes_per_row = (bits_per_row + 7) / 8;
-
- let num_rows: usize = shape.iter().rev().skip(1).product();
-
- num_rows * padded_bytes_per_row
-}
-
-// Backend-agnostic n-dimensional array type.
-pub trait Array {
- type BackendBuffer;
-
- fn shape(&self) -> &[usize];
-
- fn num_elements(&self) -> usize {
- self.shape().iter().map(|&d| d as usize).product()
- }
-
- fn data_type(&self) -> DataType;
-
- fn label(&self) -> String;
-
- /// Returns a reference to the device buffer containing the array's data.
- fn buffer(&self) -> &[u8];
-
- /// Returns a mutable reference to the device buffer containing the array's data.
- fn buffer_mut(&mut self) -> &mut [u8];
-
- fn backend_buffer(&self) -> &Self::BackendBuffer;
-
- /// Returns the size of the array in memory.
- fn size_in_bytes(&self) -> usize {
- array_size_in_bytes(self.shape(), self.data_type())
- }
-
- /// Returns true if the array holds a single scalar value.
- /// Equivalent to `self.shape().is_empty()`.
- fn is_scalar(&self) -> bool {
- self.shape().is_empty()
- }
-
- /// Returns a reference to the scalar value contained in the array.
- /// Returns an error if the array is not a scalar or if the requested type does not match the array's data type.
- fn item<T: ArrayElement>(&self) -> Result<&T, ArrayConversionError> {
- if !self.is_scalar() {
- Err(ArrayConversionError::NotAScalar(self.shape().into()))
- } else {
- let casted_buffer: &[T] = bytemuck::cast_slice(self.buffer());
- assert_eq!(casted_buffer.len(), 1);
- Ok(&casted_buffer.first().unwrap())
- }
- }
-
- /// Returns a mutable reference to the scalar value contained in the array.
- /// Returns an error if the array is not a scalar or if the requested type does not match the array's data type.
- fn item_mut<T: ArrayElement>(
- &mut self
- ) -> Result<&mut T, ArrayConversionError> {
- if !self.is_scalar() {
- Err(ArrayConversionError::NotAScalar(self.shape().into()))
- } else {
- let casted_buffer: &mut [T] =
- bytemuck::cast_slice_mut(self.buffer_mut());
- assert_eq!(casted_buffer.len(), 1);
- let first = casted_buffer.first_mut();
- Ok(first.unwrap())
- }
- }
-
- /// Returns an typed slice reffering to the underlying buffer.
- /// Returns an error if the requested type does not match the array's data type.
- fn as_slice<T: ArrayElement>(&self) -> Result<&[T], ArrayConversionError> {
- if T::data_type() != self.data_type() {
- return Err(ArrayConversionError::InvalidDataType {
- expected: T::data_type(),
- actual: self.data_type(),
- });
- }
- Ok(bytemuck::cast_slice(self.buffer()))
- }
-
- /// Returns an mutable typed slice reffering to the underlying buffer.
- /// Returns an error if the requested type does not match the array's data type.
- fn as_slice_mut<T: ArrayElement>(
- &mut self
- ) -> Result<&mut [T], ArrayConversionError> {
- if T::data_type() != self.data_type() {
- return Err(ArrayConversionError::InvalidDataType {
- expected: T::data_type(),
- actual: self.data_type(),
- });
- }
- Ok(bytemuck::cast_slice_mut(self.buffer_mut()))
- }
-
- /// Returns an ndarray::ArrayView of the underlying buffer.
- /// Returns an error if the requested type does not match the array's data type.
- fn as_view<T: ArrayElement>(
- &self
- ) -> Result<ArrayView<'_, T, IxDyn>, ArrayConversionError> {
- Ok(ArrayView::from_shape(self.shape(), self.as_slice()?).unwrap())
- }
-
- /// Returns an ndarray::ArrayViewMut of the underlying buffer.
- /// Returns an error if the requested type does not match the array's data type.
- fn as_view_mut<T: ArrayElement>(
- &mut self
- ) -> Result<ArrayViewMut<'_, T, IxDyn>, ArrayConversionError> {
- Ok(ArrayViewMut::from_shape(
- self.shape().to_owned(),
- self.as_slice_mut()?,
- )
- .unwrap())
- }
-}
diff --git a/crates/uzu/src/device/data_type.rs b/crates/uzu/src/device/data_type.rs
deleted file mode 100644
index d0ba5eb78..000000000
--- a/crates/uzu/src/device/data_type.rs
+++ /dev/null
@@ -1,125 +0,0 @@
-use bytemuck::Pod;
-use half::{bf16, f16};
-use num_traits::NumCast;
-
-#[derive(Debug, PartialEq, Eq, PartialOrd, Ord, Copy, Clone)]
-pub enum DataType {
- // Floating point
- BF16,
- F16,
- F32,
- F64,
- // Sub-byte integers
- I4,
- U4,
- // Normal integers
- I8,
- U8,
- I16,
- U16,
- I32,
- U32,
- I64,
- U64,
-}
-
-impl DataType {
- pub fn size_in_bits(&self) -> usize {
- match self {
- DataType::BF16 => 16,
- DataType::F16 => 16,
- DataType::F32 => 32,
- DataType::F64 => 64,
- DataType::I4 => 4,
- DataType::U4 => 4,
- DataType::I8 => 8,
- DataType::U8 => 8,
- DataType::I16 => 16,
- DataType::U16 => 16,
- DataType::I32 => 32,
- DataType::U32 => 32,
- DataType::I64 => 64,
- DataType::U64 => 64,
- }
- }
-
- pub fn size_in_bytes(&self) -> usize {
- (self.size_in_bits() + 7) / 8 // Round up to nearest byte
- }
-}
-
-pub trait ArrayElement: NumCast + Pod {
- fn data_type() -> DataType;
-}
-
-impl ArrayElement for f16 {
- fn data_type() -> DataType {
- DataType::F16
- }
-}
-
-impl ArrayElement for bf16 {
- fn data_type() -> DataType {
- DataType::BF16
- }
-}
-
-impl ArrayElement for f32 {
- fn data_type() -> DataType {
- DataType::F32
- }
-}
-
-impl ArrayElement for f64 {
- fn data_type() -> DataType {
- DataType::F64
- }
-}
-
-impl ArrayElement for i8 {
- fn data_type() -> DataType {
- DataType::I8
- }
-}
-
-impl ArrayElement for u8 {
- fn data_type() -> DataType {
- DataType::U8
- }
-}
-
-impl ArrayElement for i16 {
- fn data_type() -> DataType {
- DataType::I16
- }
-}
-
-impl ArrayElement for u16 {
- fn data_type() -> DataType {
- DataType::U16
- }
-}
-
-impl ArrayElement for i32 {
- fn data_type() -> DataType {
- DataType::I32
- }
-}
-
-impl ArrayElement for u32 {
- fn data_type() -> DataType {
- DataType::U32
- }
-}
-
-impl ArrayElement for i64 {
- fn data_type() -> DataType {
- DataType::I64
- }
-}
-
-impl ArrayElement for u64 {
- fn data_type() -> DataType {
- DataType::U64
- }
-}
diff --git a/crates/uzu/src/device/data_type_ext.rs b/crates/uzu/src/device/data_type_ext.rs
deleted file mode 100644
index 8a41c85df..000000000
--- a/crates/uzu/src/device/data_type_ext.rs
+++ /dev/null
@@ -1,81 +0,0 @@
-use xgrammar::{DLDataType, DLDataTypeCode};
-
-use crate::DataType;
-
-// Standard conversion from crate DataType to DLDataType
-impl Into<DLDataType> for DataType {
- fn into(self) -> DLDataType {
- match self {
- DataType::BF16 => DLDataType {
- code: DLDataTypeCode::kDLBfloat as u8,
- bits: 16,
- lanes: 1,
- },
- DataType::F16 => DLDataType {
- code: DLDataTypeCode::kDLFloat as u8,
- bits: 16,
- lanes: 1,
- },
- DataType::F32 => DLDataType {
- code: DLDataTypeCode::kDLFloat as u8,
- bits: 32,
- lanes: 1,
- },
- DataType::F64 => DLDataType {
- code: DLDataTypeCode::kDLFloat as u8,
- bits: 64,
- lanes: 1,
- },
- DataType::I4 => DLDataType {
- code: DLDataTypeCode::kDLInt as u8,
- bits: 4,
- lanes: 1,
- },
- DataType::U4 => DLDataType {
- code: DLDataTypeCode::kDLUInt as u8,
- bits: 4,
- lanes: 1,
- },
- DataType::I8 => DLDataType {
- code: DLDataTypeCode::kDLInt as u8,
- bits: 8,
- lanes: 1,
- },
- DataType::U8 => DLDataType {
- code: DLDataTypeCode::kDLUInt as u8,
- bits: 8,
- lanes: 1,
- },
- DataType::I16 => DLDataType {
- code: DLDataTypeCode::kDLInt as u8,
- bits: 16,
- lanes: 1,
- },
- DataType::U16 => DLDataType {
- code: DLDataTypeCode::kDLUInt as u8,
- bits: 16,
- lanes: 1,
- },
- DataType::I32 => DLDataType {
- code: DLDataTypeCode::kDLInt as u8,
- bits: 32,
- lanes: 1,
- },
- DataType::U32 => DLDataType {
- code: DLDataTypeCode::kDLUInt as u8,
- bits: 32,
- lanes: 1,
- },
- DataType::I64 => DLDataType {
- code: DLDataTypeCode::kDLInt as u8,
- bits: 64,
- lanes: 1,
- },
- DataType::U64 => DLDataType {
- code: DLDataTypeCode::kDLUInt as u8,
- bits: 64,
- lanes: 1,
- },
- }
- }
-}
diff --git a/crates/uzu/src/device/device_context.rs b/crates/uzu/src/device/device_context.rs
deleted file mode 100644
index 983c993dd..000000000
--- a/crates/uzu/src/device/device_context.rs
+++ /dev/null
@@ -1,272 +0,0 @@
-use bytemuck::fill_zeroes;
-use half::{bf16, f16};
-use ndarray::{ArrayView, Dimension};
-use num_traits::Zero;
-
-use crate::{Array, ArrayElement, DataType};
-
-fn slice_index_to_array_index<const D: usize>(
- index: usize,
- shape: &[usize; D],
-) -> [usize; D] {
- let mut indices = [0; D];
- let mut remaining_index = index;
-
- for (dim_index, dim_size) in shape.iter().enumerate().rev() {
- indices[dim_index] = remaining_index % dim_size;
- remaining_index = remaining_index / dim_size;
- }
- indices
-}
-
-pub trait DeviceContext {
- type DeviceArray: Array;
-
- /// Allocate a new array with the given shape and data type, but doesn't initialize it.
- unsafe fn array_uninitialized(
- &self,
- shape: &[usize],
- data_type: DataType,
- label: String,
- ) -> Self::DeviceArray;
-
- /// Allocate a new array with the given shape and data type.
- /// The result is guaranteed to be zero-initialized.
- fn array(
- &self,
- shape: &[usize],
- data_type: DataType,
- label: String,
- ) -> Self::DeviceArray {
- unsafe {
- let mut result = self.array_uninitialized(shape, data_type, label);
- fill_zeroes(result.buffer_mut());
- result
- }
- }
-
- /// Allocate a new array and populates it with the data copied from the given view.
- fn array_from_view<T: ArrayElement, D: Dimension>(
- &self,
- view: ArrayView<T, D>,
- label: String,
- ) -> Self::DeviceArray {
- unsafe {
- let mut result =
- self.array_uninitialized(view.shape(), T::data_type(), label);
- let result_buffer = result.as_slice_mut().unwrap();
-
- // If the view data is contiguous, copy it directly, otherwise .
- if let Some(slice) = view.as_slice_memory_order() {
- result_buffer.copy_from_slice(slice);
- } else {
- for (dst, src) in result_buffer.iter_mut().zip(view.iter()) {
- *dst = *src;
- }
- }
- result
- }
- }
-
- /// Allocate a new array filled with values created by the function.
- fn array_from_shape_fn<
- const D: usize,
- T: ArrayElement,
- F: FnMut(&[usize; D]) -> T,
- >(
- &self,
- shape: &[usize; D],
- label: String,
- mut f: F,
- ) -> Self::DeviceArray {
- unsafe {
- let mut result =
- self.array_uninitialized(shape, T::data_type(), label);
- for (slice_index, value) in
- result.as_slice_mut().unwrap().iter_mut().enumerate()
- {
- *value = f(&slice_index_to_array_index(slice_index, shape));
- }
- result
- }
- }
-
- /// Allocate a new array filled with copies of elem.
- fn array_from_elem<T: ArrayElement>(
- &self,
- shape: &[usize],
- elem: T,
- label: String,
- ) -> Self::DeviceArray {
- unsafe {
- let mut result =
- self.array_uninitialized(shape, T::data_type(), label);
- result.as_slice_mut().unwrap().fill(elem);
- result
- }
- }
-
- /// Allocate a new array containing a single element.
- fn scalar<T: ArrayElement>(
- &self,
- value: T,
- label: String,
- ) -> Self::DeviceArray {
- self.array_from_elem(&[], value, label)
- }
-
- /// Allocate a new array for an attention bias (also sometimes referred to as "attention mask").
- /// The result is -inf at indices where `should_be_neg_inf` is true, and zero otherwise.
- fn attention_bias<F>(
- &self,
- suffix_length: usize,
- prefix_length: usize,
- data_type: DataType,
- mut should_be_neg_inf: F,
- ) -> Self::DeviceArray
- where
- F: FnMut(usize, usize) -> bool,
- {
- let shape = [suffix_length, suffix_length + prefix_length];
- let label = String::from("attention_bias");
- match data_type {
- DataType::F16 => {
- self.array_from_shape_fn(&shape, label, |[row, col]| {
- if should_be_neg_inf(*row, *col) {
- f16::NEG_INFINITY
- } else {
- f16::zero()
- }
- })
- },
- DataType::BF16 => {
- self.array_from_shape_fn(&shape, label, |[row, col]| {
- if should_be_neg_inf(*row, *col) {
- bf16::NEG_INFINITY
- } else {
- bf16::zero()
- }
- })
- },
- DataType::F32 => {
- self.array_from_shape_fn(&shape, label, |[row, col]| {
- if should_be_neg_inf(*row, *col) {
- f32::NEG_INFINITY
- } else {
- f32::zero()
- }
- })
- },
- DataType::F64 => {
- self.array_from_shape_fn(&shape, label, |[row, col]| {
- if should_be_neg_inf(*row, *col) {
- f64::NEG_INFINITY
- } else {
- f64::zero()
- }
- })
- },
- _ => {
- panic!(
- "Attention bias can only be of a floating-point type, but requested {:?}",
- data_type
- );
- },
- }
- }
-
- /// Copy data from the given ndarray view into an already allocated device array.
- /// The destination buffer must have enough capacity to hold the view element count.
- fn copy_from_view<T: ArrayElement, D: Dimension>(
- &self,
- dst: &mut Self::DeviceArray,
- view: ArrayView<T, D>,
- ) {
- // Ensure data types match
- assert_eq!(dst.data_type(), T::data_type());
-
- let dst_slice =
- dst.as_slice_mut::<T>().expect("Destination slice not accessible");
-
- if let Some(src_slice) = view.as_slice_memory_order() {
- assert!(src_slice.len() <= dst_slice.len());
- dst_slice[..src_slice.len()].copy_from_slice(src_slice);
- } else {
- // Fallback: iterate element-wise
- assert!(view.len() <= dst_slice.len());
- for (d, s) in dst_slice.iter_mut().zip(view.iter()) {
- *d = *s;
- }
- }
- }
-
- /// TODO: Create an DeviceBuffer entity that has slice access to data, but doesn't have shape or type
- fn fill_attention_bias<F>(
- &self,
- dst: &mut Self::DeviceArray,
- suffix_length: usize,
- prefix_segment_length: usize,
- mut should_be_neg_inf: F,
- ) where
- F: FnMut(usize, usize) -> bool,
- {
- let total_elems =
- suffix_length * (suffix_length + prefix_segment_length);
- match dst.data_type() {
- DataType::F16 => {
- // TODO: Use lambda to avoid crimes
- let mut buf = Vec::<f16>::with_capacity(total_elems);
- for row in 0..suffix_length {
- for col in 0..suffix_length + prefix_segment_length {
- buf.push(if should_be_neg_inf(row, col) {
- f16::NEG_INFINITY
- } else {
- f16::zero()
- });
- }
- }
- self.copy_from_view(dst, ndarray::ArrayView1::from(&buf));
- },
- DataType::BF16 => {
- let mut buf = Vec::<bf16>::with_capacity(total_elems);
- for row in 0..suffix_length {
- for col in 0..suffix_length + prefix_segment_length {
- buf.push(if should_be_neg_inf(row, col) {
- bf16::NEG_INFINITY
- } else {
- bf16::zero()
- });
- }
- }
- self.copy_from_view(dst, ndarray::ArrayView1::from(&buf));
- },
- DataType::F32 => {
- let mut buf = Vec::<f32>::with_capacity(total_elems);
- for row in 0..suffix_length {
- for col in 0..suffix_length + prefix_segment_length {
- buf.push(if should_be_neg_inf(row, col) {
- f32::NEG_INFINITY
- } else {
- 0.0
- });
- }
- }
- self.copy_from_view(dst, ndarray::ArrayView1::from(&buf));
- },
- DataType::F64 => {
- let mut buf = Vec::<f64>::with_capacity(total_elems);
- for row in 0..suffix_length {
- for col in 0..suffix_length + prefix_segment_length {
- buf.push(if should_be_neg_inf(row, col) {
- f64::NEG_INFINITY
- } else {
- 0.0
- });
- }
- }
- self.copy_from_view(dst, ndarray::ArrayView1::from(&buf));
- },
- _ => panic!("Unsupported data type for attention bias fill"),
- }
- }
-}
diff --git a/crates/uzu/src/device/mod.rs b/crates/uzu/src/device/mod.rs
deleted file mode 100644
index 63badd4dd..000000000
--- a/crates/uzu/src/device/mod.rs
+++ /dev/null
@@ -1,7 +0,0 @@
-pub mod array;
-pub use array::*;
-pub mod data_type;
-pub use data_type::*;
-pub mod data_type_ext;
-pub mod device_context;
-pub use device_context::*;
diff --git a/crates/uzu/src/encodable_block/mod.rs b/crates/uzu/src/encodable_block/mod.rs
new file mode 100644
index 000000000..5a89ecfb4
--- /dev/null
+++ b/crates/uzu/src/encodable_block/mod.rs
@@ -0,0 +1,77 @@
+use crate::backends::{
+ common::{Backend, CommandBuffer},
+ metal::ForwardPassState,
+};
+
+mod sampling;
+
+pub use sampling::Sampling;
+
+#[derive(Clone)]
+pub struct EncodingParameters<'a, B: Backend> {
+ pub warmup: bool,
+ pub enable_commit: bool,
+ pub wait_until_completed: bool,
+ pub projection_step: Option<usize>,
+ pub predicate: Option<&'a B::NativeBuffer>,
+}
+
+impl<'a, B: Backend> EncodingParameters<'a, B> {
+ pub fn new(
+ warmup: bool,
+ enable_commit: bool,
+ wait_until_completed: bool,
+ ) -> Self {
+ Self {
+ warmup,
+ enable_commit,
+ wait_until_completed,
+ projection_step: None,
+ predicate: None,
+ }
+ }
+
+ pub fn with_projection(
+ mut self,
+ projection_step: usize,
+ ) -> Self {
+ self.projection_step = Some(projection_step);
+ self
+ }
+
+ pub fn with_predicate(
+ mut self,
+ predicate: &'a B::NativeBuffer,
+ ) -> Self {
+ self.predicate = Some(predicate);
+ self
+ }
+
+ pub fn predicate_ref(&self) -> Option<&B::NativeBuffer> {
+ self.predicate
+ }
+}
+
+pub trait EncodableBlock<B: Backend> {
+ fn encode_with_shared_encoder(
+ &self,
+ state: &mut ForwardPassState,
+ encoder: &B::EncoderRef,
+ parameters: &EncodingParameters<B>,
+ );
+
+ fn supports_shared_encoder(&self) -> bool {
+ false
+ }
+
+ fn encode(
+ &self,
+ state: &mut ForwardPassState,
+ command_buffer: &B::CommandBuffer,
+ parameters: &EncodingParameters<B>,
+ ) {
+ command_buffer.with_encoder(|encoder| {
+ self.encode_with_shared_encoder(state, encoder, parameters)
+ });
+ }
+}
diff --git a/crates/uzu/src/encodable_block/sampling.rs b/crates/uzu/src/encodable_block/sampling.rs
new file mode 100644
index 000000000..c6cb660c2
--- /dev/null
+++ b/crates/uzu/src/encodable_block/sampling.rs
@@ -0,0 +1,118 @@
+//! Sampling kernel encodable.
+
+use crate::{
+ DataType,
+ backends::{
+ common::{
+ Backend,
+ kernel::sampling::{ArgmaxStrategy, SamplingError, SamplingKernel},
+ },
+ metal::{
+ MTLComputeCommandEncoder, Metal, ProtocolObject,
+ forward_pass::{ArrayId, ForwardPassState},
+ },
+ },
+};
+
+use super::{EncodableBlock, EncodingParameters};
+
+pub struct Sampling<B: Backend> {
+ kernel: SamplingKernel<B>,
+}
+
+impl<B: Backend> Sampling<B> {
+ pub fn new(
+ context: &B::Context,
+ data_type: DataType,
+ max_batch_size: usize,
+ max_vocab_size: usize,
+ ) -> Result<Self, SamplingError<B>> {
+ let kernel = SamplingKernel::new(
+ context,
+ data_type,
+ max_batch_size,
+ max_vocab_size,
+ )?;
+ Ok(Self {
+ kernel,
+ })
+ }
+
+ pub fn new_with_strategy(
+ context: &B::Context,
+ data_type: DataType,
+ max_batch_size: usize,
+ max_vocab_size: usize,
+ argmax_strategy: ArgmaxStrategy,
+ ) -> Result<Self, SamplingError<B>> {
+ let kernel = SamplingKernel::new_with_strategy(
+ context,
+ data_type,
+ max_batch_size,
+ max_vocab_size,
+ argmax_strategy,
+ )?;
+ Ok(Self {
+ kernel,
+ })
+ }
+}
+
+impl EncodableBlock<Metal> for Sampling<Metal> {
+ fn encode_with_shared_encoder(
+ &self,
+ state: &mut ForwardPassState,
+ encoder: &ProtocolObject<dyn MTLComputeCommandEncoder>,
+ _parameters: &EncodingParameters<Metal>,
+ ) {
+ assert!(
+ state.sampling_output().is_some(),
+ "Sampling output buffer must be pre-allocated"
+ );
+
+ let logits_binding = state.arrays(&[ArrayId::Logits]);
+ let logits = logits_binding[0].borrow();
+
+ let logits_shape = logits.shape();
+ let batch_size = state.sampling_length();
+ if batch_size == 0 {
+ return;
+ }
+ let sampling_start = state.sampling_start();
+ let vocab_size = logits_shape[1];
+
+ let seeds_binding = state.arrays(&[ArrayId::TokenSeeds]);
+ let seeds = seeds_binding[0].borrow();
+
+ let output_buffer_ref = state.sampling_output().unwrap().borrow();
+
+ let sampling_method = state.sampling_method().unwrap();
+ let seeds_offset =
+ seeds.offset() + sampling_start * std::mem::size_of::<u64>();
+
+ let (bitmask_buffer, bitmask_offset) =
+ state.token_bitmask().map_or((None, 0usize), |cell| {
+ let bitmask = cell.borrow();
+ let bitmask_row_len = bitmask.shape()[1];
+ let bitmask_offset = bitmask.offset()
+ + sampling_start
+ * bitmask_row_len
+ * std::mem::size_of::<u32>();
+ (Some(bitmask.buffer().clone()), bitmask_offset)
+ });
+ if let Err(e) = self.kernel.encode_with_encoder(
+ logits.buffer(),
+ Some(seeds.buffer()),
+ seeds_offset,
+ bitmask_buffer.as_ref(),
+ bitmask_offset,
+ output_buffer_ref.buffer(),
+ sampling_method,
+ batch_size,
+ vocab_size,
+ encoder,
+ ) {
+ panic!("Sampling encoding failed: {:?}", e);
+ }
+ }
+}
diff --git a/crates/uzu/src/language_model/gpu_capture.rs b/crates/uzu/src/language_model/gpu_capture.rs
index c2d461e2d..d70da7b81 100644
--- a/crates/uzu/src/language_model/gpu_capture.rs
+++ b/crates/uzu/src/language_model/gpu_capture.rs
@@ -3,7 +3,7 @@ use std::{
time::{SystemTime, UNIX_EPOCH},
};
-use objc2_foundation::NSURL;
+use objc2_foundation::{NSURL, NSString};
use crate::{
backends::metal::{
@@ -83,7 +83,7 @@ impl GpuCaptureManager {
let url_string = format!("file://{}", trace_path.display());
unsafe {
if let Some(url) = NSURL::URLWithString(
- &objc2_foundation::NSString::from_str(&url_string),
+ &NSString::from_str(&url_string),
) {
capture_descriptor.set_output_url(Some(&url));
}
diff --git a/crates/uzu/src/language_model/language_model_generator.rs b/crates/uzu/src/language_model/language_model_generator.rs
index 70c815fdd..f7a1b0bc2 100644
--- a/crates/uzu/src/language_model/language_model_generator.rs
+++ b/crates/uzu/src/language_model/language_model_generator.rs
@@ -9,17 +9,19 @@ use super::{
gpu_capture::GpuCaptureManager,
grammar::CompiledGrammar,
result::{GenerateResult, PrefillResult},
- rng::DerivableSeed,
+ rng::PRng,
tasks::{LanguageModelGeneratorEncodedTask, LanguageModelGeneratorRunTask},
};
use crate::{
- Array,
- backends::metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandBufferExt,
- MTLCommandBufferHandler, MTLCommandEncoder, MTLCommandQueue,
- forward_pass::{
- AttentionBiasUpdate, EncodableBlock, EncodingParameters,
- ForwardPassState, INVALID_POSITION,
+ backends::{
+ common::kernel::MaskUpdateKernel,
+ metal::{
+ MTLBuffer, MTLCommandBuffer, MTLCommandBufferExt,
+ MTLCommandBufferHandler, MTLCommandEncoder, MTLCommandQueue,
+ forward_pass::{
+ AttentionBiasUpdate, EncodableBlock, EncodingParameters,
+ ForwardPassState, INVALID_POSITION,
+ },
},
},
session::{
@@ -96,7 +98,7 @@ impl LanguageModelGenerator {
let suffix_length = prefill_size - tokens_length;
let suffix_root = TrieNode::from_speculator(
&tokens,
- &mut self.context.next_seed,
+ &self.context.seed,
compiled_grammar.as_deref_mut(),
speculator.as_ref(),
&TrieCreationConfig::default(),
@@ -329,7 +331,7 @@ impl LanguageModelGenerator {
let suffix_length = self.decoding_config.generate_suffix_length();
let suffix_root = TrieNode::from_speculator(
&self.tokens,
- &mut self.context.next_seed,
+ &self.context.seed,
compiled_grammar.as_deref_mut(),
speculator.as_ref(),
&TrieCreationConfig::default(),
@@ -411,7 +413,7 @@ impl LanguageModelGenerator {
let (accepted_tokens, accepted_token_indices) =
flat_trie.accept(&sampled_tokens, compiled_grammar.as_deref_mut());
- self.update_cache_layers(&accepted_token_indices[1..], None, false);
+ self.update_cache_layers(&accepted_token_indices, None, false);
self.tokens.extend(accepted_tokens.clone());
self.sync_prefix();
@@ -447,9 +449,11 @@ impl LanguageModelGenerator {
self.context
.async_buffers
.prepare_positions(prefill_count, tokens_to_generate);
- self.context
- .async_buffers
- .prepare_seeds(&mut self.context.next_seed, tokens_to_generate);
+ self.context.async_buffers.prepare_seeds(
+ &self.context.seed,
+ prefill_count,
+ tokens_to_generate,
+ );
self.context.async_buffers.reset_counter();
}
@@ -571,10 +575,9 @@ impl LanguageModelGenerator {
let sampling_output = state
.sampling_output()
.expect("sampling_output must exist after sampling encode");
- let sampling_output_buffer =
- sampling_output.borrow().mtl_buffer_cloned();
+ let sampling_output_buffer = sampling_output.borrow().buffer().clone();
let token_ids_buffer =
- self.context.scratch_buffers.token_ids.borrow().mtl_buffer_cloned();
+ self.context.scratch_buffers.token_ids.borrow().buffer().clone();
let encoder = root_command_buffer
.new_compute_command_encoder()
@@ -625,10 +628,10 @@ impl LanguageModelGenerator {
{
if update.unmask_col >= 0 || update.mask_col >= 0 {
mask_update.encode(
- &encoder,
- mask_buffer.borrow().backend_buffer(),
+ mask_buffer.borrow().buffer(),
update.unmask_col,
update.mask_col,
+ &encoder,
);
}
}
@@ -686,8 +689,8 @@ impl LanguageModelGenerator {
self.encoded_tasks.clear();
self.gpu_capture.reset();
- let base_seed = self.decoding_config.sampling_seed.resolve();
- self.context.next_seed = DerivableSeed::new(base_seed);
+ let seed = self.decoding_config.sampling_seed.resolve();
+ self.context.seed = PRng::new(seed);
self.context.async_buffers.reset_counter();
}
@@ -725,7 +728,7 @@ impl LanguageModelGenerator {
warmup: bool,
allow_pre_encode: bool,
sampling_method: SamplingMethod,
- skip_attention_bias_fill: bool,
+ should_fill_attention_bias: bool,
) -> (ForwardPassState, f64) {
objc2::rc::autoreleasepool(|_pool| {
let run_start = Instant::now();
@@ -733,7 +736,7 @@ impl LanguageModelGenerator {
let mut state = task.create_state(
&mut self.context,
None,
- skip_attention_bias_fill,
+ should_fill_attention_bias,
);
if let Some(method) = state.sampling_method_mut() {
*method = Some(sampling_method);
@@ -818,9 +821,7 @@ impl LanguageModelGenerator {
.expect("Sampling output buffer not found - ensure sampling was encoded during forward pass");
let output_buffer = sampling_output.borrow();
- let output_view = output_buffer
- .as_view::<u32>()
- .map_err(|_| Error::SamplingFailed)?;
+ let output_view = output_buffer.as_view::<u32>();
let batch_size = state.sampling_length();
let mut result = Vec::with_capacity(batch_size);
diff --git a/crates/uzu/src/language_model/rng.rs b/crates/uzu/src/language_model/rng.rs
index d758e2fe3..3f8e0b410 100644
--- a/crates/uzu/src/language_model/rng.rs
+++ b/crates/uzu/src/language_model/rng.rs
@@ -1,18 +1,19 @@
-pub struct DerivableSeed {
- base_seed: u64,
- invocation: u64,
+pub struct PRng {
+ seed: u64,
}
-impl DerivableSeed {
+impl PRng {
pub fn new(seed: u64) -> Self {
Self {
- base_seed: seed,
- invocation: 0,
+ seed,
}
}
- pub fn current(&self) -> u64 {
- let mut hash = self.base_seed.wrapping_add(self.invocation);
+ pub fn derive(
+ &self,
+ index: u64,
+ ) -> u64 {
+ let mut hash = self.seed.wrapping_add(index);
hash ^= hash >> 33;
hash = hash.wrapping_mul(0xff51afd7ed558ccd);
hash ^= hash >> 33;
@@ -20,10 +21,4 @@ impl DerivableSeed {
hash ^= hash >> 33;
hash
}
-
- pub fn next(&mut self) -> u64 {
- self.invocation += 1;
-
- self.current()
- }
}
diff --git a/crates/uzu/src/language_model/tasks.rs b/crates/uzu/src/language_model/tasks.rs
index 66e3243fb..411195aab 100644
--- a/crates/uzu/src/language_model/tasks.rs
+++ b/crates/uzu/src/language_model/tasks.rs
@@ -1,9 +1,12 @@
use std::mem::size_of;
use super::LanguageModelGeneratorContext;
-use crate::backends::metal::{
- BufferExt, MTLBuffer, MTLDeviceExt, MTLResourceOptions, ProtocolObject, Retained,
- forward_pass::{EncodableBlock, EncodingParameters, ForwardPassState},
+use crate::backends::{
+ common::Context,
+ metal::{
+ BufferExt, MTLBuffer, ProtocolObject, Retained,
+ forward_pass::{EncodableBlock, EncodingParameters, ForwardPassState},
+ },
};
pub struct LanguageModelGeneratorEncodedTask {
@@ -81,7 +84,7 @@ impl<'a> LanguageModelGeneratorRunTask<'a> {
&self,
context: &mut LanguageModelGeneratorContext,
external_bias_fn: Option<&dyn Fn(usize, usize) -> bool>,
- skip_attention_bias_fill: bool,
+ should_fill_attention_bias: bool,
) -> ForwardPassState {
ForwardPassState::new_llm(
context.mtl_context.clone(),
@@ -100,7 +103,7 @@ impl<'a> LanguageModelGeneratorRunTask<'a> {
self.is_prefilling,
external_bias_fn,
false,
- skip_attention_bias_fill,
+ should_fill_attention_bias,
None,
None,
)
@@ -119,11 +122,7 @@ impl<'a> LanguageModelGeneratorRunTask<'a> {
key,
predicate_buffer: context
.mtl_context
- .device
- .new_buffer(
- size_of::<u32>(),
- MTLResourceOptions::STORAGE_MODE_SHARED,
- )
+ .create_buffer(size_of::<u32>())
.expect("Failed to create predicate buffer"),
}
}
diff --git a/crates/uzu/src/lib.rs b/crates/uzu/src/lib.rs
index aac000ee8..e8d43ecfb 100644
--- a/crates/uzu/src/lib.rs
+++ b/crates/uzu/src/lib.rs
@@ -1,6 +1,9 @@
+pub mod array;
pub mod backends;
pub mod classifier;
pub mod config;
+pub mod data_type;
+pub mod encodable_block;
pub mod language_model;
pub mod parameters;
pub mod prelude;
@@ -10,7 +13,7 @@ pub mod speculators;
pub mod tracer;
pub mod trie;
pub mod utils;
-pub use utils::*;
-pub mod device;
+
pub use config::*;
-pub use device::*;
+pub use data_type::*;
+pub use utils::*;
diff --git a/crates/uzu/src/parameters/loader.rs b/crates/uzu/src/parameters/loader.rs
index 428c570c9..966cfe721 100644
--- a/crates/uzu/src/parameters/loader.rs
+++ b/crates/uzu/src/parameters/loader.rs
@@ -10,7 +10,11 @@ use super::safetensors_metadata::{
HashMetadata as STMetadata, HeaderLoadingError,
read_metadata as read_st_metadata,
};
-use crate::{Array, DataType, DeviceContext};
+use crate::{
+ DataType,
+ array::{Array, ArrayContextExt},
+ backends::common::Context,
+};
pub struct ParameterMetadata {
shape: Box<[usize]>,
@@ -61,7 +65,7 @@ pub enum ParameterLoaderError {
ArrayLoadingError(#[from] std::io::Error),
}
-pub struct ParameterLoader<'context, 'file, C: DeviceContext>
+pub struct ParameterLoader<'context, 'file, C: Context>
where
'file: 'context,
{
@@ -70,7 +74,7 @@ where
file: &'file File,
}
-impl<'file, 'context, C: DeviceContext> ParameterLoader<'file, 'context, C>
+impl<'file, 'context, C: Context> ParameterLoader<'file, 'context, C>
where
'file: 'context,
{
@@ -94,7 +98,7 @@ where
pub fn get(
&self,
key: &str,
- ) -> Result<C::DeviceArray, ParameterLoaderError> {
+ ) -> Result<Array<C::Backend>, ParameterLoaderError> {
let metadata_entry = self
.index
.get(key)
@@ -102,12 +106,12 @@ where
let (offset, size) = (metadata_entry.offset, metadata_entry.size);
let array_key = key.replace(".", "_");
let array_label = format!("parameter_loader_{array_key}");
- let mut array = self.context.array(
+ let mut array = self.context.create_array(
&metadata_entry.shape,
metadata_entry.data_type,
- array_label,
+ &array_label,
);
- let expected_size = array.size_in_bytes();
+ let expected_size = array.size();
if expected_size != size {
return Err(ParameterLoaderError::SizeMismatch {
data_type: metadata_entry.data_type,
@@ -116,7 +120,7 @@ where
actual_size: size,
});
}
- self.file.read_exact_at(array.buffer_mut(), offset as u64)?;
+ self.file.read_exact_at(array.as_bytes_mut(), offset as u64)?;
Ok(array)
}
@@ -145,12 +149,12 @@ where
}
}
-pub struct ParameterTree<'loader, C: DeviceContext> {
+pub struct ParameterTree<'loader, C: Context> {
loader: &'loader ParameterLoader<'loader, 'loader, C>,
prefix: Option<String>,
}
-impl<'loader, C: DeviceContext> ParameterTree<'loader, C> {
+impl<'loader, C: Context> ParameterTree<'loader, C> {
pub fn new(loader: &'loader ParameterLoader<'loader, 'loader, C>) -> Self {
Self {
loader,
@@ -194,7 +198,7 @@ impl<'loader, C: DeviceContext> ParameterTree<'loader, C> {
pub fn leaf(
&self,
name: &str,
- ) -> Result<C::DeviceArray, ParameterLoaderError> {
+ ) -> Result<Array<C::Backend>, ParameterLoaderError> {
self.loader.get(&self.join_prefix(name))
}
diff --git a/crates/uzu/src/parameters/mod.rs b/crates/uzu/src/parameters/mod.rs
index eef146ed3..ff3f5ddc2 100644
--- a/crates/uzu/src/parameters/mod.rs
+++ b/crates/uzu/src/parameters/mod.rs
@@ -3,5 +3,5 @@ pub use loader::{ParameterLoader, ParameterLoaderError, ParameterTree};
mod safetensors_metadata;
// Re-export the safetensors header reader so other modules (e.g. decoder
-// runner) can estimate parameter memory before creating a DeviceContext.
+// runner) can estimate parameter memory before creating a Context.
pub use safetensors_metadata::read_metadata as read_safetensors_metadata;
diff --git a/crates/uzu/src/session/chat_session.rs b/crates/uzu/src/session/chat_session.rs
index e39e90f02..9ef961bf9 100644
--- a/crates/uzu/src/session/chat_session.rs
+++ b/crates/uzu/src/session/chat_session.rs
@@ -648,8 +648,8 @@ impl ChatSession {
if let Some(ctx) = context {
let mut llm_state =
language_model_generator.context.cache_layers.borrow_mut();
- for (ctx_layer, gen_layer) in
- ctx.cache_layers.data.iter().zip(llm_state.data.iter_mut())
+ for (_layer_idx, (ctx_layer, gen_layer)) in
+ ctx.cache_layers.data.iter().zip(llm_state.data.iter_mut()).enumerate()
{
match (ctx_layer, gen_layer) {
(
diff --git a/crates/uzu/src/tracer/trace_validator.rs b/crates/uzu/src/tracer/trace_validator.rs
index dc305f810..188cc3b1a 100644
--- a/crates/uzu/src/tracer/trace_validator.rs
+++ b/crates/uzu/src/tracer/trace_validator.rs
@@ -16,11 +16,10 @@ use ndarray::{IxDyn, s};
use num_traits::NumCast;
use crate::{
- Array, ArrayElement, DataType,
+ ArrayElement, DataType,
backends::metal::{
CacheLayers, KVCacheUpdate, KernelDataType, MTLCommandBuffer,
MTLCommandQueue, MTLContext, MetalArray,
- encodable_block::Sampling,
forward_pass::{
ArrayId, EncodableBlock, EncodingParameters, ForwardPassState,
ScratchBuffers, traces::ActivationTrace,
@@ -28,6 +27,7 @@ use crate::{
},
classifier::Classifier,
config::ModelMetadata,
+ encodable_block::Sampling,
language_model::{
LanguageModelGeneratorContext,
sampler::{ArgmaxSampler, LogitsSampler},
@@ -261,7 +261,7 @@ impl TraceValidator {
let traces_file =
File::open(traces_path).map_err(|_| Error::UnableToLoadWeights)?;
let traces_loader =
- ParameterLoader::new(&traces_file, &ctx.mtl_context)
+ ParameterLoader::new(&traces_file, ctx.mtl_context.as_ref())
.map_err(|_| Error::UnableToLoadWeights)?;
let traces_view = traces_loader.tree();
@@ -292,7 +292,7 @@ impl TraceValidator {
false,
None,
false,
- false,
+ true,
None,
None,
);
@@ -467,8 +467,9 @@ impl TraceValidator {
let traces_file =
File::open(traces_path).map_err(|_| Error::UnableToLoadWeights)?;
let mtl_context = classifier.context.mtl_context.clone();
- let traces_loader = ParameterLoader::new(&traces_file, &mtl_context)
- .map_err(|_| Error::UnableToLoadWeights)?;
+ let traces_loader =
+ ParameterLoader::new(&traces_file, mtl_context.as_ref())
+ .map_err(|_| Error::UnableToLoadWeights)?;
let traces_view = traces_loader.tree();
let has_token_ids =
@@ -514,7 +515,7 @@ impl TraceValidator {
}
fn handle_missing_tokens(
- traces_view: &ParameterTree<Rc<MTLContext>>
+ traces_view: &ParameterTree<MTLContext>
) -> TracerValidationResults {
if let Ok(expected_logits) = traces_view.leaf("logits") {
let reference_shape = expected_logits.shape().to_vec();
@@ -561,7 +562,7 @@ impl TraceValidator {
fn validate_layer_traces(
traces: &Rc<RefCell<ActivationTrace>>,
- traces_view: &ParameterTree<Rc<MTLContext>>,
+ traces_view: &ParameterTree<MTLContext>,
data_type: DataType,
) -> Vec<TracerValidationResult> {
let mut results = Vec::new();
@@ -676,7 +677,7 @@ impl TraceValidator {
fn validate_classifier_traces(
traces: &Rc<RefCell<ActivationTrace>>,
- traces_view: &ParameterTree<Rc<MTLContext>>,
+ traces_view: &ParameterTree<MTLContext>,
data_type: DataType,
) -> Vec<TracerValidationResult> {
let mut results = Vec::new();
@@ -746,7 +747,7 @@ impl TraceValidator {
fn validate_array_with_name(
data_type: DataType,
- traces_view: &ParameterTree<Rc<MTLContext>>,
+ traces_view: &ParameterTree<MTLContext>,
expected_array_path: &str,
produced_array: &Ref<MetalArray>,
) -> TracerValidationMetrics {
@@ -759,8 +760,8 @@ impl TraceValidator {
produced_array: &Ref<MetalArray>,
transform: Option<ArrayTransform>,
) -> TracerValidationMetrics {
- let expected_view = expected_array.as_view::<Precision>().unwrap();
- let produced_view = produced_array.as_view::<Precision>().unwrap();
+ let expected_view = expected_array.as_view::<Precision>();
+ let produced_view = produced_array.as_view::<Precision>();
let (mut expected_data, mut produced_data) = match transform {
Some(ArrayTransform::KVCacheSlice) => {
@@ -952,11 +953,11 @@ impl TraceValidator {
SourcePrecision: ArrayElement,
TargetPrecision: NumCast,
>(
- traces_view: &ParameterTree<Rc<MTLContext>>,
+ traces_view: &ParameterTree<MTLContext>,
name: &str,
) -> Vec<TargetPrecision> {
let array = traces_view.leaf(name).unwrap();
- let slice = array.as_slice::<SourcePrecision>().unwrap();
+ let slice = array.as_slice::<SourcePrecision>();
slice.iter().map(|x| NumCast::from(*x).unwrap()).collect()
}
@@ -1030,8 +1031,8 @@ impl TraceValidator {
);
context.gpu_sampler = Sampling::new(
- &context.mtl_context,
- kernel_dtype,
+ context.mtl_context.as_ref(),
+ intermediate_dtype,
desired_suffix_length,
decoder_config.vocab_size,
)
@@ -1058,6 +1059,6 @@ impl TraceValidator {
logits: &MetalArray
) -> Vec<u64> {
let sampler = ArgmaxSampler {};
- sampler.sample(logits.as_view::<Precision>().unwrap())
+ sampler.sample(logits.as_view::<Precision>())
}
}
diff --git a/crates/uzu/src/trie.rs b/crates/uzu/src/trie.rs
index 1bd28cf4d..4e827aa47 100644
--- a/crates/uzu/src/trie.rs
+++ b/crates/uzu/src/trie.rs
@@ -2,7 +2,7 @@ use thiserror::Error;
use crate::{
language_model::{
- grammar::CompiledGrammar, gumbel::speculator_sample, rng::DerivableSeed,
+ grammar::CompiledGrammar, gumbel::speculator_sample, rng::PRng,
},
speculators::speculator::Speculator,
};
@@ -88,7 +88,7 @@ impl TrieNode {
pub fn from_speculator(
prefix: &[u64],
- next_seed: &mut DerivableSeed,
+ seed: &PRng,
mut compiled_grammar: Option<&mut CompiledGrammar>,
speculator: &dyn Speculator,
creation_config: &TrieCreationConfig,
@@ -97,15 +97,19 @@ impl TrieNode {
assert!(max_length >= 1, "can't have zero sized trie");
assert!(prefix.len() >= 1, "need seed node");
+ let prefix_length = prefix.len();
let mut speculated_suffix = prefix.to_vec();
let mut length = 1;
- let mut grammar_accept_count = 0;
+ let mut height = 0;
let mask = compiled_grammar
.as_deref_mut()
.and_then(|g| g.next_bitmask().unwrap());
- let mut root =
- Self::new(*prefix.last().unwrap(), mask, next_seed.next());
+ let mut root = Self::new(
+ *prefix.last().unwrap(),
+ mask,
+ seed.derive((prefix_length - 1) as u64),
+ );
let mut cur_node = &mut root;
let mut cur_node_width = 0;
@@ -139,8 +143,11 @@ impl TrieNode {
None
};
- let leaf_node =
- Self::new(next_speculated_token, mask, next_seed.next());
+ let leaf_node = Self::new(
+ next_speculated_token,
+ mask,
+ seed.derive((prefix_length + height) as u64),
+ );
cur_node.add(leaf_node).unwrap();
// If this is the first node we sampled (most likely to be selected after gumbel noise application) - set it as the next one
@@ -164,8 +171,8 @@ impl TrieNode {
if compiled_grammar.accept_token(next_node_token).is_err() {
break;
}
- grammar_accept_count += 1;
}
+ height += 1;
cur_node = cur_node.get_mut(next_node_token).unwrap();
cur_node_width = 0;
cur_node_speculator_weights =
@@ -178,7 +185,7 @@ impl TrieNode {
}
if let Some(compiled_grammar) = compiled_grammar.as_deref_mut() {
- compiled_grammar.rollback(grammar_accept_count);
+ compiled_grammar.rollback(height);
}
root
diff --git a/crates/uzu/src/utils/attention.rs b/crates/uzu/src/utils/attention.rs
new file mode 100644
index 000000000..67ea27ae3
--- /dev/null
+++ b/crates/uzu/src/utils/attention.rs
@@ -0,0 +1,45 @@
+use half::{bf16, f16};
+use num_traits::Float;
+
+use crate::{ArrayElement, DataType, array::Array, backends::common::Backend};
+
+pub fn fill_attention_bias(
+ dst: &mut Array<impl Backend>,
+ suffix_length: usize,
+ prefix_segment_length: usize,
+ should_be_neg_inf: impl Fn(usize, usize) -> bool,
+) {
+ let fill_typed_for_dtype = match dst.data_type() {
+ DataType::F16 => fill_typed::<f16>,
+ DataType::BF16 => fill_typed::<bf16>,
+ DataType::F32 => fill_typed::<f32>,
+ DataType::F64 => fill_typed::<f64>,
+ _ => panic!("Unsupported data type for attention bias fill"),
+ };
+
+ fill_typed_for_dtype(
+ dst,
+ suffix_length,
+ prefix_segment_length,
+ should_be_neg_inf,
+ );
+}
+
+fn fill_typed<T: ArrayElement + Float>(
+ dst: &mut Array<impl Backend>,
+ suffix_length: usize,
+ prefix_segment_length: usize,
+ should_be_neg_inf: impl Fn(usize, usize) -> bool,
+) {
+ let cols = suffix_length + prefix_segment_length;
+ let slice = dst.as_slice_mut::<T>();
+ for row in 0..suffix_length {
+ for col in 0..cols {
+ slice[row * cols + col] = if should_be_neg_inf(row, col) {
+ T::neg_infinity()
+ } else {
+ T::zero()
+ };
+ }
+ }
+}
diff --git a/crates/uzu/src/utils/device.rs b/crates/uzu/src/utils/device.rs
index a100e2d23..4d805505a 100644
--- a/crates/uzu/src/utils/device.rs
+++ b/crates/uzu/src/utils/device.rs
@@ -11,11 +11,14 @@ pub enum DeviceClass {
impl DeviceClass {
pub fn detect() -> Self {
use metal::MTLDeviceExt as _;
- let device: crate::backends::metal::Retained<crate::backends::metal::ProtocolObject<dyn crate::backends::metal::MTLDevice>> =
- match <dyn metal::MTLDevice>::system_default() {
- Some(dev) => dev,
- None => return DeviceClass::Unknown,
- };
+ let device: crate::backends::metal::Retained<
+ crate::backends::metal::ProtocolObject<
+ dyn crate::backends::metal::MTLDevice,
+ >,
+ > = match <dyn metal::MTLDevice>::system_default() {
+ Some(dev) => dev,
+ None => return DeviceClass::Unknown,
+ };
let name = device.name().to_lowercase();
diff --git a/crates/uzu/src/utils/mod.rs b/crates/uzu/src/utils/mod.rs
index 13264fa5a..da72c7209 100644
--- a/crates/uzu/src/utils/mod.rs
+++ b/crates/uzu/src/utils/mod.rs
@@ -1,3 +1,4 @@
+pub mod attention;
pub mod device;
pub mod env_utils;
pub mod model_size;
diff --git a/crates/uzu/tests/allocator_test/allocation_stats.rs b/crates/uzu/tests/allocator_test/allocation_stats.rs
new file mode 100644
index 000000000..cf151c835
--- /dev/null
+++ b/crates/uzu/tests/allocator_test/allocation_stats.rs
@@ -0,0 +1,46 @@
+use std::time::Duration;
+
+#[derive(Clone, Copy, Default)]
+pub struct AllocationStats {
+ pub total_allocs: usize,
+ pub total_frees: usize,
+ pub total_alloc_time_ns: u128,
+ pub total_free_time_ns: u128,
+ pub total_bytes_requested: usize,
+}
+
+impl AllocationStats {
+ pub fn record_alloc(
+ &mut self,
+ size: usize,
+ duration: Duration,
+ ) {
+ self.total_allocs += 1;
+ self.total_alloc_time_ns += duration.as_nanos();
+ self.total_bytes_requested += size;
+ }
+
+ pub fn record_free(
+ &mut self,
+ duration: Duration,
+ ) {
+ self.total_frees += 1;
+ self.total_free_time_ns += duration.as_nanos();
+ }
+
+ pub fn avg_alloc_time_us(&self) -> f64 {
+ if self.total_allocs == 0 {
+ 0.0
+ } else {
+ self.total_alloc_time_ns as f64 / self.total_allocs as f64 / 1000.0
+ }
+ }
+
+ pub fn avg_free_time_us(&self) -> f64 {
+ if self.total_frees == 0 {
+ 0.0
+ } else {
+ self.total_free_time_ns as f64 / self.total_frees as f64 / 1000.0
+ }
+ }
+}
diff --git a/crates/uzu/tests/allocator_test/allocator_trait.rs b/crates/uzu/tests/allocator_test/allocator_trait.rs
new file mode 100644
index 000000000..db5eb325d
--- /dev/null
+++ b/crates/uzu/tests/allocator_test/allocator_trait.rs
@@ -0,0 +1,15 @@
+pub trait AllocatorTrait {
+ type Buffer;
+
+ fn alloc(
+ &self,
+ size: usize,
+ ) -> Self::Buffer;
+ fn free(
+ &self,
+ buffer: Self::Buffer,
+ );
+ fn peak_memory(&self) -> usize;
+ fn cache_memory(&self) -> usize;
+ fn active_memory(&self) -> usize;
+}
diff --git a/crates/uzu/tests/allocator_test/caching_allocator.rs b/crates/uzu/tests/allocator_test/caching_allocator.rs
new file mode 100644
index 000000000..e24c049e5
--- /dev/null
+++ b/crates/uzu/tests/allocator_test/caching_allocator.rs
@@ -0,0 +1,54 @@
+use std::rc::Rc;
+
+use uzu::backends::{
+ common::{Buffer, BufferLifetime, Context},
+ metal::{MTLContext, Metal},
+};
+
+use super::allocator_trait::AllocatorTrait;
+
+pub struct CachingAllocator {
+ pub context: Rc<MTLContext>,
+}
+
+impl CachingAllocator {
+ pub fn new() -> Self {
+ let context = MTLContext::new().expect("Failed to create MTLContext");
+ Self {
+ context,
+ }
+ }
+}
+
+impl AllocatorTrait for CachingAllocator {
+ type Buffer = Buffer<Metal>;
+
+ fn alloc(
+ &self,
+ size: usize,
+ ) -> Self::Buffer {
+ self.context
+ .allocator()
+ .alloc(BufferLifetime::Scratch, size)
+ .expect("Allocation failed")
+ }
+
+ fn free(
+ &self,
+ buffer: Self::Buffer,
+ ) {
+ drop(buffer);
+ }
+
+ fn peak_memory(&self) -> usize {
+ self.context.allocator().peak_memory()
+ }
+
+ fn cache_memory(&self) -> usize {
+ self.context.allocator().cache_memory()
+ }
+
+ fn active_memory(&self) -> usize {
+ self.context.allocator().active_memory()
+ }
+}
diff --git a/crates/uzu/tests/allocator_test/device_allocator.rs b/crates/uzu/tests/allocator_test/device_allocator.rs
new file mode 100644
index 000000000..905a429f8
--- /dev/null
+++ b/crates/uzu/tests/allocator_test/device_allocator.rs
@@ -0,0 +1,70 @@
+use std::cell::Cell;
+
+use metal::{MTLBuffer, MTLDevice, MTLDeviceExt, MTLResourceOptions};
+use objc2::{rc::Retained, runtime::ProtocolObject};
+
+use super::allocator_trait::AllocatorTrait;
+
+pub struct DeviceAllocator {
+ device: Retained<ProtocolObject<dyn MTLDevice>>,
+ active_memory: Cell<usize>,
+ peak_memory: Cell<usize>,
+}
+
+impl DeviceAllocator {
+ pub fn new() -> Self {
+ let device =
+ <dyn MTLDevice>::system_default().expect("No Metal device");
+
+ Self {
+ device,
+ active_memory: Cell::new(0),
+ peak_memory: Cell::new(0),
+ }
+ }
+}
+
+impl AllocatorTrait for DeviceAllocator {
+ type Buffer = Retained<ProtocolObject<dyn MTLBuffer>>;
+
+ fn alloc(
+ &self,
+ size: usize,
+ ) -> Self::Buffer {
+ let buffer = self
+ .device
+ .new_buffer(size, MTLResourceOptions::STORAGE_MODE_SHARED)
+ .expect("Failed to create buffer");
+
+ let buf_size = buffer.length();
+ let new_active = self.active_memory.get() + buf_size;
+ self.active_memory.set(new_active);
+
+ if new_active > self.peak_memory.get() {
+ self.peak_memory.set(new_active);
+ }
+
+ buffer
+ }
+
+ fn free(
+ &self,
+ buffer: Self::Buffer,
+ ) {
+ let size = buffer.length();
+ self.active_memory.set(self.active_memory.get().saturating_sub(size));
+ drop(buffer);
+ }
+
+ fn peak_memory(&self) -> usize {
+ self.peak_memory.get()
+ }
+
+ fn cache_memory(&self) -> usize {
+ 0
+ }
+
+ fn active_memory(&self) -> usize {
+ self.active_memory.get()
+ }
+}
diff --git a/crates/uzu/tests/allocator_test/heap_allocator.rs b/crates/uzu/tests/allocator_test/heap_allocator.rs
new file mode 100644
index 000000000..2793328eb
--- /dev/null
+++ b/crates/uzu/tests/allocator_test/heap_allocator.rs
@@ -0,0 +1,82 @@
+use std::cell::Cell;
+
+use metal::{
+ MTLBuffer, MTLDevice, MTLDeviceExt, MTLHeap, MTLHeapDescriptor,
+ MTLResourceOptions, MTLStorageMode,
+};
+use objc2::{rc::Retained, runtime::ProtocolObject};
+
+use super::allocator_trait::AllocatorTrait;
+
+pub struct MTLHeapAllocator {
+ heap: Retained<ProtocolObject<dyn MTLHeap>>,
+ peak_used: Cell<usize>,
+ active_memory: Cell<usize>,
+}
+
+impl MTLHeapAllocator {
+ pub fn new(heap_size: usize) -> Self {
+ let device =
+ <dyn MTLDevice>::system_default().expect("No Metal device");
+
+ let descriptor = MTLHeapDescriptor::new();
+ descriptor.set_size(heap_size);
+ descriptor.set_storage_mode(MTLStorageMode::Shared);
+
+ let heap = device
+ .new_heap_with_descriptor(&descriptor)
+ .expect("Failed to create heap");
+
+ Self {
+ heap,
+ peak_used: Cell::new(0),
+ active_memory: Cell::new(0),
+ }
+ }
+}
+
+impl AllocatorTrait for MTLHeapAllocator {
+ type Buffer = Retained<ProtocolObject<dyn MTLBuffer>>;
+
+ fn alloc(
+ &self,
+ size: usize,
+ ) -> Self::Buffer {
+ let buffer = self
+ .heap
+ .new_buffer(size, MTLResourceOptions::STORAGE_MODE_SHARED)
+ .expect("Failed to create buffer from heap");
+
+ let buf_size = buffer.length();
+ let new_active = self.active_memory.get() + buf_size;
+ self.active_memory.set(new_active);
+
+ let used = self.heap.used_size();
+ if used > self.peak_used.get() {
+ self.peak_used.set(used);
+ }
+
+ buffer
+ }
+
+ fn free(
+ &self,
+ buffer: Self::Buffer,
+ ) {
+ let size = buffer.length();
+ self.active_memory.set(self.active_memory.get().saturating_sub(size));
+ drop(buffer);
+ }
+
+ fn peak_memory(&self) -> usize {
+ self.peak_used.get()
+ }
+
+ fn cache_memory(&self) -> usize {
+ 0
+ }
+
+ fn active_memory(&self) -> usize {
+ self.active_memory.get()
+ }
+}
diff --git a/crates/uzu/tests/allocator_test/inference_simulation.rs b/crates/uzu/tests/allocator_test/inference_simulation.rs
new file mode 100644
index 000000000..cd92d610d
--- /dev/null
+++ b/crates/uzu/tests/allocator_test/inference_simulation.rs
@@ -0,0 +1,210 @@
+use std::time::Instant;
+
+use super::{
+ allocation_stats::AllocationStats, allocator_trait::AllocatorTrait,
+ model_config::ModelConfig,
+};
+
+const F16_SIZE: usize = 2;
+
+#[derive(Clone, Copy, Debug)]
+pub struct MemorySnapshot {
+ pub step: usize,
+ pub active_memory: usize,
+ pub cache_memory: usize,
+ pub peak_memory: usize,
+}
+
+pub struct InferenceSimulation<'a, A: AllocatorTrait> {
+ allocator: &'a A,
+ config: ModelConfig,
+ pub stats: AllocationStats,
+ pub current_prefix_len: usize,
+ step_counter: usize,
+}
+
+impl<'a, A: AllocatorTrait> InferenceSimulation<'a, A> {
+ pub fn new(
+ allocator: &'a A,
+ config: ModelConfig,
+ ) -> Self {
+ Self {
+ allocator,
+ config,
+ stats: AllocationStats::default(),
+ current_prefix_len: 0,
+ step_counter: 0,
+ }
+ }
+
+ pub fn capture_snapshot(&self) -> MemorySnapshot {
+ MemorySnapshot {
+ step: self.step_counter,
+ active_memory: self.allocator.active_memory(),
+ cache_memory: self.allocator.cache_memory(),
+ peak_memory: self.allocator.peak_memory(),
+ }
+ }
+
+ fn alloc_scratch(
+ &mut self,
+ size: usize,
+ ) -> A::Buffer {
+ let start = Instant::now();
+ let result = self.allocator.alloc(size);
+ self.stats.record_alloc(size, start.elapsed());
+ result
+ }
+
+ fn free(
+ &mut self,
+ buffer: A::Buffer,
+ ) {
+ let start = Instant::now();
+ self.allocator.free(buffer);
+ self.stats.record_free(start.elapsed());
+ }
+
+ fn simulate_embedding_kernel(
+ &mut self,
+ batch_size: usize,
+ ) -> A::Buffer {
+ let size = batch_size * self.config.model_dim * F16_SIZE;
+ self.alloc_scratch(size)
+ }
+
+ fn simulate_attention_kernel(
+ &mut self,
+ batch_size: usize,
+ ) -> Vec<A::Buffer> {
+ let mut buffers = Vec::new();
+
+ let qkv_size = batch_size * self.config.qkv_dim() * F16_SIZE;
+ buffers.push(self.alloc_scratch(qkv_size));
+
+ let bias_size =
+ batch_size * (batch_size + self.current_prefix_len) * F16_SIZE;
+ buffers.push(self.alloc_scratch(bias_size));
+
+ let rotated_q_size = self.config.num_heads
+ * batch_size
+ * self.config.head_dim
+ * F16_SIZE;
+ buffers.push(self.alloc_scratch(rotated_q_size));
+
+ let rotated_k_size = self.config.num_groups
+ * batch_size
+ * self.config.head_dim
+ * F16_SIZE;
+ buffers.push(self.alloc_scratch(rotated_k_size));
+
+ let attn_out_size = batch_size
+ * self.config.num_heads
+ * self.config.head_dim
+ * F16_SIZE;
+ buffers.push(self.alloc_scratch(attn_out_size));
+
+ buffers
+ }
+
+ fn simulate_mlp_kernel(
+ &mut self,
+ batch_size: usize,
+ ) -> Vec<A::Buffer> {
+ let mut buffers = Vec::new();
+
+ let up_size = batch_size * 2 * self.config.hidden_dim * F16_SIZE;
+ buffers.push(self.alloc_scratch(up_size));
+
+ let hidden_size = batch_size * self.config.hidden_dim * F16_SIZE;
+ buffers.push(self.alloc_scratch(hidden_size));
+
+ buffers
+ }
+
+ fn simulate_logits_kernel(
+ &mut self,
+ batch_size: usize,
+ ) -> A::Buffer {
+ let size = batch_size * self.config.vocab_size * F16_SIZE;
+ self.alloc_scratch(size)
+ }
+
+ fn simulate_forward_pass(
+ &mut self,
+ batch_size: usize,
+ ) {
+ self.step_counter += 1;
+
+ let main = self.simulate_embedding_kernel(batch_size);
+
+ for _layer in 0..self.config.num_layers {
+ let attn_buffers = self.simulate_attention_kernel(batch_size);
+ for buf in attn_buffers {
+ self.free(buf);
+ }
+
+ let mlp_buffers = self.simulate_mlp_kernel(batch_size);
+ for buf in mlp_buffers {
+ self.free(buf);
+ }
+ }
+
+ let logits = self.simulate_logits_kernel(batch_size);
+ self.free(logits);
+ self.free(main);
+ }
+
+ pub fn run_prefill(
+ &mut self,
+ prompt_length: usize,
+ step_size: usize,
+ ) -> Vec<MemorySnapshot> {
+ let mut snapshots = Vec::new();
+ let mut remaining = prompt_length;
+
+ while remaining > 0 {
+ let batch_size = remaining.min(step_size);
+ self.simulate_forward_pass(batch_size);
+ self.current_prefix_len += batch_size;
+ remaining = remaining.saturating_sub(step_size);
+ snapshots.push(self.capture_snapshot());
+ }
+
+ snapshots
+ }
+
+ pub fn run_generation(
+ &mut self,
+ num_tokens: usize,
+ ) -> Vec<MemorySnapshot> {
+ let mut snapshots = Vec::new();
+
+ for _ in 0..num_tokens {
+ self.simulate_forward_pass(1);
+ self.current_prefix_len += 1;
+ snapshots.push(self.capture_snapshot());
+ }
+
+ snapshots
+ }
+
+ pub fn run_generation_with_sampling(
+ &mut self,
+ num_tokens: usize,
+ snapshot_interval: usize,
+ ) -> Vec<MemorySnapshot> {
+ let mut snapshots = Vec::new();
+
+ for i in 0..num_tokens {
+ self.simulate_forward_pass(1);
+ self.current_prefix_len += 1;
+
+ if i % snapshot_interval == 0 || i == num_tokens - 1 {
+ snapshots.push(self.capture_snapshot());
+ }
+ }
+
+ snapshots
+ }
+}
diff --git a/crates/uzu/tests/allocator_test/main.rs b/crates/uzu/tests/allocator_test/main.rs
new file mode 100644
index 000000000..8cc4c59e1
--- /dev/null
+++ b/crates/uzu/tests/allocator_test/main.rs
@@ -0,0 +1,344 @@
+#![cfg(target_os = "macos")]
+
+mod allocation_stats;
+mod allocator_trait;
+mod caching_allocator;
+mod device_allocator;
+mod heap_allocator;
+mod inference_simulation;
+mod model_config;
+mod simulation_result;
+
+use caching_allocator::CachingAllocator;
+use device_allocator::DeviceAllocator;
+use heap_allocator::MTLHeapAllocator;
+use model_config::ModelConfig;
+use simulation_result::{
+ run_long_simulation, run_simulation, run_simulation_with_config,
+};
+use uzu::backends::common::Context;
+
+const KB: f64 = 1024.0;
+const MB: f64 = 1024.0 * 1024.0;
+
+fn print_comparison(
+ name: &str,
+ uzu_result: &simulation_result::SimulationResult,
+ heap_result: &simulation_result::SimulationResult,
+ device_result: &simulation_result::SimulationResult,
+) {
+ println!("\n=== {} ===", name);
+ println!(
+ "{:<20} {:>14} {:>14} {:>14}",
+ "Metric", "UZU Caching", "Heap", "Device"
+ );
+ println!("{:-<62}", "");
+
+ println!(
+ "{:<20} {:>14} {:>14} {:>14}",
+ "Allocations",
+ uzu_result.stats.total_allocs,
+ heap_result.stats.total_allocs,
+ device_result.stats.total_allocs
+ );
+
+ println!(
+ "{:<20} {:>14} {:>14} {:>14}",
+ "Frees",
+ uzu_result.stats.total_frees,
+ heap_result.stats.total_frees,
+ device_result.stats.total_frees
+ );
+
+ println!(
+ "{:<20} {:>13.2}µs {:>13.2}µs {:>13.2}µs",
+ "Avg alloc time",
+ uzu_result.stats.avg_alloc_time_us(),
+ heap_result.stats.avg_alloc_time_us(),
+ device_result.stats.avg_alloc_time_us()
+ );
+
+ println!(
+ "{:<20} {:>13.2}µs {:>13.2}µs {:>13.2}µs",
+ "Avg free time",
+ uzu_result.stats.avg_free_time_us(),
+ heap_result.stats.avg_free_time_us(),
+ device_result.stats.avg_free_time_us()
+ );
+
+ println!(
+ "{:<20} {:>12.2} MB {:>12.2} MB {:>12.2} MB",
+ "Bytes requested",
+ uzu_result.stats.total_bytes_requested as f64 / MB,
+ heap_result.stats.total_bytes_requested as f64 / MB,
+ device_result.stats.total_bytes_requested as f64 / MB
+ );
+
+ println!(
+ "{:<20} {:>12.2} MB {:>12.2} MB {:>12.2} MB",
+ "Peak memory",
+ uzu_result.peak_memory_bytes as f64 / MB,
+ heap_result.peak_memory_bytes as f64 / MB,
+ device_result.peak_memory_bytes as f64 / MB
+ );
+
+ println!(
+ "{:<20} {:>12.2} KB {:>12.2} KB {:>12.2} KB",
+ "Final cache",
+ uzu_result.cache_memory_bytes as f64 / KB,
+ heap_result.cache_memory_bytes as f64 / KB,
+ device_result.cache_memory_bytes as f64 / KB
+ );
+
+ println!();
+
+ let device_time = device_result.stats.avg_alloc_time_us().max(0.001);
+ let uzu_speedup =
+ device_time / uzu_result.stats.avg_alloc_time_us().max(0.001);
+ let heap_speedup =
+ device_time / heap_result.stats.avg_alloc_time_us().max(0.001);
+
+ println!(
+ "Alloc speedup vs device: UZU {:.1}x, Heap {:.1}x",
+ uzu_speedup, heap_speedup
+ );
+}
+
+fn print_memory_over_time(
+ name: &str,
+ snapshots: &[inference_simulation::MemorySnapshot],
+ sample_count: usize,
+) {
+ if snapshots.is_empty() {
+ return;
+ }
+
+ println!("\n--- {} Memory Over Time (sampled) ---", name);
+ println!("{:>8} {:>12} {:>12} {:>12}", "Step", "Active", "Cache", "Peak");
+
+ let step = (snapshots.len() / sample_count).max(1);
+ for (i, snapshot) in snapshots.iter().enumerate() {
+ if i % step == 0 || i == snapshots.len() - 1 {
+ println!(
+ "{:>8} {:>10.2}MB {:>10.2}MB {:>10.2}MB",
+ snapshot.step,
+ snapshot.active_memory as f64 / MB,
+ snapshot.cache_memory as f64 / MB,
+ snapshot.peak_memory as f64 / MB
+ );
+ }
+ }
+}
+
+#[test]
+fn test_allocator_comparison() {
+ let prompt_length = 256;
+ let generate_length = 2048;
+ let prefill_step_size = 512;
+
+ println!("\n========================================");
+ println!(" Short Conversation Test");
+ println!(" {}p + {}g tokens", prompt_length, generate_length);
+ println!("========================================");
+
+ let uzu_allocator = CachingAllocator::new();
+ let uzu_result = run_simulation(
+ &uzu_allocator,
+ prompt_length,
+ generate_length,
+ prefill_step_size,
+ );
+
+ let heap_allocator = MTLHeapAllocator::new(512 * 1024 * 1024);
+ let heap_result = run_simulation(
+ &heap_allocator,
+ prompt_length,
+ generate_length,
+ prefill_step_size,
+ );
+
+ let device_allocator = DeviceAllocator::new();
+ let device_result = run_simulation(
+ &device_allocator,
+ prompt_length,
+ generate_length,
+ prefill_step_size,
+ );
+
+ print_comparison(
+ "Short Conversation Results",
+ &uzu_result,
+ &heap_result,
+ &device_result,
+ );
+
+ assert_eq!(
+ uzu_result.stats.total_allocs, uzu_result.stats.total_frees,
+ "UZU allocator: allocs should equal frees"
+ );
+ assert_eq!(
+ heap_result.stats.total_allocs, heap_result.stats.total_frees,
+ "Heap allocator: allocs should equal frees"
+ );
+ assert_eq!(
+ device_result.stats.total_allocs, device_result.stats.total_frees,
+ "Device allocator: allocs should equal frees"
+ );
+}
+
+#[test]
+fn test_long_conversation_comparison() {
+ let num_conversations = 10;
+ let avg_prompt_length = 512;
+ let avg_generate_length = 1024;
+ let prefill_step_size = 512;
+ let snapshot_interval = 100;
+
+ println!("\n========================================");
+ println!(" Long Conversation Test");
+ println!(
+ " {} conversations, ~{}p + ~{}g tokens each",
+ num_conversations, avg_prompt_length, avg_generate_length
+ );
+ println!("========================================");
+
+ let gc_limit = 256 * 1024 * 1024;
+ let max_pool = 128 * 1024 * 1024;
+
+ let uzu_allocator = CachingAllocator::new();
+ uzu_allocator.context.allocator().set_gc_limit(gc_limit);
+ uzu_allocator.context.allocator().set_max_pool_size(max_pool);
+
+ let uzu_result = run_long_simulation(
+ &uzu_allocator,
+ num_conversations,
+ avg_prompt_length,
+ avg_generate_length,
+ prefill_step_size,
+ snapshot_interval,
+ );
+
+ let heap_allocator = MTLHeapAllocator::new(1024 * 1024 * 1024);
+ let heap_result = run_long_simulation(
+ &heap_allocator,
+ num_conversations,
+ avg_prompt_length,
+ avg_generate_length,
+ prefill_step_size,
+ snapshot_interval,
+ );
+
+ let device_allocator = DeviceAllocator::new();
+ let device_result = run_long_simulation(
+ &device_allocator,
+ num_conversations,
+ avg_prompt_length,
+ avg_generate_length,
+ prefill_step_size,
+ snapshot_interval,
+ );
+
+ print_comparison(
+ "Long Conversation Results",
+ &uzu_result,
+ &heap_result,
+ &device_result,
+ );
+
+ print_memory_over_time("UZU Caching", &uzu_result.snapshots, 15);
+ print_memory_over_time("Heap", &heap_result.snapshots, 15);
+ print_memory_over_time("Device (baseline)", &device_result.snapshots, 15);
+
+ println!("\n--- Memory Efficiency Summary ---");
+ println!(
+ "UZU peak/requested ratio: {:.2}%",
+ (uzu_result.peak_memory_bytes as f64
+ / uzu_result.stats.total_bytes_requested as f64)
+ * 100.0
+ );
+ println!(
+ "Heap peak/requested ratio: {:.2}%",
+ (heap_result.peak_memory_bytes as f64
+ / heap_result.stats.total_bytes_requested as f64)
+ * 100.0
+ );
+ println!(
+ "Device peak/requested ratio: {:.2}%",
+ (device_result.peak_memory_bytes as f64
+ / device_result.stats.total_bytes_requested as f64)
+ * 100.0
+ );
+
+ assert_eq!(
+ uzu_result.stats.total_allocs, uzu_result.stats.total_frees,
+ "UZU allocator: allocs should equal frees"
+ );
+ assert_eq!(
+ device_result.stats.total_allocs, device_result.stats.total_frees,
+ "Device allocator: allocs should equal frees"
+ );
+}
+
+#[test]
+fn test_multi_model_comparison() {
+ let prompt_length = 256;
+ let generate_length = 512;
+ let prefill_step_size = 256;
+
+ println!("\n========================================");
+ println!(" Multi-Model Comparison Test");
+ println!(" {}p + {}g tokens per model", prompt_length, generate_length);
+ println!("========================================");
+
+ println!(
+ "\n{:<15} {:>12} {:>12} {:>12}",
+ "Model", "UZU (µs)", "Heap (µs)", "Device (µs)"
+ );
+ println!("{:-<55}", "");
+
+ for config in ModelConfig::all_models() {
+ let uzu_allocator = CachingAllocator::new();
+ let uzu_result = run_simulation_with_config(
+ &uzu_allocator,
+ config,
+ prompt_length,
+ generate_length,
+ prefill_step_size,
+ );
+
+ let heap_allocator = MTLHeapAllocator::new(512 * 1024 * 1024);
+ let heap_result = run_simulation_with_config(
+ &heap_allocator,
+ config,
+ prompt_length,
+ generate_length,
+ prefill_step_size,
+ );
+
+ let device_allocator = DeviceAllocator::new();
+ let device_result = run_simulation_with_config(
+ &device_allocator,
+ config,
+ prompt_length,
+ generate_length,
+ prefill_step_size,
+ );
+
+ println!(
+ "{:<15} {:>12.2} {:>12.2} {:>12.2}",
+ config.name,
+ uzu_result.stats.avg_alloc_time_us(),
+ heap_result.stats.avg_alloc_time_us(),
+ device_result.stats.avg_alloc_time_us()
+ );
+
+ assert_eq!(
+ uzu_result.stats.total_allocs, uzu_result.stats.total_frees,
+ "{}: UZU allocs should equal frees",
+ config.name
+ );
+ }
+
+ println!();
+ println!("Summary: Lower allocation time is better.");
+}
diff --git a/crates/uzu/tests/allocator_test/mod.rs b/crates/uzu/tests/allocator_test/mod.rs
new file mode 100644
index 000000000..32883483e
--- /dev/null
+++ b/crates/uzu/tests/allocator_test/mod.rs
@@ -0,0 +1,279 @@
+#![cfg(target_os = "macos")]
+
+mod allocation_stats;
+mod allocator_trait;
+mod caching_allocator;
+mod device_allocator;
+mod heap_allocator;
+mod inference_simulation;
+mod mlx_allocator;
+mod model_config;
+mod simulation_result;
+
+use caching_allocator::CachingAllocator;
+use device_allocator::DeviceAllocator;
+use heap_allocator::MTLHeapAllocator;
+use mlx_allocator::MlxAllocator;
+use simulation_result::{run_long_simulation, run_simulation};
+use uzu::backends::common::Context;
+
+const KB: f64 = 1024.0;
+const MB: f64 = 1024.0 * 1024.0;
+const GB: f64 = 1024.0 * 1024.0 * 1024.0;
+
+fn print_three_way_comparison(
+ name: &str,
+ uzu_result: &simulation_result::SimulationResult,
+ mlx_result: &simulation_result::SimulationResult,
+ heap_result: &simulation_result::SimulationResult,
+) {
+ println!("\n=== {} ===", name);
+ println!(
+ "{:<25} {:>15} {:>15} {:>15}",
+ "Metric", "UZU Caching", "MLX-style LRU", "Heap (baseline)"
+ );
+ println!("{:-<72}", "");
+
+ println!(
+ "{:<25} {:>15} {:>15} {:>15}",
+ "Allocations",
+ uzu_result.stats.total_allocs,
+ mlx_result.stats.total_allocs,
+ heap_result.stats.total_allocs
+ );
+
+ println!(
+ "{:<25} {:>15} {:>15} {:>15}",
+ "Frees",
+ uzu_result.stats.total_frees,
+ mlx_result.stats.total_frees,
+ heap_result.stats.total_frees
+ );
+
+ println!(
+ "{:<25} {:>14.2}µs {:>14.2}µs {:>14.2}µs",
+ "Avg alloc time",
+ uzu_result.stats.avg_alloc_time_us(),
+ mlx_result.stats.avg_alloc_time_us(),
+ heap_result.stats.avg_alloc_time_us()
+ );
+
+ println!(
+ "{:<25} {:>14.2}µs {:>14.2}µs {:>14.2}µs",
+ "Avg free time",
+ uzu_result.stats.avg_free_time_us(),
+ mlx_result.stats.avg_free_time_us(),
+ heap_result.stats.avg_free_time_us()
+ );
+
+ println!(
+ "{:<25} {:>13.2} MB {:>13.2} MB {:>13.2} MB",
+ "Bytes requested",
+ uzu_result.stats.total_bytes_requested as f64 / MB,
+ mlx_result.stats.total_bytes_requested as f64 / MB,
+ heap_result.stats.total_bytes_requested as f64 / MB
+ );
+
+ println!(
+ "{:<25} {:>13.2} MB {:>13.2} MB {:>13.2} MB",
+ "Peak memory",
+ uzu_result.peak_memory_bytes as f64 / MB,
+ mlx_result.peak_memory_bytes as f64 / MB,
+ heap_result.peak_memory_bytes as f64 / MB
+ );
+
+ println!(
+ "{:<25} {:>13.2} KB {:>13.2} KB {:>13.2} KB",
+ "Final cache",
+ uzu_result.cache_memory_bytes as f64 / KB,
+ mlx_result.cache_memory_bytes as f64 / KB,
+ heap_result.cache_memory_bytes as f64 / KB
+ );
+
+ println!();
+
+ let uzu_speedup = heap_result.stats.avg_alloc_time_us()
+ / uzu_result.stats.avg_alloc_time_us().max(0.001);
+ let mlx_speedup = heap_result.stats.avg_alloc_time_us()
+ / mlx_result.stats.avg_alloc_time_us().max(0.001);
+
+ println!(
+ "Alloc speedup vs heap: UZU {:.1}x, MLX {:.1}x",
+ uzu_speedup, mlx_speedup
+ );
+}
+
+fn print_memory_over_time(
+ name: &str,
+ snapshots: &[inference_simulation::MemorySnapshot],
+ sample_count: usize,
+) {
+ if snapshots.is_empty() {
+ return;
+ }
+
+ println!("\n--- {} Memory Over Time (sampled) ---", name);
+ println!("{:>8} {:>12} {:>12} {:>12}", "Step", "Active", "Cache", "Peak");
+
+ let step = (snapshots.len() / sample_count).max(1);
+ for (i, snapshot) in snapshots.iter().enumerate() {
+ if i % step == 0 || i == snapshots.len() - 1 {
+ println!(
+ "{:>8} {:>10.2}MB {:>10.2}MB {:>10.2}MB",
+ snapshot.step,
+ snapshot.active_memory as f64 / MB,
+ snapshot.cache_memory as f64 / MB,
+ snapshot.peak_memory as f64 / MB
+ );
+ }
+ }
+}
+
+#[test]
+fn test_allocator_comparison() {
+ let prompt_length = 256;
+ let generate_length = 2048;
+ let prefill_step_size = 512;
+
+ println!("\n========================================");
+ println!(" Short Conversation Test");
+ println!(" {}p + {}g tokens", prompt_length, generate_length);
+ println!("========================================");
+
+ let uzu_allocator = CachingAllocator::new();
+ let uzu_result = run_simulation(
+ &uzu_allocator,
+ prompt_length,
+ generate_length,
+ prefill_step_size,
+ );
+
+ let mlx_allocator = MlxAllocator::new(512 * 1024 * 1024, 256 * 1024 * 1024);
+ let mlx_result = run_simulation(
+ &mlx_allocator,
+ prompt_length,
+ generate_length,
+ prefill_step_size,
+ );
+
+ let heap_allocator = MTLHeapAllocator::new(512 * 1024 * 1024);
+ let heap_result = run_simulation(
+ &heap_allocator,
+ prompt_length,
+ generate_length,
+ prefill_step_size,
+ );
+
+ print_three_way_comparison(
+ "Short Conversation Results",
+ &uzu_result,
+ &mlx_result,
+ &heap_result,
+ );
+
+ assert_eq!(
+ uzu_result.stats.total_allocs, uzu_result.stats.total_frees,
+ "UZU allocator: allocs should equal frees"
+ );
+ assert_eq!(
+ mlx_result.stats.total_allocs, mlx_result.stats.total_frees,
+ "MLX allocator: allocs should equal frees"
+ );
+ assert_eq!(
+ heap_result.stats.total_allocs, heap_result.stats.total_frees,
+ "Heap allocator: allocs should equal frees"
+ );
+}
+
+#[test]
+fn test_long_conversation_comparison() {
+ let num_conversations = 10;
+ let avg_prompt_length = 512;
+ let avg_generate_length = 1024;
+ let prefill_step_size = 512;
+ let snapshot_interval = 100;
+
+ println!("\n========================================");
+ println!(" Long Conversation Test");
+ println!(
+ " {} conversations, ~{}p + ~{}g tokens each",
+ num_conversations, avg_prompt_length, avg_generate_length
+ );
+ println!("========================================");
+
+ let gc_limit = 256 * 1024 * 1024;
+ let max_pool = 128 * 1024 * 1024;
+
+ let uzu_allocator = CachingAllocator::new();
+ uzu_allocator.context.allocator().set_gc_limit(gc_limit);
+ uzu_allocator.context.allocator().set_max_pool_size(max_pool);
+
+ let uzu_result = run_long_simulation(
+ &uzu_allocator,
+ num_conversations,
+ avg_prompt_length,
+ avg_generate_length,
+ prefill_step_size,
+ snapshot_interval,
+ );
+
+ let mlx_allocator = MlxAllocator::new(gc_limit, max_pool);
+ let mlx_result = run_long_simulation(
+ &mlx_allocator,
+ num_conversations,
+ avg_prompt_length,
+ avg_generate_length,
+ prefill_step_size,
+ snapshot_interval,
+ );
+
+ let heap_allocator = MTLHeapAllocator::new(1024 * 1024 * 1024);
+ let heap_result = run_long_simulation(
+ &heap_allocator,
+ num_conversations,
+ avg_prompt_length,
+ avg_generate_length,
+ prefill_step_size,
+ snapshot_interval,
+ );
+
+ print_three_way_comparison(
+ "Long Conversation Results",
+ &uzu_result,
+ &mlx_result,
+ &heap_result,
+ );
+
+ print_memory_over_time("UZU Caching", &uzu_result.snapshots, 15);
+ print_memory_over_time("MLX-style LRU", &mlx_result.snapshots, 15);
+ print_memory_over_time("Heap (baseline)", &heap_result.snapshots, 15);
+
+ println!("\n--- Memory Efficiency Summary ---");
+ println!(
+ "UZU peak/requested ratio: {:.2}%",
+ (uzu_result.peak_memory_bytes as f64
+ / uzu_result.stats.total_bytes_requested as f64)
+ * 100.0
+ );
+ println!(
+ "MLX peak/requested ratio: {:.2}%",
+ (mlx_result.peak_memory_bytes as f64
+ / mlx_result.stats.total_bytes_requested as f64)
+ * 100.0
+ );
+ println!(
+ "Heap peak/requested ratio: {:.2}%",
+ (heap_result.peak_memory_bytes as f64
+ / heap_result.stats.total_bytes_requested as f64)
+ * 100.0
+ );
+
+ assert_eq!(
+ uzu_result.stats.total_allocs, uzu_result.stats.total_frees,
+ "UZU allocator: allocs should equal frees"
+ );
+ assert_eq!(
+ mlx_result.stats.total_allocs, mlx_result.stats.total_frees,
+ "MLX allocator: allocs should equal frees"
+ );
+}
diff --git a/crates/uzu/tests/allocator_test/model_config.rs b/crates/uzu/tests/allocator_test/model_config.rs
new file mode 100644
index 000000000..f398da479
--- /dev/null
+++ b/crates/uzu/tests/allocator_test/model_config.rs
@@ -0,0 +1,106 @@
+#[derive(Clone, Copy)]
+pub struct ModelConfig {
+ pub name: &'static str,
+ pub model_dim: usize,
+ pub num_heads: usize,
+ pub num_groups: usize,
+ pub head_dim: usize,
+ pub hidden_dim: usize,
+ pub vocab_size: usize,
+ pub num_layers: usize,
+}
+
+impl ModelConfig {
+ pub fn llama_3_2_1b() -> Self {
+ Self {
+ name: "Llama-3.2-1B",
+ model_dim: 2048,
+ num_heads: 32,
+ num_groups: 8,
+ head_dim: 64,
+ hidden_dim: 8192,
+ vocab_size: 128256,
+ num_layers: 16,
+ }
+ }
+
+ pub fn llama_3_2_3b() -> Self {
+ Self {
+ name: "Llama-3.2-3B",
+ model_dim: 3072,
+ num_heads: 24,
+ num_groups: 8,
+ head_dim: 128,
+ hidden_dim: 8192,
+ vocab_size: 128256,
+ num_layers: 28,
+ }
+ }
+
+ pub fn qwen3_4b() -> Self {
+ Self {
+ name: "Qwen3-4B",
+ model_dim: 2560,
+ num_heads: 32,
+ num_groups: 8,
+ head_dim: 128,
+ hidden_dim: 9728,
+ vocab_size: 151936,
+ num_layers: 36,
+ }
+ }
+
+ pub fn qwen3_8b() -> Self {
+ Self {
+ name: "Qwen3-8B",
+ model_dim: 4096,
+ num_heads: 32,
+ num_groups: 8,
+ head_dim: 128,
+ hidden_dim: 12288,
+ vocab_size: 151936,
+ num_layers: 36,
+ }
+ }
+
+ pub fn gemma_3_4b() -> Self {
+ Self {
+ name: "Gemma-3-4B",
+ model_dim: 2560,
+ num_heads: 8,
+ num_groups: 4,
+ head_dim: 256,
+ hidden_dim: 10240,
+ vocab_size: 262208,
+ num_layers: 34,
+ }
+ }
+
+ pub fn qwen3_14b() -> Self {
+ Self {
+ name: "Qwen3-14B",
+ model_dim: 5120,
+ num_heads: 40,
+ num_groups: 8,
+ head_dim: 128,
+ hidden_dim: 17408,
+ vocab_size: 151936,
+ num_layers: 40,
+ }
+ }
+
+ pub fn all_models() -> Vec<Self> {
+ vec![
+ Self::llama_3_2_1b(),
+ Self::llama_3_2_3b(),
+ Self::qwen3_4b(),
+ Self::qwen3_8b(),
+ Self::gemma_3_4b(),
+ Self::qwen3_14b(),
+ ]
+ }
+
+ pub fn qkv_dim(&self) -> usize {
+ (2 * self.num_groups + self.num_heads) * self.head_dim
+ }
+}
diff --git a/crates/uzu/tests/allocator_test/simulation_result.rs b/crates/uzu/tests/allocator_test/simulation_result.rs
new file mode 100644
index 000000000..822ae6e7e
--- /dev/null
+++ b/crates/uzu/tests/allocator_test/simulation_result.rs
@@ -0,0 +1,116 @@
+use super::{
+ allocation_stats::AllocationStats,
+ allocator_trait::AllocatorTrait,
+ inference_simulation::{InferenceSimulation, MemorySnapshot},
+ model_config::ModelConfig,
+};
+
+pub struct SimulationResult {
+ pub stats: AllocationStats,
+ pub peak_memory_bytes: usize,
+ pub cache_memory_bytes: usize,
+ #[allow(dead_code)]
+ pub final_prefix_len: usize,
+ pub snapshots: Vec<MemorySnapshot>,
+}
+
+pub fn run_simulation<A: AllocatorTrait>(
+ allocator: &A,
+ prompt_length: usize,
+ generate_length: usize,
+ prefill_step_size: usize,
+) -> SimulationResult {
+ let config = ModelConfig::llama_3_2_1b();
+ let mut sim = InferenceSimulation::new(allocator, config);
+
+ let mut snapshots = sim.run_prefill(prompt_length, prefill_step_size);
+ let gen_snapshots = sim.run_generation(generate_length);
+ snapshots.extend(gen_snapshots);
+
+ SimulationResult {
+ stats: sim.stats,
+ peak_memory_bytes: allocator.peak_memory(),
+ cache_memory_bytes: allocator.cache_memory(),
+ final_prefix_len: sim.current_prefix_len,
+ snapshots,
+ }
+}
+
+pub fn run_long_simulation<A: AllocatorTrait>(
+ allocator: &A,
+ num_conversations: usize,
+ avg_prompt_length: usize,
+ avg_generate_length: usize,
+ prefill_step_size: usize,
+ snapshot_interval: usize,
+) -> SimulationResult {
+ let config = ModelConfig::llama_3_2_1b();
+ run_long_simulation_with_config(
+ allocator,
+ config,
+ num_conversations,
+ avg_prompt_length,
+ avg_generate_length,
+ prefill_step_size,
+ snapshot_interval,
+ )
+}
+
+pub fn run_simulation_with_config<A: AllocatorTrait>(
+ allocator: &A,
+ config: ModelConfig,
+ prompt_length: usize,
+ generate_length: usize,
+ prefill_step_size: usize,
+) -> SimulationResult {
+ let mut sim = InferenceSimulation::new(allocator, config);
+
+ let mut snapshots = sim.run_prefill(prompt_length, prefill_step_size);
+ let gen_snapshots = sim.run_generation(generate_length);
+ snapshots.extend(gen_snapshots);
+
+ SimulationResult {
+ stats: sim.stats,
+ peak_memory_bytes: allocator.peak_memory(),
+ cache_memory_bytes: allocator.cache_memory(),
+ final_prefix_len: sim.current_prefix_len,
+ snapshots,
+ }
+}
+
+pub fn run_long_simulation_with_config<A: AllocatorTrait>(
+ allocator: &A,
+ config: ModelConfig,
+ num_conversations: usize,
+ avg_prompt_length: usize,
+ avg_generate_length: usize,
+ prefill_step_size: usize,
+ snapshot_interval: usize,
+) -> SimulationResult {
+ let mut sim = InferenceSimulation::new(allocator, config);
+ let mut all_snapshots = Vec::new();
+
+ for conv_idx in 0..num_conversations {
+ let prompt_len =
+ avg_prompt_length + (conv_idx % 3) * (avg_prompt_length / 4);
+ let gen_len =
+ avg_generate_length + (conv_idx % 5) * (avg_generate_length / 8);
+
+ let prefill_snapshots = sim.run_prefill(prompt_len, prefill_step_size);
+ all_snapshots.extend(prefill_snapshots);
+
+ let gen_snapshots =
+ sim.run_generation_with_sampling(gen_len, snapshot_interval);
+ all_snapshots.extend(gen_snapshots);
+
+ sim.current_prefix_len = 0;
+ }
+
+ SimulationResult {
+ stats: sim.stats,
+ peak_memory_bytes: allocator.peak_memory(),
+ cache_memory_bytes: allocator.cache_memory(),
+ final_prefix_len: sim.current_prefix_len,
+ snapshots: all_snapshots,
+ }
+}
diff --git a/crates/uzu/tests/attention_kernel_test.rs b/crates/uzu/tests/attention_kernel_test.rs
index 9aa9f3234..88ea91700 100644
--- a/crates/uzu/tests/attention_kernel_test.rs
+++ b/crates/uzu/tests/attention_kernel_test.rs
@@ -6,17 +6,20 @@ use std::mem::size_of;
use bytemuck;
use metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue, MTLDevice, MTLDeviceExt,
- MTLResourceOptions,
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue,
+ MTLDeviceExt, MTLResourceOptions,
};
use ndarray::{Array3, Array4, s};
-use uzu::backends::metal::{
- KernelDataType, MTLContext, ProtocolObject, Retained,
- kernel::attention::{
- AttentionGemmArguments, AttentionKernel, AttentionKernelVariant,
- AttentionSinglePassArguments, AttentionTwoPassArguments,
+use uzu::backends::{
+ common::Context,
+ metal::{
+ KernelDataType, MTLContext, ProtocolObject, Retained,
+ kernel::attention::{
+ AttentionGemmArguments, AttentionKernel, AttentionKernelVariant,
+ AttentionSinglePassArguments, AttentionTwoPassArguments,
+ },
+ metal_extensions::CommandBufferTimingExt,
},
- metal_extensions::CommandBufferTimingExt,
};
fn reference_attention(
@@ -41,7 +44,8 @@ fn reference_attention(
for repeat in 0..n_repeats {
let q_head = kv_head * n_repeats + repeat;
- let q = scaled_queries.slice(s![b, q_head, .., ..]).to_owned(); // [L, D]
+ let q =
+ scaled_queries.slice(s![b, q_head, .., ..]).to_owned(); // [L, D]
let k = keys.slice(s![b, kv_head, .., ..]).to_owned(); // [L_kv, D]
let v = values.slice(s![b, kv_head, .., ..]).to_owned(); // [L_kv, D]
@@ -49,8 +53,13 @@ fn reference_attention(
let mut scores = q.dot(&k.t());
if let Some(mask_data) = mask {
- let mask_slice = mask_data.slice(s![b, .., ..kv_seq_len.min(seq_len)]);
- scores.slice_mut(s![.., ..kv_seq_len.min(seq_len)])
+ let mask_slice = mask_data.slice(s![
+ b,
+ ..,
+ ..kv_seq_len.min(seq_len)
+ ]);
+ scores
+ .slice_mut(s![.., ..kv_seq_len.min(seq_len)])
.zip_mut_with(&mask_slice, |s, &m| *s += m);
}
@@ -75,7 +84,9 @@ fn reference_attention(
// Efficient matrix multiplication: output = scores @ V
let head_output = scores.dot(&v);
- output.slice_mut(s![b, q_head, .., ..]).assign(&head_output);
+ output
+ .slice_mut(s![b, q_head, .., ..])
+ .assign(&head_output);
}
}
} else {
@@ -88,8 +99,10 @@ fn reference_attention(
let mut scores = q.dot(&k.t());
if let Some(mask_data) = mask {
- let mask_slice = mask_data.slice(s![b, .., ..kv_seq_len.min(seq_len)]);
- scores.slice_mut(s![.., ..kv_seq_len.min(seq_len)])
+ let mask_slice =
+ mask_data.slice(s![b, .., ..kv_seq_len.min(seq_len)]);
+ scores
+ .slice_mut(s![.., ..kv_seq_len.min(seq_len)])
.zip_mut_with(&mask_slice, |s, &m| *s += m);
}
@@ -130,7 +143,7 @@ fn create_test_data(
head_dim: usize,
seed: u64,
) -> (Array4<f32>, Array4<f32>, Array4<f32>, Array3<f32>) {
- use rand::{Rng, SeedableRng, rngs::StdRng};
+ use rand::{RngExt, SeedableRng, rngs::StdRng};
let mut rng = StdRng::seed_from_u64(seed);
@@ -646,16 +659,7 @@ fn compare_results(
#[test]
#[ignore]
fn test_single_pass_attention_basic() {
- let device =
- <dyn MTLDevice>::system_default().expect("No Metal device found");
- let command_queue =
- device.new_command_queue().expect("Failed to create command queue");
- let context = match MTLContext::new(device, command_queue) {
- Ok(ctx) => ctx,
- Err(e) => {
- panic!("Failed to create MTLContext: {:?}", e);
- },
- };
+ let context = MTLContext::new().expect("Failed to create MTLContext");
let batch_size = 1;
let num_heads = 4;
@@ -734,16 +738,7 @@ fn test_single_pass_attention_basic() {
#[test]
#[ignore]
fn test_gemm_attention_basic() {
- let device =
- <dyn MTLDevice>::system_default().expect("No Metal device found");
- let command_queue =
- device.new_command_queue().expect("Failed to create command queue");
- let context = match MTLContext::new(device, command_queue) {
- Ok(ctx) => ctx,
- Err(e) => {
- panic!("Failed to create MTLContext: {:?}", e);
- },
- };
+ let context = MTLContext::new().expect("Failed to create MTLContext");
let batch_size = 1;
let num_heads = 4;
@@ -787,16 +782,7 @@ fn test_gemm_attention_basic() {
#[test]
#[ignore]
fn test_gemm_attention_f32_head_dim_128() {
- let device =
- <dyn MTLDevice>::system_default().expect("No Metal device found");
- let command_queue =
- device.new_command_queue().expect("Failed to create command queue");
- let context = match MTLContext::new(device, command_queue) {
- Ok(ctx) => ctx,
- Err(e) => {
- panic!("Failed to create MTLContext: {:?}", e);
- },
- };
+ let context = MTLContext::new().expect("Failed to create MTLContext");
let batch_size = 1;
let num_heads = 4;
@@ -840,16 +826,7 @@ fn test_gemm_attention_f32_head_dim_128() {
#[test]
#[ignore]
fn test_matrix_attention_matches_vector_and_cpu_seq256() {
- let device =
- <dyn MTLDevice>::system_default().expect("No Metal device found");
- let command_queue =
- device.new_command_queue().expect("Failed to create command queue");
- let context = match MTLContext::new(device, command_queue) {
- Ok(ctx) => ctx,
- Err(e) => {
- panic!("Failed to create MTLContext: {:?}", e);
- },
- };
+ let context = MTLContext::new().expect("Failed to create MTLContext");
let batch_size = 1;
let num_heads = 8;
@@ -941,16 +918,7 @@ fn test_matrix_attention_matches_vector_and_cpu_seq256() {
#[test]
#[ignore]
fn test_single_pass_attention_with_mask() {
- let device =
- <dyn MTLDevice>::system_default().expect("No Metal device found");
- let command_queue =
- device.new_command_queue().expect("Failed to create command queue");
- let context = match MTLContext::new(device, command_queue) {
- Ok(ctx) => ctx,
- Err(e) => {
- panic!("Failed to create MTLContext: {:?}", e);
- },
- };
+ let context = MTLContext::new().expect("Failed to create MTLContext");
let batch_size = 1;
let num_heads = 4;
@@ -1020,16 +988,7 @@ fn test_single_pass_attention_with_mask() {
#[test]
#[ignore]
fn test_single_pass_attention_with_sinks() {
- let device =
- <dyn MTLDevice>::system_default().expect("No Metal device found");
- let command_queue =
- device.new_command_queue().expect("Failed to create command queue");
- let context = match MTLContext::new(device, command_queue) {
- Ok(ctx) => ctx,
- Err(e) => {
- panic!("Failed to create MTLContext: {:?}", e);
- },
- };
+ let context = MTLContext::new().expect("Failed to create MTLContext");
let batch_size = 1;
let num_heads = 4;
@@ -1102,16 +1061,7 @@ fn test_single_pass_attention_with_sinks() {
#[test]
#[ignore]
fn test_single_pass_attention_with_sinks_long_sequence() {
- let device =
- <dyn MTLDevice>::system_default().expect("No Metal device found");
- let command_queue =
- device.new_command_queue().expect("Failed to create command queue");
- let context = match MTLContext::new(device, command_queue) {
- Ok(ctx) => ctx,
- Err(e) => {
- panic!("Failed to create MTLContext: {:?}", e);
- },
- };
+ let context = MTLContext::new().expect("Failed to create MTLContext");
let batch_size = 1;
let num_heads = 4;
@@ -1186,16 +1136,7 @@ fn test_single_pass_attention_with_sinks_long_sequence() {
#[test]
#[ignore]
fn test_single_pass_attention_gqa() {
- let device =
- <dyn MTLDevice>::system_default().expect("No Metal device found");
- let command_queue =
- device.new_command_queue().expect("Failed to create command queue");
- let context = match MTLContext::new(device, command_queue) {
- Ok(ctx) => ctx,
- Err(e) => {
- panic!("Failed to create MTLContext: {:?}", e);
- },
- };
+ let context = MTLContext::new().expect("Failed to create MTLContext");
let batch_size = 1;
let num_heads = 8;
@@ -1366,16 +1307,7 @@ fn run_two_pass_attention(
#[test]
#[ignore]
fn test_two_pass_attention() {
- let device =
- <dyn MTLDevice>::system_default().expect("No Metal device found");
- let command_queue =
- device.new_command_queue().expect("Failed to create command queue");
- let context = match MTLContext::new(device, command_queue) {
- Ok(ctx) => ctx,
- Err(e) => {
- panic!("Failed to create MTLContext: {:?}", e);
- },
- };
+ let context = MTLContext::new().expect("Failed to create MTLContext");
let kernel = match AttentionKernel::new(&context, KernelDataType::Float32) {
Ok(k) => k,
@@ -1439,16 +1371,7 @@ fn test_two_pass_attention() {
#[test]
#[ignore]
fn test_two_pass_attention_gqa() {
- let device =
- <dyn MTLDevice>::system_default().expect("No Metal device found");
- let command_queue =
- device.new_command_queue().expect("Failed to create command queue");
- let context = match MTLContext::new(device, command_queue) {
- Ok(ctx) => ctx,
- Err(e) => {
- panic!("Failed to create MTLContext: {:?}", e);
- },
- };
+ let context = MTLContext::new().expect("Failed to create MTLContext");
let kernel = match AttentionKernel::new(&context, KernelDataType::Float32) {
Ok(k) => k,
@@ -1514,16 +1437,7 @@ fn test_two_pass_attention_gqa() {
fn perf_two_pass_attention() {
use std::time::Instant;
- let device =
- <dyn MTLDevice>::system_default().expect("No Metal device found");
- let command_queue =
- device.new_command_queue().expect("Failed to create command queue");
- let context = match MTLContext::new(device, command_queue) {
- Ok(ctx) => ctx,
- Err(e) => {
- panic!("Failed to create MTLContext: {:?}", e);
- },
- };
+ let context = MTLContext::new().expect("Failed to create MTLContext");
let kernel = match AttentionKernel::new(&context, KernelDataType::Float32) {
Ok(k) => k,
diff --git a/crates/uzu/tests/grammar_json_schema_test.rs b/crates/uzu/tests/grammar_json_schema_test.rs
index 8621f613a..542a9207e 100644
--- a/crates/uzu/tests/grammar_json_schema_test.rs
+++ b/crates/uzu/tests/grammar_json_schema_test.rs
@@ -113,7 +113,6 @@ fn test_grammar_json_schema() {
}
#[test]
-#[ignore = "still broken (?)"]
fn test_grammar_json_schema_with_speculator() {
test_grammar(SpeculatorConfig {
number_of_speculated_tokens: 16,
diff --git a/crates/uzu/tests/kv_cache_state_test.rs b/crates/uzu/tests/kv_cache_state_test.rs
index 806a06972..23a5f8577 100644
--- a/crates/uzu/tests/kv_cache_state_test.rs
+++ b/crates/uzu/tests/kv_cache_state_test.rs
@@ -1,11 +1,15 @@
#![cfg(any(target_os = "macos", target_os = "ios"))]
-use metal::{MTLCommandBuffer, MTLCommandQueue, MTLDevice, MTLDeviceExt};
+use metal::{MTLCommandBuffer, MTLCommandQueue};
use uzu::{
- Array, DataType, DeviceContext,
- backends::metal::{
- KVCacheUpdate, KernelDataType, MTLContext,
- forward_pass::{INVALID_POSITION, KVCacheLayer, KVCacheLayerState},
+ DataType,
+ array::ArrayContextExt,
+ backends::{
+ common::Context,
+ metal::{
+ KVCacheUpdate, KernelDataType, MTLContext,
+ forward_pass::{INVALID_POSITION, KVCacheLayer, KVCacheLayerState},
+ },
},
};
@@ -28,21 +32,6 @@ struct Scenario {
expected_prefix_segment_length: usize,
}
-fn create_test_context() -> Option<MTLContext> {
- let device = <dyn MTLDevice>::system_default()?;
- let command_queue = device.new_command_queue()?;
- match MTLContext::new(device, command_queue) {
- Ok(ctx) => Some(ctx),
- Err(e) => {
- eprintln!(
- "Skipping KV cache tests: failed to create Metal context: {:?}",
- e
- );
- None
- },
- }
-}
-
fn make_test_layer(
context: &MTLContext,
state: KVCacheLayerState,
@@ -60,15 +49,15 @@ fn make_test_layer(
};
let shape = [1, total_len.max(1), 1];
- let keys = std::cell::RefCell::new(context.array(
+ let keys = std::cell::RefCell::new(context.create_array(
&shape,
DataType::F32,
- "kv_cache_keys".to_string(),
+ "kv_cache_keys",
));
- let values = std::cell::RefCell::new(context.array(
+ let values = std::cell::RefCell::new(context.create_array(
&shape,
DataType::F32,
- "kv_cache_values".to_string(),
+ "kv_cache_values",
));
let prefix_token_positions = match &state {
@@ -93,7 +82,7 @@ fn make_test_layer(
fn fill_arrays(layer: &mut KVCacheLayer) -> (Vec<f32>, Vec<f32>) {
let initial_keys = {
let mut keys_ref = layer.keys.borrow_mut();
- let slice = keys_ref.as_slice_mut::<f32>().unwrap();
+ let slice = keys_ref.as_slice_mut::<f32>();
for (idx, value) in slice.iter_mut().enumerate() {
*value = 1_000.0 + idx as f32;
}
@@ -102,7 +91,7 @@ fn fill_arrays(layer: &mut KVCacheLayer) -> (Vec<f32>, Vec<f32>) {
let initial_values = {
let mut values_ref = layer.values.borrow_mut();
- let slice = values_ref.as_slice_mut::<f32>().unwrap();
+ let slice = values_ref.as_slice_mut::<f32>();
for (idx, value) in slice.iter_mut().enumerate() {
*value = 2_000.0 + idx as f32;
}
@@ -251,7 +240,7 @@ fn run_scenario(
let actual_keys = {
let keys_ref = layer.keys.borrow();
- keys_ref.as_slice::<f32>().unwrap().to_vec()
+ keys_ref.as_slice::<f32>().to_vec()
};
assert_eq!(
actual_keys, expected_keys,
@@ -261,7 +250,7 @@ fn run_scenario(
let actual_values = {
let values_ref = layer.values.borrow();
- values_ref.as_slice::<f32>().unwrap().to_vec()
+ values_ref.as_slice::<f32>().to_vec()
};
assert_eq!(
actual_values, expected_values,
@@ -319,7 +308,7 @@ fn run_scenario(
#[test]
fn kv_cache_state_and_mask_scenarios() {
- let Some(context) = create_test_context() else {
+ let Some(context) = MTLContext::new().ok() else {
return;
};
@@ -446,7 +435,7 @@ fn kv_cache_state_and_mask_scenarios() {
#[test]
fn kv_cache_slice_apply_contiguous_window() {
- let Some(context) = create_test_context() else {
+ let Some(context) = MTLContext::new().ok() else {
return;
};
@@ -470,10 +459,10 @@ fn kv_cache_slice_apply_contiguous_window() {
layer.prefix_token_positions[1] = 998;
let mut keys = layer.keys.borrow_mut();
let mut values = layer.values.borrow_mut();
- keys.as_slice_mut::<f32>().unwrap()[0] = -1.0;
- keys.as_slice_mut::<f32>().unwrap()[1] = -2.0;
- values.as_slice_mut::<f32>().unwrap()[0] = -3.0;
- values.as_slice_mut::<f32>().unwrap()[1] = -4.0;
+ keys.as_slice_mut::<f32>()[0] = -1.0;
+ keys.as_slice_mut::<f32>()[1] = -2.0;
+ values.as_slice_mut::<f32>()[0] = -3.0;
+ values.as_slice_mut::<f32>()[1] = -4.0;
}
layer.apply_slice(&slice, None);
@@ -484,9 +473,8 @@ fn kv_cache_slice_apply_contiguous_window() {
"positions restored for contiguous slice"
);
- let keys_after = layer.keys.borrow().as_slice::<f32>().unwrap().to_vec();
- let values_after =
- layer.values.borrow().as_slice::<f32>().unwrap().to_vec();
+ let keys_after = layer.keys.borrow().as_slice::<f32>().to_vec();
+ let values_after = layer.values.borrow().as_slice::<f32>().to_vec();
assert_eq!(
keys_after[0..4],
initial_keys[0..4],
@@ -501,7 +489,7 @@ fn kv_cache_slice_apply_contiguous_window() {
#[test]
fn kv_cache_slice_apply_wrap_window() {
- let Some(context) = create_test_context() else {
+ let Some(context) = MTLContext::new().ok() else {
return;
};
@@ -525,10 +513,10 @@ fn kv_cache_slice_apply_wrap_window() {
layer.prefix_token_positions[3] = 778;
let mut keys = layer.keys.borrow_mut();
let mut values = layer.values.borrow_mut();
- keys.as_slice_mut::<f32>().unwrap()[2] = -11.0;
- keys.as_slice_mut::<f32>().unwrap()[3] = -12.0;
- values.as_slice_mut::<f32>().unwrap()[2] = -13.0;
- values.as_slice_mut::<f32>().unwrap()[3] = -14.0;
+ keys.as_slice_mut::<f32>()[2] = -11.0;
+ keys.as_slice_mut::<f32>()[3] = -12.0;
+ values.as_slice_mut::<f32>()[2] = -13.0;
+ values.as_slice_mut::<f32>()[3] = -14.0;
}
layer.apply_slice(&slice, None);
@@ -539,9 +527,8 @@ fn kv_cache_slice_apply_wrap_window() {
"positions restored for wrapped slice"
);
- let keys_after = layer.keys.borrow().as_slice::<f32>().unwrap().to_vec();
- let values_after =
- layer.values.borrow().as_slice::<f32>().unwrap().to_vec();
+ let keys_after = layer.keys.borrow().as_slice::<f32>().to_vec();
+ let values_after = layer.values.borrow().as_slice::<f32>().to_vec();
assert_eq!(
keys_after[0..4],
initial_keys[0..4],
@@ -556,7 +543,7 @@ fn kv_cache_slice_apply_wrap_window() {
#[test]
fn kv_cache_slice_apply_full_restores_metadata() {
- let Some(context) = create_test_context() else {
+ let Some(context) = MTLContext::new().ok() else {
return;
};
diff --git a/crates/uzu/tests/kv_cache_update_test.rs b/crates/uzu/tests/kv_cache_update_test.rs
index 6d8e8000e..922032495 100644
--- a/crates/uzu/tests/kv_cache_update_test.rs
+++ b/crates/uzu/tests/kv_cache_update_test.rs
@@ -1,13 +1,16 @@
#![cfg(any(target_os = "macos", target_os = "ios"))]
use bytemuck;
use metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandQueue, MTLDevice, MTLDeviceExt,
+ MTLBuffer, MTLCommandBuffer, MTLCommandQueue, MTLDeviceExt,
MTLResourceOptions,
};
use ndarray::{Array, Array3, s};
-use uzu::backends::metal::{
- KVCacheUpdate, KernelDataType, MTLContext,
- kernel::kv_cache_update::{KVLayerData, Swap, create_swaps_direct},
+use uzu::backends::{
+ common::Context,
+ metal::{
+ KVCacheUpdate, KernelDataType, MTLContext,
+ kernel::kv_cache_update::{KVLayerData, Swap, create_swaps_direct},
+ },
};
fn apply_swaps_3d<T: Clone>(
@@ -31,11 +34,7 @@ fn apply_swaps_3d<T: Clone>(
#[test]
fn test_kv_cache_update_kernel() {
- let device =
- <dyn MTLDevice>::system_default().expect("No Metal device found");
- let command_queue =
- device.new_command_queue().expect("Failed to create command queue");
- let metal_context = match MTLContext::new(device, command_queue) {
+ let metal_context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Failed to create MetalContext: {:?}. Skipping test.", e);
diff --git a/crates/uzu/tests/matmul_correctness_test.rs b/crates/uzu/tests/matmul_correctness_test.rs
index 6d67c10aa..25247df29 100644
--- a/crates/uzu/tests/matmul_correctness_test.rs
+++ b/crates/uzu/tests/matmul_correctness_test.rs
@@ -3,24 +3,21 @@
use bytemuck;
use half::bf16;
use metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue, MTLDevice,
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue,
MTLDeviceExt, MTLResourceOptions,
};
use ndarray::Array2;
use uzu::{
DataType,
- backends::metal::{
- MTLContext,
- kernel::{MatmulArguments, MatmulKernel},
+ backends::{
+ common::Context,
+ metal::{
+ MTLContext,
+ kernel::{MatmulArguments, MatmulKernel},
+ },
},
};
-fn create_test_context() -> Option<MTLContext> {
- let device = <dyn MTLDevice>::system_default()?;
- let command_queue = device.new_command_queue()?;
- MTLContext::new(device, command_queue).ok()
-}
-
fn run_metal_matmul(
ctx: &MTLContext,
a_data: &[bf16],
@@ -294,7 +291,7 @@ fn compare_results(
#[test]
#[ignore]
fn matmul_correctness_comprehensive() {
- let Some(ctx) = create_test_context() else {
+ let Some(ctx) = MTLContext::new().ok() else {
eprintln!("No Metal device available, skipping test");
return;
};
diff --git a/crates/uzu/tests/mlp_fused_correctness_test.rs b/crates/uzu/tests/mlp_fused_correctness_test.rs
index de423a5d8..88575a151 100644
--- a/crates/uzu/tests/mlp_fused_correctness_test.rs
+++ b/crates/uzu/tests/mlp_fused_correctness_test.rs
@@ -4,27 +4,24 @@
use bytemuck;
use half::f16;
use metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue, MTLDevice,
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue,
MTLDeviceExt, MTLResourceOptions,
};
use objc2::rc::autoreleasepool;
use uzu::{
DataType,
- backends::metal::{
- MTLContext,
- kernel::{
- mlp::MlpActivationType,
- mlp_fused::{MlpFusedArguments, MlpFusedKernel},
+ backends::{
+ common::Context,
+ metal::{
+ MTLContext,
+ kernel::{
+ mlp::MlpActivationType,
+ mlp_fused::{MlpFusedArguments, MlpFusedKernel},
+ },
},
},
};
-fn create_test_context() -> Option<MTLContext> {
- let device = <dyn MTLDevice>::system_default()?;
- let command_queue = device.new_command_queue()?;
- MTLContext::new(device, command_queue).ok()
-}
-
/// CPU reference implementation for SiLU activation
fn silu(x: f32) -> f32 {
x / (1.0 + (-x).exp())
@@ -242,7 +239,7 @@ fn compare_outputs(
#[test]
fn test_mlp_fused_gemv_silu_small() {
autoreleasepool(|_| {
- let ctx = match create_test_context() {
+ let ctx = match MTLContext::new().ok() {
Some(c) => c,
None => {
eprintln!("No Metal device available, skipping test");
@@ -306,7 +303,7 @@ fn test_mlp_fused_gemv_silu_small() {
#[test]
fn test_mlp_fused_gemv_gelu_small() {
autoreleasepool(|_| {
- let ctx = match create_test_context() {
+ let ctx = match MTLContext::new().ok() {
Some(c) => c,
None => {
eprintln!("No Metal device available, skipping test");
@@ -365,7 +362,7 @@ fn test_mlp_fused_gemv_gelu_small() {
#[test]
fn test_mlp_fused_gemm_silu_medium() {
autoreleasepool(|_| {
- let ctx = match create_test_context() {
+ let ctx = match MTLContext::new().ok() {
Some(c) => c,
None => {
eprintln!("No Metal device available, skipping test");
@@ -425,7 +422,7 @@ fn test_mlp_fused_gemm_silu_medium() {
#[test]
fn test_mlp_fused_gemv_realistic_dimensions() {
autoreleasepool(|_| {
- let ctx = match create_test_context() {
+ let ctx = match MTLContext::new().ok() {
Some(c) => c,
None => {
eprintln!("No Metal device available, skipping test");
diff --git a/crates/uzu/tests/mlp_fused_performance_test.rs b/crates/uzu/tests/mlp_fused_performance_test.rs
index 966b8d9d4..fc3542408 100644
--- a/crates/uzu/tests/mlp_fused_performance_test.rs
+++ b/crates/uzu/tests/mlp_fused_performance_test.rs
@@ -5,34 +5,31 @@ use std::time::{Duration, Instant};
use half::f16;
use metal::{
- MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue, MTLDevice,
- MTLDeviceExt, MTLResourceOptions,
+ MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue, MTLDeviceExt,
+ MTLResourceOptions,
};
use objc2::rc::autoreleasepool;
use uzu::{
DataType,
- backends::metal::{
- MTLContext,
- kernel::{
- matmul::{
- MatmulArguments, MatmulKernel,
- determine_kernel_variant as determine_matmul_variant,
- },
- mlp::MlpActivationType,
- mlp_fused::{
- MlpFusedArguments, MlpFusedKernel,
- determine_kernel_variant as determine_mlp_fused_variant,
+ backends::{
+ common::Context,
+ metal::{
+ MTLContext,
+ kernel::{
+ matmul::{
+ MatmulArguments, MatmulKernel,
+ determine_kernel_variant as determine_matmul_variant,
+ },
+ mlp::MlpActivationType,
+ mlp_fused::{
+ MlpFusedArguments, MlpFusedKernel,
+ determine_kernel_variant as determine_mlp_fused_variant,
+ },
},
},
},
};
-fn create_test_context() -> Option<MTLContext> {
- let device = <dyn MTLDevice>::system_default()?;
- let command_queue = device.new_command_queue()?;
- MTLContext::new(device, command_queue).ok()
-}
-
struct BenchmarkResult {
fused_time_microseconds: f64,
unfused_time_microseconds: f64,
@@ -77,9 +74,7 @@ fn benchmark_gemv_fused(
let weights_buffer = context
.device
.new_buffer(
- 2 * hidden_dimension
- * input_dimension
- * std::mem::size_of::<f16>(),
+ 2 * hidden_dimension * input_dimension * std::mem::size_of::<f16>(),
MTLResourceOptions::STORAGE_MODE_SHARED,
)
.expect("Failed to create buffer");
@@ -241,9 +236,7 @@ fn benchmark_gemm_fused(
let weights_buffer = context
.device
.new_buffer(
- 2 * hidden_dimension
- * input_dimension
- * std::mem::size_of::<f16>(),
+ 2 * hidden_dimension * input_dimension * std::mem::size_of::<f16>(),
MTLResourceOptions::STORAGE_MODE_SHARED,
)
.expect("Failed to create buffer");
@@ -419,7 +412,7 @@ const BATCH_SIZES: &[usize] = &[1, 8, 16, 32, 64, 128, 256, 512];
#[ignore] // Run with: cargo test --package uzu --test mlp_fused_performance_test -- --ignored --nocapture
fn mlp_fused_performance_benchmark() {
autoreleasepool(|_| {
- let context = match create_test_context() {
+ let context = match MTLContext::new().ok() {
Some(context) => context,
None => {
eprintln!("No Metal device available");
diff --git a/crates/uzu/tests/moe/counts_offsets_fused_test.rs b/crates/uzu/tests/moe/counts_offsets_fused_test.rs
index 7bde09c6b..4e8a96a6a 100644
--- a/crates/uzu/tests/moe/counts_offsets_fused_test.rs
+++ b/crates/uzu/tests/moe/counts_offsets_fused_test.rs
@@ -1,21 +1,19 @@
#![cfg(any(target_os = "macos", target_os = "ios"))]
+use super::test_utils::{alloc_buffer, alloc_buffer_with_data, create_ctx};
use half::bf16;
-use metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandQueue,
-};
-use rand::{Rng, SeedableRng, rngs::StdRng};
+use metal::{MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue};
+use rand::{RngExt, SeedableRng, rngs::StdRng};
+use uzu::backends::common::kernel::MoeCountsOffsetsFusedKernel;
use uzu::backends::metal::{
MTLContext, ProtocolObject, Retained,
kernel::{
- KernelDataType, MoeCountsOffsetsFusedArguments,
- MoeCountsOffsetsFusedKernel,
+ KernelDataType,
+ dsl::MoeCountsOffsetsFusedMetalKernel,
moe::{MoeRouterTopKArguments, MoeRouterTopKKernel},
},
};
-use super::test_utils::{alloc_buffer, alloc_buffer_with_data, create_ctx};
-
fn cpu_bucket_counts(
topk_ids: &[i32],
t: usize,
@@ -132,22 +130,26 @@ fn test_counts_offsets_fused_parity_random() {
let num_tiles = ((e + 511) / 512).max(1);
let partials_buf = alloc_buffer::<u32>(&ctx, num_tiles * 512);
- let kernel =
- MoeCountsOffsetsFusedKernel::new(&ctx).expect("fused kernel");
+ let kernel = MoeCountsOffsetsFusedMetalKernel::new(&ctx)
+ .expect("fused kernel");
let cb = ctx
.command_queue
.command_buffer()
.expect("Failed to create command buffer");
- let args = MoeCountsOffsetsFusedArguments {
- topk_ids_buffer: &topk_ids_buf,
- offsets_buffer: &offsets_buf,
- sum_k_buffer: &sum_k_buf,
- partials_buffer: &partials_buf,
- t,
- e,
- k,
- };
- kernel.encode(&cb, args).expect("encode fused");
+ let encoder = cb.new_compute_command_encoder().expect("encoder");
+
+ kernel.encode(
+ &topk_ids_buf,
+ &offsets_buf,
+ &sum_k_buf,
+ &partials_buf,
+ t as u32,
+ e as u32,
+ k as u32,
+ &encoder,
+ );
+
+ encoder.end_encoding();
cb.commit();
cb.wait_until_completed();
@@ -195,21 +197,24 @@ fn test_counts_offsets_fused_edge_cases() {
let num_tiles = ((e + 511) / 512).max(1);
let partials_buf = alloc_buffer::<u32>(&ctx, num_tiles * 512);
- let kernel = MoeCountsOffsetsFusedKernel::new(&ctx).expect("fused kernel");
+ let kernel =
+ MoeCountsOffsetsFusedMetalKernel::new(&ctx).expect("fused kernel");
let cb = ctx
.command_queue
.command_buffer()
.expect("Failed to create command buffer");
- let args = MoeCountsOffsetsFusedArguments {
- topk_ids_buffer: &topk_ids_buf,
- offsets_buffer: &offsets_buf,
- sum_k_buffer: &sum_k_buf,
- partials_buffer: &partials_buf,
- t,
- e,
- k,
- };
- kernel.encode(&cb, args).expect("encode fused");
+ let encoder = cb.new_compute_command_encoder().expect("encoder");
+ kernel.encode(
+ &topk_ids_buf,
+ &offsets_buf,
+ &sum_k_buf,
+ &partials_buf,
+ t as u32,
+ e as u32,
+ k as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
cb.commit();
cb.wait_until_completed();
@@ -233,21 +238,24 @@ fn test_counts_offsets_fused_edge_cases() {
let num_tiles = ((e + 511) / 512).max(1);
let partials_buf = alloc_buffer::<u32>(&ctx, num_tiles * 512);
- let kernel = MoeCountsOffsetsFusedKernel::new(&ctx).expect("fused kernel");
+ let kernel =
+ MoeCountsOffsetsFusedMetalKernel::new(&ctx).expect("fused kernel");
let cb = ctx
.command_queue
.command_buffer()
.expect("Failed to create command buffer");
- let args = MoeCountsOffsetsFusedArguments {
- topk_ids_buffer: &topk_ids_buf,
- offsets_buffer: &offsets_buf,
- sum_k_buffer: &sum_k_buf,
- partials_buffer: &partials_buf,
- t,
- e,
- k,
- };
- kernel.encode(&cb, args).expect("encode fused");
+ let encoder = cb.new_compute_command_encoder().expect("encoder");
+ kernel.encode(
+ &topk_ids_buf,
+ &offsets_buf,
+ &sum_k_buf,
+ &partials_buf,
+ t as u32,
+ e as u32,
+ k as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
cb.commit();
cb.wait_until_completed();
diff --git a/crates/uzu/tests/moe/moe_block_e2e_test.rs b/crates/uzu/tests/moe/moe_block_e2e_test.rs
index bd48a1c67..46937eb8c 100644
--- a/crates/uzu/tests/moe/moe_block_e2e_test.rs
+++ b/crates/uzu/tests/moe/moe_block_e2e_test.rs
@@ -1,16 +1,19 @@
use half::bf16;
-use metal::{MTLBuffer, MTLCommandBuffer, MTLCommandQueue};
-use rand::{Rng, SeedableRng, rngs::StdRng};
-use uzu::backends::metal::{
- MTLContext,
- kernel::{
- KernelDataType, MoeBlockBasesArguments, MoeCountsOffsetsFusedArguments,
- MoeCountsOffsetsFusedKernel, MoeFinalizeArguments, MoeFinalizeKernel,
- MoeScatterKernels, MoeScatterWithMapArguments,
- moe::{
- MoeExpertsTwoPassArguments, MoeExpertsTwoPassPrefillKernel,
- MoeGatherArguments, MoeGatherKernel, MoeRouterTopKArguments,
- MoeRouterTopKKernel,
+use metal::{MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue};
+use rand::{RngExt, SeedableRng, rngs::StdRng};
+use uzu::backends::{
+ common::kernel::{MoeCountsOffsetsFusedKernel, MoeFinalizeKernel},
+ metal::{
+ MTLContext,
+ kernel::{
+ KernelDataType, MoeBlockBasesArguments, MoeScatterKernels,
+ MoeScatterWithMapArguments,
+ dsl::{MoeCountsOffsetsFusedMetalKernel, MoeFinalizeMetalKernel},
+ moe::{
+ MoeExpertsTwoPassArguments, MoeExpertsTwoPassPrefillKernel,
+ MoeGatherArguments, MoeGatherKernels, MoeRouterTopKArguments,
+ MoeRouterTopKKernel,
+ },
},
},
};
@@ -405,21 +408,19 @@ fn run_moe_parity_test_internal(
.expect("router+topk encode");
let fused_kernel =
- MoeCountsOffsetsFusedKernel::new(&ctx).expect("fused kernel");
- fused_kernel
- .encode(
- &cb,
- MoeCountsOffsetsFusedArguments {
- topk_ids_buffer: &topk_ids_buf,
- offsets_buffer: &offsets_buf,
- sum_k_buffer: &sumk_buf,
- partials_buffer: &partials_buf,
- t,
- e,
- k,
- },
- )
- .expect("fused encode");
+ MoeCountsOffsetsFusedMetalKernel::new(&ctx).expect("fused kernel");
+ let encoder = cb.new_compute_command_encoder().expect("encoder");
+ fused_kernel.encode(
+ &topk_ids_buf,
+ &offsets_buf,
+ &sumk_buf,
+ &partials_buf,
+ t as u32,
+ e as u32,
+ k as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
let scatter = MoeScatterKernels::new(&ctx).expect("scatter");
scatter
@@ -460,22 +461,20 @@ fn run_moe_parity_test_internal(
)
.expect("scatter");
- let gather = MoeGatherKernel::new(&ctx).expect("gather");
- gather
- .encode(
- &cb,
- KernelDataType::BFloat16,
- MoeGatherArguments {
- x_buffer: &x_buf,
- bucketed_ids_buffer: &bucketed_ids_buf,
- x_perm_buffer: &x_perm_buf,
- sumk_buffer: &sumk_buf,
- t,
- k,
- d_model,
- },
- )
- .expect("gather");
+ let gather = MoeGatherKernels::new(&ctx).expect("gather");
+ gather.encode(
+ &cb,
+ KernelDataType::BFloat16,
+ &MoeGatherArguments {
+ x_buffer: &x_buf,
+ bucketed_ids_buffer: &bucketed_ids_buf,
+ x_perm_buffer: &x_perm_buf,
+ sumk_buffer: &sumk_buf,
+ t,
+ k,
+ d_model,
+ },
+ );
// Additional buffers for 2-pass
let total_rows = t * k;
@@ -518,22 +517,21 @@ fn run_moe_parity_test_internal(
)
.expect("experts encode");
- let finalize = MoeFinalizeKernel::new(&ctx).expect("finalize");
- finalize
- .encode(
- &cb,
- MoeFinalizeArguments {
- tok2row_buffer: &tok2row_buf,
- probs_buffer: &topk_probs_buf,
- y_partial_buffer: &y_partial_buf,
- y_out_buffer: &y_out_buf,
- t,
- d_model,
- k,
- },
- KernelDataType::BFloat16,
- )
- .expect("finalize");
+ let finalize =
+ MoeFinalizeMetalKernel::new(&ctx, KernelDataType::BFloat16.into())
+ .expect("finalize");
+ let encoder = cb.new_compute_command_encoder().expect("encoder");
+ finalize.encode(
+ &tok2row_buf,
+ &topk_probs_buf,
+ &y_partial_buf,
+ &y_out_buf,
+ t as u32,
+ d_model as u32,
+ k as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
eprintln!("[E2E] All kernels encoded. Committing ONCE and waiting...");
cb.commit();
diff --git a/crates/uzu/tests/moe/moe_experts_perf_test.rs b/crates/uzu/tests/moe/moe_experts_perf_test.rs
index a2eaba42c..f154d8195 100644
--- a/crates/uzu/tests/moe/moe_experts_perf_test.rs
+++ b/crates/uzu/tests/moe/moe_experts_perf_test.rs
@@ -1,13 +1,13 @@
-use metal::{MTLCommandBuffer, MTLCommandQueue};
use std::time::Instant;
use half::bf16;
-use rand::{Rng, SeedableRng, rngs::StdRng};
+use metal::{MTLCommandBuffer, MTLCommandQueue};
+use rand::{RngExt, SeedableRng, rngs::StdRng};
use uzu::backends::metal::{
KernelDataType, MTLContext,
kernel::moe::{
- MoeExpertsSingleDecodeArguments, MoeExpertsSingleDecodeKernel,
- MoeExpertsTwoPassArguments, MoeExpertsTwoPassDecodeKernel,
+ MoeExpertsSingleDecodeArguments, MoeExpertsSingleDecodeKernels,
+ MoeExpertsTwoPassArguments, MoeExpertsTwoPassDecodeKernels,
MoeExpertsTwoPassPrefillKernel,
},
};
@@ -86,8 +86,8 @@ fn run_decode_case(
.map(|_| bf16::from_f32(rng.random_range(-0.01..0.01)))
.collect();
- let experts_kernel =
- MoeExpertsTwoPassDecodeKernel::new(ctx).expect("experts decode kernel");
+ let experts_kernel = MoeExpertsTwoPassDecodeKernels::new(ctx)
+ .expect("experts decode kernel");
let x_perm_buf = alloc_buffer_with_data(&ctx, &x_perm);
let offsets_buf = alloc_buffer_with_data(&ctx, &offsets);
@@ -143,10 +143,11 @@ fn run_decode_case(
};
for _ in 0..warmup {
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- experts_kernel
- .encode(&cb, make_two_pass_args())
- .expect("two-pass encode");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ experts_kernel.encode(&cb, &make_two_pass_args());
cb.commit();
cb.wait_until_completed();
}
@@ -154,10 +155,11 @@ fn run_decode_case(
let mut times = Vec::with_capacity(iters);
for _ in 0..iters {
let start = Instant::now();
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- experts_kernel
- .encode(&cb, make_two_pass_args())
- .expect("two-pass encode");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ experts_kernel.encode(&cb, &make_two_pass_args());
cb.commit();
cb.wait_until_completed();
times.push(start.elapsed().as_secs_f64() * 1000.0);
@@ -301,7 +303,10 @@ fn run_two_pass_prefill_case(
};
for _ in 0..warmup {
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
experts_kernel
.encode(&cb, make_two_pass_args())
.expect("two-pass prefill encode");
@@ -312,7 +317,10 @@ fn run_two_pass_prefill_case(
let mut times = Vec::with_capacity(iters);
for _ in 0..iters {
let start = Instant::now();
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
experts_kernel
.encode(&cb, make_two_pass_args())
.expect("two-pass prefill encode");
@@ -402,7 +410,7 @@ fn run_fused_single_token_case(
.collect();
let fused_kernel =
- MoeExpertsSingleDecodeKernel::new(ctx).expect("fused kernel");
+ MoeExpertsSingleDecodeKernels::new(ctx).expect("fused kernel");
let x_buf = alloc_buffer_with_data(ctx, &x);
let topk_ids_buf = alloc_buffer_with_data(ctx, &topk_ids);
@@ -437,8 +445,11 @@ fn run_fused_single_token_case(
};
for _ in 0..warmup {
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- fused_kernel.encode(&cb, make_args()).expect("encode");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ fused_kernel.encode(&cb, make_args());
cb.commit();
cb.wait_until_completed();
}
@@ -446,8 +457,11 @@ fn run_fused_single_token_case(
let mut times = Vec::with_capacity(iters);
for _ in 0..iters {
let start = Instant::now();
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- fused_kernel.encode(&cb, make_args()).expect("encode");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ fused_kernel.encode(&cb, make_args());
cb.commit();
cb.wait_until_completed();
times.push(start.elapsed().as_secs_f64() * 1000.0);
@@ -571,7 +585,7 @@ fn run_indirect_decode_timed(
.collect();
let experts_kernel =
- MoeExpertsTwoPassDecodeKernel::new(ctx).expect("decode kernel");
+ MoeExpertsTwoPassDecodeKernels::new(ctx).expect("decode kernel");
let x_perm_buf = alloc_buffer_with_data(ctx, &x_perm);
let offsets_buf = alloc_buffer_with_data(ctx, &offsets);
@@ -622,8 +636,11 @@ fn run_indirect_decode_timed(
};
for _ in 0..warmup {
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- experts_kernel.encode(&cb, make_args()).expect("encode");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ experts_kernel.encode(&cb, &make_args());
cb.commit();
cb.wait_until_completed();
}
@@ -631,8 +648,11 @@ fn run_indirect_decode_timed(
let mut times = Vec::with_capacity(iters);
for _ in 0..iters {
let start = Instant::now();
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- experts_kernel.encode(&cb, make_args()).expect("encode");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ experts_kernel.encode(&cb, &make_args());
cb.commit();
cb.wait_until_completed();
times.push(start.elapsed().as_secs_f64() * 1000.0);
@@ -679,7 +699,7 @@ fn run_fused_decode_timed(
.collect();
let fused_kernel =
- MoeExpertsSingleDecodeKernel::new(ctx).expect("fused kernel");
+ MoeExpertsSingleDecodeKernels::new(ctx).expect("fused kernel");
let x_buf = alloc_buffer_with_data(ctx, &x);
let topk_ids_buf = alloc_buffer_with_data(ctx, &topk_ids);
@@ -714,8 +734,11 @@ fn run_fused_decode_timed(
};
for _ in 0..warmup {
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- fused_kernel.encode(&cb, make_args()).expect("encode");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ fused_kernel.encode(&cb, make_args());
cb.commit();
cb.wait_until_completed();
}
@@ -723,8 +746,11 @@ fn run_fused_decode_timed(
let mut times = Vec::with_capacity(iters);
for _ in 0..iters {
let start = Instant::now();
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- fused_kernel.encode(&cb, make_args()).expect("encode");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ fused_kernel.encode(&cb, make_args());
cb.commit();
cb.wait_until_completed();
times.push(start.elapsed().as_secs_f64() * 1000.0);
diff --git a/crates/uzu/tests/moe/moe_experts_test.rs b/crates/uzu/tests/moe/moe_experts_test.rs
index 1045483f3..3461f7c23 100644
--- a/crates/uzu/tests/moe/moe_experts_test.rs
+++ b/crates/uzu/tests/moe/moe_experts_test.rs
@@ -8,16 +8,14 @@
#![cfg(any(target_os = "macos", target_os = "ios"))]
-use metal::{MTLBuffer, MTLCommandBuffer, MTLCommandQueue};
-
use half::bf16;
-use rand::{Rng, SeedableRng, rngs::StdRng};
+use metal::{MTLBuffer, MTLCommandBuffer, MTLCommandQueue};
+use rand::{RngExt, SeedableRng, rngs::StdRng};
use uzu::backends::metal::{
KernelDataType,
kernel::moe::{
- MoeExpertsSingleDecodeArguments, MoeExpertsSingleDecodeKernel,
- MoeExpertsTwoPassArguments, MoeExpertsTwoPassDecodeKernel,
- MoeExpertsTwoPassPrefillKernel,
+ MoeExpertsSingleDecodeKernels, MoeExpertsTwoPassArguments,
+ MoeExpertsTwoPassDecodeKernels, MoeExpertsTwoPassPrefillKernel,
},
};
@@ -27,6 +25,7 @@ use test_utils::{
alloc_buffer, alloc_buffer_with_data, assert_bf16_close, cpu_tile_counts,
cpu_tile_scan, create_ctx,
};
+use uzu::backends::metal::kernel::moe::MoeExpertsSingleDecodeArguments;
/// Test data for MoE experts
struct MoeTestData {
@@ -446,46 +445,47 @@ fn test_two_pass_decode_correctness() {
let dispatch_args_buf = alloc_buffer::<u32>(&ctx, 3);
// Execute 2-pass decode kernel
- let experts_kernel = MoeExpertsTwoPassDecodeKernel::new(&ctx)
+ let experts_kernel = MoeExpertsTwoPassDecodeKernels::new(&ctx)
.expect("MoeExpertsTwoPassDecodeKernel::new");
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
const K_TILE: usize = 64;
let num_tiles_k = ((d_ff + K_TILE - 1) / K_TILE) as u32;
- experts_kernel
- .encode(
- &cb,
- MoeExpertsTwoPassArguments {
- x_perm_buffer: &x_perm_buf,
- expert_offsets: &offsets_buf,
- row_expert_map: &row_expert_map_buf,
- hidden_buffer: &hidden_buf,
- output_buffer: &y_partial_buf,
- w13_all: &w13_buf,
- w2_all: &w2_buf,
- up_biases: &up_biases_buf,
- down_biases: &down_biases_buf,
- tile_counts: &tile_counts_buf,
- tile_offsets: &tile_offsets_buf,
- tile_map: &tile_map_buf,
- total_tiles: &total_tiles_buf,
- dispatch_args: &dispatch_args_buf,
- total_rows: sum_k,
- d_model,
- d_ff,
- e,
- num_tiles_k,
- gating_code,
- gate_clip_min: f32::NEG_INFINITY,
- gate_clip_max: f32::INFINITY,
- up_clip_min: f32::NEG_INFINITY,
- up_clip_max: f32::INFINITY,
- silu_alpha,
- data_type: KernelDataType::BFloat16,
- },
- )
- .expect("encode_two_pass_decode");
+ experts_kernel.encode(
+ &cb,
+ &MoeExpertsTwoPassArguments {
+ x_perm_buffer: &x_perm_buf,
+ expert_offsets: &offsets_buf,
+ row_expert_map: &row_expert_map_buf,
+ hidden_buffer: &hidden_buf,
+ output_buffer: &y_partial_buf,
+ w13_all: &w13_buf,
+ w2_all: &w2_buf,
+ up_biases: &up_biases_buf,
+ down_biases: &down_biases_buf,
+ tile_counts: &tile_counts_buf,
+ tile_offsets: &tile_offsets_buf,
+ tile_map: &tile_map_buf,
+ total_tiles: &total_tiles_buf,
+ dispatch_args: &dispatch_args_buf,
+ total_rows: sum_k,
+ d_model,
+ d_ff,
+ e,
+ num_tiles_k,
+ gating_code,
+ gate_clip_min: f32::NEG_INFINITY,
+ gate_clip_max: f32::INFINITY,
+ up_clip_min: f32::NEG_INFINITY,
+ up_clip_max: f32::INFINITY,
+ silu_alpha,
+ data_type: KernelDataType::BFloat16,
+ },
+ );
cb.commit();
cb.wait_until_completed();
@@ -604,41 +604,42 @@ fn test_two_pass_decode_multi_token() {
let dispatch_args_buf = alloc_buffer::<u32>(&ctx, 3);
let experts_kernel =
- MoeExpertsTwoPassDecodeKernel::new(&ctx).expect("kernel");
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- experts_kernel
- .encode(
- &cb,
- MoeExpertsTwoPassArguments {
- x_perm_buffer: &x_perm_buf,
- expert_offsets: &offsets_buf,
- row_expert_map: &row_expert_map_buf,
- hidden_buffer: &hidden_buf,
- output_buffer: &y_partial_buf,
- w13_all: &w13_buf,
- w2_all: &w2_buf,
- up_biases: &up_biases_buf,
- down_biases: &down_biases_buf,
- tile_counts: &tile_counts_buf,
- tile_offsets: &tile_offsets_buf,
- tile_map: &tile_map_buf,
- total_tiles: &total_tiles_buf,
- dispatch_args: &dispatch_args_buf,
- total_rows: sum_k,
- d_model,
- d_ff,
- e,
- num_tiles_k: ((d_ff + 63) / 64) as u32,
- gating_code,
- gate_clip_min: f32::NEG_INFINITY,
- gate_clip_max: f32::INFINITY,
- up_clip_min: f32::NEG_INFINITY,
- up_clip_max: f32::INFINITY,
- silu_alpha,
- data_type: KernelDataType::BFloat16,
- },
- )
- .expect("encode");
+ MoeExpertsTwoPassDecodeKernels::new(&ctx).expect("kernel");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ experts_kernel.encode(
+ &cb,
+ &MoeExpertsTwoPassArguments {
+ x_perm_buffer: &x_perm_buf,
+ expert_offsets: &offsets_buf,
+ row_expert_map: &row_expert_map_buf,
+ hidden_buffer: &hidden_buf,
+ output_buffer: &y_partial_buf,
+ w13_all: &w13_buf,
+ w2_all: &w2_buf,
+ up_biases: &up_biases_buf,
+ down_biases: &down_biases_buf,
+ tile_counts: &tile_counts_buf,
+ tile_offsets: &tile_offsets_buf,
+ tile_map: &tile_map_buf,
+ total_tiles: &total_tiles_buf,
+ dispatch_args: &dispatch_args_buf,
+ total_rows: sum_k,
+ d_model,
+ d_ff,
+ e,
+ num_tiles_k: ((d_ff + 63) / 64) as u32,
+ gating_code,
+ gate_clip_min: f32::NEG_INFINITY,
+ gate_clip_max: f32::INFINITY,
+ up_clip_min: f32::NEG_INFINITY,
+ up_clip_max: f32::INFINITY,
+ silu_alpha,
+ data_type: KernelDataType::BFloat16,
+ },
+ );
cb.commit();
cb.wait_until_completed();
@@ -725,7 +726,10 @@ fn test_two_pass_prefill_correctness() {
let experts_kernel =
MoeExpertsTwoPassPrefillKernel::new(&ctx).expect("kernel");
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
experts_kernel
.encode(
&cb,
@@ -881,43 +885,47 @@ fn test_fused_single_token_decode() {
let y_buf = alloc_buffer::<bf16>(&ctx, d_model);
// Run fused decode kernel
- let fused_kernel = MoeExpertsSingleDecodeKernel::new(&ctx)
+ let fused_kernel = MoeExpertsSingleDecodeKernels::new(&ctx)
.expect("MoeExpertsSingleDecodeKernel::new");
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
-
- fused_kernel
- .encode(
- &cb,
- MoeExpertsSingleDecodeArguments {
- x: &x_buf,
- topk_ids: &topk_ids_buf,
- topk_probs: &topk_probs_buf,
- w13_all: &w13_buf,
- w2_all: &w2_buf,
- up_biases: &up_biases_buf,
- down_biases: &down_biases_buf,
- hidden: &hidden_buf,
- y: &y_buf,
- d_model,
- d_ff,
- k,
- gating_code,
- silu_alpha: 1.0, // Standard SiLU for testing
- gate_clip_min: f32::NEG_INFINITY,
- gate_clip_max: f32::INFINITY,
- up_clip_min: f32::NEG_INFINITY,
- up_clip_max: f32::INFINITY,
- data_type: KernelDataType::BFloat16,
- },
- )
- .expect("fused decode encode");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+
+ fused_kernel.encode(
+ &cb,
+ MoeExpertsSingleDecodeArguments {
+ x: &x_buf,
+ topk_ids: &topk_ids_buf,
+ topk_probs: &topk_probs_buf,
+ w13_all: &w13_buf,
+ w2_all: &w2_buf,
+ up_biases: &up_biases_buf,
+ down_biases: &down_biases_buf,
+ hidden: &hidden_buf,
+ y: &y_buf,
+ d_model,
+ d_ff,
+ k,
+ gating_code,
+ silu_alpha: 1.0, // Standard SiLU for testing
+ gate_clip_min: f32::NEG_INFINITY,
+ gate_clip_max: f32::INFINITY,
+ up_clip_min: f32::NEG_INFINITY,
+ up_clip_max: f32::INFINITY,
+ data_type: KernelDataType::BFloat16,
+ },
+ );
cb.commit();
cb.wait_until_completed();
// Read GPU output
let y_gpu = unsafe {
- std::slice::from_raw_parts(y_buf.contents().as_ptr() as *const bf16, d_model)
+ std::slice::from_raw_parts(
+ y_buf.contents().as_ptr() as *const bf16,
+ d_model,
+ )
};
// Compute error metrics
@@ -1031,39 +1039,43 @@ fn test_fused_single_token_k4() {
let y_buf = alloc_buffer::<bf16>(&ctx, d_model);
let fused_kernel =
- MoeExpertsSingleDecodeKernel::new(&ctx).expect("fused kernel");
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- fused_kernel
- .encode(
- &cb,
- MoeExpertsSingleDecodeArguments {
- x: &x_buf,
- topk_ids: &topk_ids_buf,
- topk_probs: &topk_probs_buf,
- w13_all: &w13_buf,
- w2_all: &w2_buf,
- up_biases: &up_biases_buf,
- down_biases: &down_biases_buf,
- hidden: &hidden_buf,
- y: &y_buf,
- d_model,
- d_ff,
- k,
- gating_code,
- silu_alpha,
- gate_clip_min: f32::NEG_INFINITY,
- gate_clip_max: f32::INFINITY,
- up_clip_min: f32::NEG_INFINITY,
- up_clip_max: f32::INFINITY,
- data_type: KernelDataType::BFloat16,
- },
- )
- .expect("fused encode");
+ MoeExpertsSingleDecodeKernels::new(&ctx).expect("fused kernel");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ fused_kernel.encode(
+ &cb,
+ MoeExpertsSingleDecodeArguments {
+ x: &x_buf,
+ topk_ids: &topk_ids_buf,
+ topk_probs: &topk_probs_buf,
+ w13_all: &w13_buf,
+ w2_all: &w2_buf,
+ up_biases: &up_biases_buf,
+ down_biases: &down_biases_buf,
+ hidden: &hidden_buf,
+ y: &y_buf,
+ d_model,
+ d_ff,
+ k,
+ gating_code,
+ silu_alpha,
+ gate_clip_min: f32::NEG_INFINITY,
+ gate_clip_max: f32::INFINITY,
+ up_clip_min: f32::NEG_INFINITY,
+ up_clip_max: f32::INFINITY,
+ data_type: KernelDataType::BFloat16,
+ },
+ );
cb.commit();
cb.wait_until_completed();
let y_gpu = unsafe {
- std::slice::from_raw_parts(y_buf.contents().as_ptr() as *const bf16, d_model)
+ std::slice::from_raw_parts(
+ y_buf.contents().as_ptr() as *const bf16,
+ d_model,
+ )
};
// Compute error metrics
diff --git a/crates/uzu/tests/moe/moe_finalize_test.rs b/crates/uzu/tests/moe/moe_finalize_test.rs
index b4f7ae6da..e6656dd96 100644
--- a/crates/uzu/tests/moe/moe_finalize_test.rs
+++ b/crates/uzu/tests/moe/moe_finalize_test.rs
@@ -1,11 +1,12 @@
#![cfg(any(target_os = "macos", target_os = "ios"))]
-use metal::{MTLBuffer, MTLCommandBuffer, MTLCommandQueue};
+use metal::{MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue};
use half::bf16;
-use rand::{Rng, SeedableRng, rngs::StdRng};
-use uzu::backends::metal::kernel::{
- KernelDataType, MoeFinalizeArguments, MoeFinalizeKernel,
+use rand::{RngExt, SeedableRng, rngs::StdRng};
+use uzu::backends::common::kernel::MoeFinalizeKernel;
+use uzu::backends::metal::{
+ KernelDataType, kernel::dsl::MoeFinalizeMetalKernel,
};
use super::test_utils::{
@@ -95,23 +96,25 @@ fn test_finalize_correctness() {
let y_out_buf = alloc_buffer::<bf16>(&ctx, t * d_model);
// Execute finalize kernel
- let finalize = MoeFinalizeKernel::new(&ctx).expect("finalize kernel");
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- finalize
- .encode(
- &cb,
- MoeFinalizeArguments {
- tok2row_buffer: &tok2row_buf,
- probs_buffer: &probs_buf,
- y_partial_buffer: &y_partial_buf,
- y_out_buffer: &y_out_buf,
- t,
- d_model,
- k,
- },
- KernelDataType::BFloat16,
- )
- .expect("encode finalize");
+ let finalize =
+ MoeFinalizeMetalKernel::new(&ctx, KernelDataType::BFloat16.into())
+ .expect("finalize kernel");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ let encoder = cb.new_compute_command_encoder().expect("encoder");
+ finalize.encode(
+ &tok2row_buf,
+ &probs_buf,
+ &y_partial_buf,
+ &y_out_buf,
+ t as u32,
+ d_model as u32,
+ k as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
cb.commit();
cb.wait_until_completed();
diff --git a/crates/uzu/tests/moe/moe_gather_test.rs b/crates/uzu/tests/moe/moe_gather_test.rs
index e905c06f4..19fad640f 100644
--- a/crates/uzu/tests/moe/moe_gather_test.rs
+++ b/crates/uzu/tests/moe/moe_gather_test.rs
@@ -3,10 +3,10 @@
use metal::{MTLBuffer, MTLCommandBuffer, MTLCommandQueue};
use half::bf16;
-use rand::{Rng, SeedableRng, rngs::StdRng};
+use rand::{RngExt, SeedableRng, rngs::StdRng};
use uzu::backends::metal::kernel::{
KernelDataType,
- moe::{MoeGatherArguments, MoeGatherKernel},
+ moe::{MoeGatherArguments, MoeGatherKernels},
};
use super::test_utils::{
@@ -81,23 +81,24 @@ fn test_gather_correctness() {
let sumk_buf = alloc_buffer_with_data(&ctx, &sum_k_u32);
// Execute gather kernel using kernel struct
- let gather = MoeGatherKernel::new(&ctx).expect("MoeGatherKernel::new");
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- gather
- .encode(
- &cb,
- KernelDataType::BFloat16,
- MoeGatherArguments {
- x_buffer: &x_buf,
- bucketed_ids_buffer: &ids_buf,
- x_perm_buffer: &x_perm_buf,
- sumk_buffer: &sumk_buf,
- t: t,
- k: sum_k / t, // Decompose sum_k into k per token
- d_model,
- },
- )
- .expect("encode gather");
+ let gather = MoeGatherKernels::new(&ctx).expect("MoeGatherKernel::new");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ gather.encode(
+ &cb,
+ KernelDataType::BFloat16,
+ &MoeGatherArguments {
+ x_buffer: &x_buf,
+ bucketed_ids_buffer: &ids_buf,
+ x_perm_buffer: &x_perm_buf,
+ sumk_buffer: &sumk_buf,
+ t: t,
+ k: sum_k / t, // Decompose sum_k into k per token
+ d_model,
+ },
+ );
cb.commit();
cb.wait_until_completed();
diff --git a/crates/uzu/tests/moe/moe_perf_test.rs b/crates/uzu/tests/moe/moe_perf_test.rs
index 854b4975a..6af1e2978 100644
--- a/crates/uzu/tests/moe/moe_perf_test.rs
+++ b/crates/uzu/tests/moe/moe_perf_test.rs
@@ -1,16 +1,21 @@
-use metal::{MTLCommandBuffer, MTLCommandQueue};
use std::time::Instant;
use half::bf16;
-use rand::{Rng, SeedableRng, rngs::StdRng};
-use uzu::backends::metal::{
- KernelDataType,
- kernel::moe::{
- MoeCountsOffsetsFusedArguments, MoeCountsOffsetsFusedKernel,
- MoeExpertsTwoPassArguments, MoeExpertsTwoPassDecodeKernel,
- MoeFinalizeArguments, MoeFinalizeKernel, MoeGatherArguments,
- MoeGatherKernel, MoeRouterTopKArguments, MoeRouterTopKKernel,
- MoeScatterKernels, MoeScatterWithMapArguments,
+use metal::{MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue};
+use rand::{RngExt, SeedableRng, rngs::StdRng};
+use uzu::backends::{
+ common::kernel::{MoeCountsOffsetsFusedKernel, MoeFinalizeKernel},
+ metal::{
+ KernelDataType,
+ kernel::{
+ dsl::{MoeCountsOffsetsFusedMetalKernel, MoeFinalizeMetalKernel},
+ moe::{
+ MoeExpertsTwoPassArguments, MoeExpertsTwoPassDecodeKernels,
+ MoeGatherArguments, MoeGatherKernels, MoeRouterTopKArguments,
+ MoeRouterTopKKernel, MoeScatterKernels,
+ MoeScatterWithMapArguments,
+ },
+ },
},
};
@@ -135,7 +140,10 @@ fn test_moe_e2e_decode_perf() {
// Time fused Router+TopK
let fused_perf = time_kernel("Router+TopK (FUSED)", 5, 20, || {
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
router_topk
.encode(
&cb,
@@ -208,7 +216,10 @@ fn test_moe_e2e_prefill_perf() {
// Time fused Router+TopK
let fused_perf = time_kernel("Router+TopK (FUSED)", 5, 20, || {
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
router_topk
.encode(
&cb,
@@ -337,20 +348,25 @@ fn test_moe_pipeline_breakdown_decode() {
let bucketed_probs_buf = alloc_buffer::<bf16>(&ctx, t * k);
// Create kernel structs (use production-validated encoding logic)
- let counts_offsets_kernel =
- MoeCountsOffsetsFusedKernel::new(&ctx).expect("counts+offsets fused");
+ let counts_offsets_kernel = MoeCountsOffsetsFusedMetalKernel::new(&ctx)
+ .expect("counts+offsets fused");
let scatter_kernel = MoeScatterKernels::new(&ctx).expect("scatter");
- let gather_kernel = MoeGatherKernel::new(&ctx).expect("gather");
- let experts_kernel = MoeExpertsTwoPassDecodeKernel::new(&ctx)
+ let gather_kernel = MoeGatherKernels::new(&ctx).expect("gather");
+ let experts_kernel = MoeExpertsTwoPassDecodeKernels::new(&ctx)
.expect("experts two-pass decode");
- let finalize_kernel = MoeFinalizeKernel::new(&ctx).expect("finalize");
+ let finalize_kernel =
+ MoeFinalizeMetalKernel::new(&ctx, KernelDataType::BFloat16.into())
+ .expect("finalize");
let router_topk_fused_kernel =
MoeRouterTopKKernel::new(&ctx).expect("router+topk fused");
// Testing: Router + TopK + Counts+Offsets (FUSED)
let router_topk_fused_perf =
time_kernel("Router+TopK (FUSED)", 2, 5, || {
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
router_topk_fused_kernel
.encode(
&cb,
@@ -375,27 +391,31 @@ fn test_moe_pipeline_breakdown_decode() {
let counts_offsets_perf =
time_kernel("Counts+Offsets (FUSED)", 2, 5, || {
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- counts_offsets_kernel
- .encode(
- &cb,
- MoeCountsOffsetsFusedArguments {
- topk_ids_buffer: &topk_ids_buf,
- offsets_buffer: &offsets_buf,
- sum_k_buffer: &sumk_buf,
- partials_buffer: &partials_buf,
- t,
- e,
- k,
- },
- )
- .expect("counts+offsets fused");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ let encoder = cb.new_compute_command_encoder().expect("encoder");
+ counts_offsets_kernel.encode(
+ &topk_ids_buf,
+ &offsets_buf,
+ &sumk_buf,
+ &partials_buf,
+ t as u32,
+ e as u32,
+ k as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
cb.commit();
cb.wait_until_completed();
});
let scatter_perf = time_kernel("Scatter", 2, 5, || {
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
scatter_kernel
.encode_block_bases(
&cb,
@@ -438,82 +458,84 @@ fn test_moe_pipeline_breakdown_decode() {
});
let gather_perf = time_kernel("Gather", 2, 5, || {
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- gather_kernel
- .encode(
- &cb,
- KernelDataType::BFloat16,
- MoeGatherArguments {
- x_buffer: &x_buf,
- bucketed_ids_buffer: &bucketed_ids_buf,
- x_perm_buffer: &x_perm_buf,
- sumk_buffer: &sumk_buf,
- t,
- k,
- d_model,
- },
- )
- .expect("gather");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ gather_kernel.encode(
+ &cb,
+ KernelDataType::BFloat16,
+ &MoeGatherArguments {
+ x_buffer: &x_buf,
+ bucketed_ids_buffer: &bucketed_ids_buf,
+ x_perm_buffer: &x_perm_buf,
+ sumk_buffer: &sumk_buf,
+ t,
+ k,
+ d_model,
+ },
+ );
cb.commit();
cb.wait_until_completed();
});
let experts_perf = time_kernel("Experts (MAIN COMPUTE)", 2, 5, || {
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- experts_kernel
- .encode(
- &cb,
- MoeExpertsTwoPassArguments {
- x_perm_buffer: &x_perm_buf,
- expert_offsets: &offsets_buf,
- row_expert_map: &row_expert_map_buf,
- hidden_buffer: &hidden_buf,
- output_buffer: &y_partial_buf,
- w13_all: &w13_buf,
- w2_all: &w2_buf,
- up_biases: &up_biases_buf,
- down_biases: &down_biases_buf,
- tile_counts: &tile_counts_buf,
- tile_offsets: &tile_offsets_buf,
- tile_map: &tile_map_buf,
- total_tiles: &total_tiles_buf,
- dispatch_args: &dispatch_args_buf,
- total_rows: sum_k,
- d_model,
- d_ff,
- e,
- num_tiles_k: num_tiles_k as u32,
- gating_code: 2, // SILU
- gate_clip_min: f32::NEG_INFINITY,
- gate_clip_max: 20.0,
- up_clip_min: -19.0,
- up_clip_max: 21.0,
- silu_alpha: 1.702,
- data_type: KernelDataType::BFloat16,
- },
- )
- .expect("experts");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ experts_kernel.encode(
+ &cb,
+ &MoeExpertsTwoPassArguments {
+ x_perm_buffer: &x_perm_buf,
+ expert_offsets: &offsets_buf,
+ row_expert_map: &row_expert_map_buf,
+ hidden_buffer: &hidden_buf,
+ output_buffer: &y_partial_buf,
+ w13_all: &w13_buf,
+ w2_all: &w2_buf,
+ up_biases: &up_biases_buf,
+ down_biases: &down_biases_buf,
+ tile_counts: &tile_counts_buf,
+ tile_offsets: &tile_offsets_buf,
+ tile_map: &tile_map_buf,
+ total_tiles: &total_tiles_buf,
+ dispatch_args: &dispatch_args_buf,
+ total_rows: sum_k,
+ d_model,
+ d_ff,
+ e,
+ num_tiles_k: num_tiles_k as u32,
+ gating_code: 2, // SILU
+ gate_clip_min: f32::NEG_INFINITY,
+ gate_clip_max: 20.0,
+ up_clip_min: -19.0,
+ up_clip_max: 21.0,
+ silu_alpha: 1.702,
+ data_type: KernelDataType::BFloat16,
+ },
+ );
cb.commit();
cb.wait_until_completed();
});
let finalize_perf = time_kernel("Finalize", 2, 5, || {
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- finalize_kernel
- .encode(
- &cb,
- MoeFinalizeArguments {
- tok2row_buffer: &tok2row_buf,
- probs_buffer: &topk_probs_buf,
- y_partial_buffer: &y_partial_buf,
- y_out_buffer: &y_out_buf,
- t,
- d_model,
- k,
- },
- KernelDataType::BFloat16,
- )
- .expect("finalize");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ let encoder = cb.new_compute_command_encoder().expect("encoder");
+ finalize_kernel.encode(
+ &tok2row_buf,
+ &topk_probs_buf,
+ &y_partial_buf,
+ &y_out_buf,
+ t as u32,
+ d_model as u32,
+ k as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
cb.commit();
cb.wait_until_completed();
});
diff --git a/crates/uzu/tests/moe/moe_router_topk_test.rs b/crates/uzu/tests/moe/moe_router_topk_test.rs
index d3427279d..d3f0717f7 100644
--- a/crates/uzu/tests/moe/moe_router_topk_test.rs
+++ b/crates/uzu/tests/moe/moe_router_topk_test.rs
@@ -3,7 +3,7 @@
use metal::{MTLBuffer, MTLCommandBuffer, MTLCommandQueue};
use half::bf16;
-use rand::{Rng, SeedableRng, rngs::StdRng};
+use rand::{RngExt, SeedableRng, rngs::StdRng};
use uzu::backends::metal::{
KernelDataType, MTLContext,
kernel::moe::{MoeRouterTopKArguments, MoeRouterTopKKernel},
@@ -165,7 +165,10 @@ fn run_router_topk_once(
// For BFloat16 kernel, probs buffer must be bf16, not f32
let probs_buf = alloc_buffer::<bf16>(ctx, t * k);
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
let args = MoeRouterTopKArguments {
input_buffer: &input_buf,
weight_buffer: &weight_buf,
diff --git a/crates/uzu/tests/moe/moe_scatter_test.rs b/crates/uzu/tests/moe/moe_scatter_test.rs
index 27f849eda..a964fc5a3 100644
--- a/crates/uzu/tests/moe/moe_scatter_test.rs
+++ b/crates/uzu/tests/moe/moe_scatter_test.rs
@@ -1,17 +1,18 @@
#![cfg(any(target_os = "macos", target_os = "ios"))]
-use metal::{MTLBuffer, MTLCommandBuffer, MTLCommandQueue};
+use metal::{MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue};
+use super::test_utils::{alloc_buffer, alloc_buffer_with_data, create_ctx};
use half::bf16;
-use rand::{Rng, SeedableRng, rngs::StdRng};
+use rand::{RngExt, SeedableRng, rngs::StdRng};
+use uzu::backends::common::kernel::MoeCountsOffsetsFusedKernel;
+use uzu::backends::metal::kernel::dsl::MoeCountsOffsetsFusedMetalKernel;
use uzu::backends::metal::kernel::{
- KernelDataType, MoeBlockBasesArguments, MoeCountsOffsetsFusedArguments,
- MoeCountsOffsetsFusedKernel, MoeScatterArguments, MoeScatterKernels,
+ KernelDataType, MoeBlockBasesArguments, MoeScatterArguments,
+ MoeScatterKernels,
moe::{MoeRouterTopKArguments, MoeRouterTopKKernel},
};
-use super::test_utils::{alloc_buffer, alloc_buffer_with_data, create_ctx};
-
fn cpu_expert_buckets(
topk_ids: &[i32],
topk_probs: &[f32],
@@ -82,7 +83,10 @@ fn test_scatter_buckets_parity() {
// Use fused router+topk kernel
let router_topk = MoeRouterTopKKernel::new(&ctx).expect("router_topk");
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
router_topk
.encode(
&cb,
@@ -126,22 +130,23 @@ fn test_scatter_buckets_parity() {
let partials_buf = alloc_buffer::<u32>(&ctx, num_tiles * 512);
let fused_kernel =
- MoeCountsOffsetsFusedKernel::new(&ctx).expect("fused kernel");
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- fused_kernel
- .encode(
- &cb,
- MoeCountsOffsetsFusedArguments {
- topk_ids_buffer: &topk_ids_buf,
- offsets_buffer: &offsets_buf,
- sum_k_buffer: &sumk_buf,
- partials_buffer: &partials_buf,
- t,
- e,
- k,
- },
- )
- .expect("encode fused");
+ MoeCountsOffsetsFusedMetalKernel::new(&ctx).expect("fused kernel");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ let encoder = cb.new_compute_command_encoder().expect("encoder");
+ fused_kernel.encode(
+ &topk_ids_buf,
+ &offsets_buf,
+ &sumk_buf,
+ &partials_buf,
+ t as u32,
+ e as u32,
+ k as u32,
+ &encoder,
+ );
+ encoder.end_encoding();
cb.commit();
cb.wait_until_completed();
@@ -152,7 +157,10 @@ fn test_scatter_buckets_parity() {
let block_bases_buf = alloc_buffer::<u32>(&ctx, entries);
let scatter = MoeScatterKernels::new(&ctx).expect("scatter kernels");
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
let block_alloc_buf = alloc_buffer::<u32>(&ctx, entries);
scatter
.encode_block_bases(
@@ -171,12 +179,18 @@ fn test_scatter_buckets_parity() {
cb.wait_until_completed();
let sumk = unsafe {
- std::slice::from_raw_parts(sumk_buf.contents().as_ptr() as *const u32, 1)
+ std::slice::from_raw_parts(
+ sumk_buf.contents().as_ptr() as *const u32,
+ 1,
+ )
}[0] as usize;
let out_ids_buf = alloc_buffer::<i32>(&ctx, sumk);
let out_probs_buf = alloc_buffer::<bf16>(&ctx, sumk);
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
scatter
.encode_scatter(
&cb,
diff --git a/crates/uzu/tests/moe/moe_test_utils.rs b/crates/uzu/tests/moe/moe_test_utils.rs
index 183f853b7..f33b21708 100644
--- a/crates/uzu/tests/moe/moe_test_utils.rs
+++ b/crates/uzu/tests/moe/moe_test_utils.rs
@@ -5,17 +5,19 @@
#![cfg(any(target_os = "macos", target_os = "ios"))]
+use std::rc::Rc;
+
use bytemuck;
use half::bf16;
-use metal::{MTLDevice, MTLDeviceExt, MTLResourceOptions};
-use uzu::backends::metal::{MTLBuffer, MTLContext, ProtocolObject, Retained};
+use metal::{MTLDeviceExt, MTLResourceOptions};
+use uzu::backends::{
+ common::Context,
+ metal::{MTLBuffer, MTLContext, ProtocolObject, Retained},
+};
/// Create Metal context for testing
-pub fn create_ctx() -> MTLContext {
- let device = <dyn MTLDevice>::system_default().expect("No Metal device");
- let queue =
- device.new_command_queue().expect("Failed to create command queue");
- MTLContext::new(device, queue).expect("Failed to create MTLContext")
+pub fn create_ctx() -> Rc<MTLContext> {
+ MTLContext::new().expect("Failed to create MTLContext")
}
/// Helper to allocate buffer with data
@@ -25,9 +27,7 @@ pub fn alloc_buffer_with_data<T: bytemuck::NoUninit>(
) -> Retained<ProtocolObject<dyn MTLBuffer>> {
if data.is_empty() {
// Metal doesn't allow creating 0-byte buffers, create a minimal buffer instead
- ctx.device
- .new_buffer(1, MTLResourceOptions::STORAGE_MODE_SHARED)
- .expect("Failed to create empty buffer")
+ ctx.create_buffer(1).expect("Failed to create empty buffer")
} else {
ctx.device
.new_buffer_with_data(
@@ -43,12 +43,7 @@ pub fn alloc_buffer<T>(
ctx: &MTLContext,
count: usize,
) -> Retained<ProtocolObject<dyn MTLBuffer>> {
- ctx.device
- .new_buffer(
- count * std::mem::size_of::<T>(),
- MTLResourceOptions::STORAGE_MODE_SHARED,
- )
- .expect("Failed to create buffer")
+ ctx.create_buffer(count * size_of::<T>()).expect("Failed to create buffer")
}
/// Compare two bf16 slices with tolerance
diff --git a/crates/uzu/tests/moe/moe_tiles_test.rs b/crates/uzu/tests/moe/moe_tiles_test.rs
index 5879bd478..41a6d92a2 100644
--- a/crates/uzu/tests/moe/moe_tiles_test.rs
+++ b/crates/uzu/tests/moe/moe_tiles_test.rs
@@ -1,10 +1,9 @@
#![cfg(any(target_os = "macos", target_os = "ios"))]
use metal::{MTLBuffer, MTLCommandBuffer, MTLCommandQueue};
-
-use rand::{Rng, SeedableRng, rngs::StdRng};
-use uzu::backends::metal::kernel::moe::{
- MoeTileCountsArguments, MoeTileMapKernel, MoeTileScanArguments,
+use rand::{RngExt, SeedableRng, rngs::StdRng};
+use uzu::backends::metal::kernel::moe::tiles_map::{
+ MoeTileCountsArguments, MoeTileMapKernels, MoeTileScanArguments,
};
use super::test_utils::{
@@ -38,18 +37,19 @@ fn test_tile_counts_correctness() {
// Execute kernel using kernel struct
let tile_kernel =
- MoeTileMapKernel::new(&ctx).expect("MoeTileMapKernel::new");
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- tile_kernel
- .encode_counts(
- &cb,
- &MoeTileCountsArguments {
- offsets_buffer: &offsets_buf,
- tile_counts_buffer: &tile_counts_buf,
- e,
- },
- )
- .expect("encode tile counts");
+ MoeTileMapKernels::new(&ctx).expect("MoeTileMapKernel::new");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ tile_kernel.encode_counts(
+ &cb,
+ &MoeTileCountsArguments {
+ offsets_buffer: &offsets_buf,
+ tile_counts_buffer: &tile_counts_buf,
+ e,
+ },
+ );
cb.commit();
cb.wait_until_completed();
@@ -96,19 +96,20 @@ fn test_tile_scan_correctness() {
// Execute kernel using kernel struct
let tile_kernel =
- MoeTileMapKernel::new(&ctx).expect("MoeTileMapKernel::new");
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- tile_kernel
- .encode_scan(
- &cb,
- &MoeTileScanArguments {
- tile_counts_buffer: &tile_counts_buf,
- tile_offsets_buffer: &tile_offsets_buf,
- total_tiles_buffer: &total_tiles_buf,
- e,
- },
- )
- .expect("encode tile scan");
+ MoeTileMapKernels::new(&ctx).expect("MoeTileMapKernel::new");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ tile_kernel.encode_scan(
+ &cb,
+ &MoeTileScanArguments {
+ tile_counts_buffer: &tile_counts_buf,
+ tile_offsets_buffer: &tile_offsets_buf,
+ total_tiles_buffer: &total_tiles_buf,
+ e,
+ },
+ );
cb.commit();
cb.wait_until_completed();
@@ -153,18 +154,19 @@ fn test_tile_edge_cases() {
let tile_counts_buf = alloc_buffer::<u32>(&ctx, e);
let tile_kernel =
- MoeTileMapKernel::new(&ctx).expect("MoeTileMapKernel::new");
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- tile_kernel
- .encode_counts(
- &cb,
- &MoeTileCountsArguments {
- offsets_buffer: &offsets_buf,
- tile_counts_buffer: &tile_counts_buf,
- e,
- },
- )
- .expect("encode tile counts");
+ MoeTileMapKernels::new(&ctx).expect("MoeTileMapKernel::new");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ tile_kernel.encode_counts(
+ &cb,
+ &MoeTileCountsArguments {
+ offsets_buffer: &offsets_buf,
+ tile_counts_buffer: &tile_counts_buf,
+ e,
+ },
+ );
cb.commit();
cb.wait_until_completed();
@@ -191,18 +193,19 @@ fn test_tile_edge_cases() {
let tile_counts_buf = alloc_buffer::<u32>(&ctx, e);
let tile_kernel =
- MoeTileMapKernel::new(&ctx).expect("MoeTileMapKernel::new");
- let cb = ctx.command_queue.command_buffer().expect("Failed to create command buffer");
- tile_kernel
- .encode_counts(
- &cb,
- &MoeTileCountsArguments {
- offsets_buffer: &offsets_buf,
- tile_counts_buffer: &tile_counts_buf,
- e,
- },
- )
- .expect("encode tile counts");
+ MoeTileMapKernels::new(&ctx).expect("MoeTileMapKernel::new");
+ let cb = ctx
+ .command_queue
+ .command_buffer()
+ .expect("Failed to create command buffer");
+ tile_kernel.encode_counts(
+ &cb,
+ &MoeTileCountsArguments {
+ offsets_buffer: &offsets_buf,
+ tile_counts_buffer: &tile_counts_buf,
+ e,
+ },
+ );
cb.commit();
cb.wait_until_completed();
diff --git a/crates/uzu/tests/parameters_loader_test.rs b/crates/uzu/tests/parameters_loader_test.rs
index 30178eae1..6fb1b2b48 100644
--- a/crates/uzu/tests/parameters_loader_test.rs
+++ b/crates/uzu/tests/parameters_loader_test.rs
@@ -3,23 +3,27 @@ mod common;
// New integration test for ParameterLoader
use half::bf16;
use is_close::is_close;
-use uzu::{Array, backends::cpu::CPUContext, parameters::ParameterLoader};
+use uzu::{
+ backends::{common::Context, metal::MTLContext},
+ parameters::ParameterLoader,
+};
#[test]
fn test_parameter_loader_basic() {
let weights_path = crate::common::get_test_weights_path();
- let context = CPUContext::new();
+ let context = MTLContext::new().expect("Failed to create MTLContext");
let file = std::fs::File::open(&weights_path)
.expect("Weights file not found; run download script");
- let loader = ParameterLoader::new(&file, &context).expect("create loader");
+ let loader =
+ ParameterLoader::new(&file, context.as_ref()).expect("create loader");
let embeddings =
loader.get("embedding.weights").expect("weights embeddings");
- let view = embeddings.as_view::<bf16>().unwrap();
+ let view = embeddings.as_view::<bf16>();
assert!(is_close!(view[[5usize, 3usize]], bf16::from_f32(-0.01819)));
// tree API check
let subtree = loader.tree().subtree("embedding").unwrap();
let same = subtree.leaf("weights").unwrap();
- assert_eq!(view, same.as_view::<bf16>().unwrap());
+ assert_eq!(view, same.as_view::<bf16>());
}
diff --git a/crates/uzu/tests/person_schema_metal_bitmask_test.rs b/crates/uzu/tests/person_schema_metal_bitmask_test.rs
index 6d3253a84..4b045991c 100644
--- a/crates/uzu/tests/person_schema_metal_bitmask_test.rs
+++ b/crates/uzu/tests/person_schema_metal_bitmask_test.rs
@@ -2,11 +2,11 @@
use std::path::PathBuf;
-use metal::{MTLBuffer, MTLDevice, MTLDeviceExt, MTLResourceOptions};
+use metal::{MTLDevice, MTLDeviceExt, MTLResourceOptions};
use schemars::JsonSchema;
use serde::{Deserialize, Serialize};
use tokenizers::Tokenizer;
-use uzu::{Array, DataType, backends::metal::MetalArray};
+use uzu::{DataType, backends::metal::MetalArray};
use xgrammar::{
DLDevice, DLDeviceType, DLTensor, Grammar, GrammarCompiler, GrammarMatcher,
TokenizerInfo,
@@ -83,12 +83,12 @@ fn person_schema_metal_bitmask() {
let buffer = device
.new_buffer(bytes, MTLResourceOptions::STORAGE_MODE_SHARED)
.expect("Failed to create buffer");
- let mut metal_bitmask =
- unsafe { MetalArray::new(buffer, &shape, DataType::I32) };
+ let metal_bitmask =
+ unsafe { MetalArray::from_parts(buffer, 0, &shape, DataType::I32) };
let mut shape_i64 = [buffer_size as i64];
let mut bitmask_tensor = DLTensor {
- data: unsafe { metal_bitmask.mtl_buffer().contents().as_ptr() },
+ data: metal_bitmask.cpu_ptr().as_ptr(),
device: DLDevice {
device_type: DLDeviceType::kDLCPU,
device_id,
@@ -103,5 +103,5 @@ fn person_schema_metal_bitmask() {
let _ok = matcher.fill_next_token_bitmask(&mut bitmask_tensor, 0, false);
assert_eq!(metal_bitmask.shape(), &shape);
- assert_eq!(metal_bitmask.as_slice::<i32>().unwrap().len(), elems);
+ assert_eq!(metal_bitmask.as_slice::<i32>().len(), elems);
}
diff --git a/crates/uzu/tests/quant_matmul_test.rs b/crates/uzu/tests/quant_matmul_test.rs
index cf849ca2f..8ea1e3b1c 100644
--- a/crates/uzu/tests/quant_matmul_test.rs
+++ b/crates/uzu/tests/quant_matmul_test.rs
@@ -3,26 +3,24 @@ use std::time::Instant;
use bytemuck;
use half::{bf16, f16};
use metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue, MTLDevice,
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue,
MTLDeviceExt, MTLResourceOptions,
};
use uzu::{
DataType,
- backends::metal::{
- MTLContext, ProtocolObject, Retained,
- kernel::quant_matmul::{
- QuantizationType, QuantizedMatmulArguments, QuantizedMatmulKernel,
+ backends::{
+ common::Context,
+ metal::{
+ MTLContext, ProtocolObject, Retained,
+ kernel::quant_matmul::{
+ QuantizationType, QuantizedMatmulArguments,
+ QuantizedMatmulKernel,
+ },
},
},
config::QuantizationMode,
};
-fn create_test_context() -> Option<MTLContext> {
- let device = <dyn MTLDevice>::system_default()?;
- let command_queue = device.new_command_queue()?;
- MTLContext::new(device, command_queue).ok()
-}
-
fn pack_u4_weights(values: &[u8]) -> Vec<u8> {
let mut out = Vec::with_capacity((values.len() + 3) / 4 * 2);
for chunk in values.chunks(4) {
@@ -526,11 +524,7 @@ fn execute_quantized_matmul(
};
let x_buf = buffer_from_f32_slice(ctx, data_type, &x_f32);
let y_buf = ctx
- .device
- .new_buffer(
- batch * output_dim * data_type.size_in_bytes() ,
- MTLResourceOptions::STORAGE_MODE_SHARED,
- )
+ .create_buffer(batch * output_dim * data_type.size_in_bytes())
.expect("Failed to create buffer");
let kernel = QuantizedMatmulKernel::new(
@@ -747,9 +741,9 @@ fn run_kernel_test(
#[test]
fn test_quant_gmv() {
- let ctx = match create_test_context() {
- Some(c) => c,
- None => {
+ let ctx = match MTLContext::new() {
+ Ok(c) => c,
+ Err(_) => {
println!("Metal not available — skipping QMV test");
return;
},
@@ -793,9 +787,9 @@ fn test_quant_gmv() {
#[test]
fn test_quant_qvm() {
- let ctx = match create_test_context() {
- Some(c) => c,
- None => {
+ let ctx = match MTLContext::new() {
+ Ok(c) => c,
+ Err(_) => {
println!("Metal not available — skipping QVM test");
return;
},
@@ -839,9 +833,9 @@ fn test_quant_qvm() {
#[test]
fn test_quant_gmm() {
- let ctx = match create_test_context() {
- Some(c) => c,
- None => {
+ let ctx = match MTLContext::new() {
+ Ok(c) => c,
+ Err(_) => {
println!("Metal not available — skipping QMM test");
return;
},
@@ -885,9 +879,9 @@ fn test_quant_gmm() {
#[test]
fn test_quant_gmm_transposed() {
- let ctx = match create_test_context() {
- Some(c) => c,
- None => {
+ let ctx = match MTLContext::new() {
+ Ok(c) => c,
+ Err(_) => {
println!("Metal not available — skipping QMM transposed test");
return;
},
@@ -932,9 +926,9 @@ fn test_quant_gmm_transposed() {
#[test]
#[ignore]
fn test_quant_matmul_perf() {
- let ctx = match create_test_context() {
- Some(c) => c,
- None => {
+ let ctx = match MTLContext::new() {
+ Ok(c) => c,
+ Err(_) => {
println!("Metal not available — skipping Perf test");
return;
},
diff --git a/crates/uzu/tests/rms_norm_kernel_test.rs b/crates/uzu/tests/rms_norm_kernel_test.rs
index ac7ac3d39..c4225ad4b 100644
--- a/crates/uzu/tests/rms_norm_kernel_test.rs
+++ b/crates/uzu/tests/rms_norm_kernel_test.rs
@@ -4,30 +4,24 @@
use bytemuck;
use half::{bf16, f16};
use metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue, MTLDevice, MTLDeviceExt,
- MTLResourceOptions,
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue,
+ MTLDeviceExt, MTLResourceOptions,
};
use uzu::{
DataType,
- backends::metal::{
- MTLContext,
- kernel::rms_norm::{
- QKNormArguments, QKNormTarget, RMSNormArguments, RMSNormKernel,
- RMSNormKernelType,
+ backends::{
+ common::Context,
+ metal::{
+ MTLContext,
+ kernel::rms_norm::{
+ QKNormArguments, QKNormTarget, RMSNormArguments, RMSNormKernel,
+ RMSNormKernelType,
+ },
+ metal_extensions::CommandBufferTimingExt,
},
- metal_extensions::CommandBufferTimingExt,
},
};
-fn create_test_context() -> Result<MTLContext, String> {
- let device =
- <dyn MTLDevice>::system_default().ok_or("No Metal device available")?;
- let command_queue =
- device.new_command_queue().ok_or("Failed to create command queue")?;
- MTLContext::new(device, command_queue)
- .map_err(|e| format!("Failed to create MTLContext: {:?}", e))
-}
-
// Helper trait to unify different float types for testing
trait TestFloat:
Copy + Clone + std::fmt::Debug + PartialEq + bytemuck::NoUninit
@@ -315,7 +309,7 @@ fn test_rms_norm_basic_typed<InputT, ScaleT, OutputT>(
OutputT: TestFloat,
{
// Create Metal context
- let mtl_context = match create_test_context() {
+ let mtl_context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping RMS norm test: {}", e);
@@ -484,7 +478,7 @@ fn test_rms_norm_edge_cases_typed<InputT, ScaleT, OutputT>(
ScaleT: TestFloat,
OutputT: TestFloat,
{
- let mtl_context = match create_test_context() {
+ let mtl_context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping RMS norm edge case test: {}", e);
@@ -830,10 +824,10 @@ fn perf_rms_norm_with_size(
) {
use std::time::Instant;
- use rand::{Rng, SeedableRng, rngs::StdRng};
+ use rand::{RngExt, SeedableRng, rngs::StdRng};
// ---- Metal context ----
- let mtl_context = match create_test_context() {
+ let mtl_context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping RMS norm perf test: {}", e);
@@ -996,7 +990,7 @@ fn perf_rms_norm_16k() {
#[test]
fn qk_norm_test() {
// Test to verify that the QK norm kernel now accesses the correct data ranges
- let mtl_context = match create_test_context() {
+ let mtl_context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping QK norm buffer addressing test: {}", e);
diff --git a/crates/uzu/tests/sampling_test.rs b/crates/uzu/tests/sampling_test.rs
index a9df612b9..5d34cb4e5 100644
--- a/crates/uzu/tests/sampling_test.rs
+++ b/crates/uzu/tests/sampling_test.rs
@@ -1,23 +1,30 @@
mod common;
use bytemuck;
use metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue, MTLDevice, MTLDeviceExt,
- MTLResourceOptions,
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue,
+ MTLDeviceExt, MTLResourceOptions,
};
-use rand::{Rng, seq::SliceRandom};
+use rand::{RngExt, seq::SliceRandom};
// for Vec::shuffle
use uzu::{
- backends::metal::{
- KernelDataType, MTLContext,
- kernel::{
- SamplingKernel,
- dsl::{
+ DataType,
+ backends::{
+ common::{
+ Context,
+ kernel::{
GumbelKernel, MinPKernel, TemperatureKernel, TopKKernel,
TopPKernel,
+ sampling::{ArgmaxStrategy, SamplingKernel},
},
- sampling::ArgmaxStrategy,
},
- metal_extensions::CommandBufferTimingExt,
+ metal::{
+ KernelDataType, MTLContext, Metal,
+ kernel::dsl::{
+ GumbelMetalKernel, MinPMetalKernel, TemperatureMetalKernel,
+ TopKMetalKernel, TopPMetalKernel,
+ },
+ metal_extensions::CommandBufferTimingExt,
+ },
},
language_model::gumbel::{gumbel_float, revidx},
session::parameter::SamplingMethod,
@@ -26,15 +33,6 @@ use uzu::{
// Constant seed for reproducible test results
const TEST_SAMPLING_SEED: u64 = 42;
-fn create_test_context() -> Result<MTLContext, String> {
- let device =
- <dyn MTLDevice>::system_default().ok_or("No Metal device available")?;
- let command_queue =
- device.new_command_queue().ok_or("Failed to create command queue")?;
- MTLContext::new(device, command_queue)
- .map_err(|e| format!("Failed to create MTLContext: {:?}", e))
-}
-
fn cpu_reference_top_p(
row_logits: &[f32],
top_p: f32,
@@ -93,7 +91,7 @@ fn cpu_reference_min_p(
}
fn test_argmax_sampling_with_strategy(strategy: ArgmaxStrategy) {
- let context = match create_test_context() {
+ let context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping argmax test: {}", e);
@@ -104,9 +102,9 @@ fn test_argmax_sampling_with_strategy(strategy: ArgmaxStrategy) {
let batch_size = 2;
let vocab_size = 4;
- let kernel = SamplingKernel::new_with_strategy(
+ let kernel = SamplingKernel::<Metal>::new_with_strategy(
&context,
- KernelDataType::Float32,
+ DataType::F32,
batch_size,
vocab_size,
strategy,
@@ -144,8 +142,11 @@ fn test_argmax_sampling_with_strategy(strategy: ArgmaxStrategy) {
let command_buffer = command_buffer_ref.to_owned();
// Run sampling
+ let compute_encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute encoder");
kernel
- .encode(
+ .encode_with_encoder(
&logits_buffer,
None,
0,
@@ -155,9 +156,10 @@ fn test_argmax_sampling_with_strategy(strategy: ArgmaxStrategy) {
SamplingMethod::Greedy,
batch_size,
vocab_size,
- &command_buffer,
+ &compute_encoder,
)
.expect("Argmax sampling should succeed");
+ compute_encoder.end_encoding();
command_buffer_ref.commit();
command_buffer_ref.wait_until_completed();
@@ -206,7 +208,7 @@ fn test_topp_sampling_from_prob_exact_match(
use rand::{SeedableRng, rngs::StdRng};
- let context = match create_test_context() {
+ let context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
eprintln!("Skipping topP exact match test: {}", e);
@@ -218,9 +220,9 @@ fn test_topp_sampling_from_prob_exact_match(
let p = (k as f32) * 0.1;
// Create kernel
- let kernel = SamplingKernel::new(
+ let kernel = SamplingKernel::<Metal>::new(
&context,
- KernelDataType::Float32,
+ DataType::F32,
batch_size,
vocab_size,
)
@@ -300,8 +302,11 @@ fn test_topp_sampling_from_prob_exact_match(
.command_buffer()
.expect("Failed to create command buffer");
let cb = cb_ref.to_owned();
+ let compute_encoder = cb
+ .new_compute_command_encoder()
+ .expect("Failed to create compute encoder");
kernel
- .encode(
+ .encode_with_encoder(
&logits_buf,
Some(&seeds_buf),
0,
@@ -316,9 +321,10 @@ fn test_topp_sampling_from_prob_exact_match(
},
batch_size,
vocab_size,
- &cb,
+ &compute_encoder,
)
.expect("encode");
+ compute_encoder.end_encoding();
cb_ref.commit();
cb_ref.wait_until_completed();
@@ -367,10 +373,10 @@ fn test_topp_sampling_match_large() {
#[test]
fn test_topp_sampling_statistical_large() {
- use rand::{Rng, SeedableRng, rngs::StdRng};
+ use rand::{RngExt, SeedableRng, rngs::StdRng};
// ===== 1. Create Metal context =====
- let context = match create_test_context() {
+ let context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping large statistical test: {}", e);
@@ -387,7 +393,7 @@ fn test_topp_sampling_statistical_large() {
// ===== 3. Build kernel =====
let kernel =
- SamplingKernel::new(&context, KernelDataType::Float32, BATCH, VOCAB)
+ SamplingKernel::<Metal>::new(&context, DataType::F32, BATCH, VOCAB)
.expect("Failed to create sampling kernel");
// ===== 4. Generate reproducible random logits =====
@@ -441,9 +447,12 @@ fn test_topp_sampling_statistical_large() {
.command_buffer()
.expect("Failed to create command buffer");
let cb = cb_ref.to_owned();
+ let compute_encoder = cb
+ .new_compute_command_encoder()
+ .expect("Failed to create compute encoder");
kernel
- .encode(
+ .encode_with_encoder(
&logits_buf,
Some(&seeds_buf),
0,
@@ -458,9 +467,10 @@ fn test_topp_sampling_statistical_large() {
},
BATCH,
VOCAB,
- &cb,
+ &compute_encoder,
)
.expect("encode");
+ compute_encoder.end_encoding();
cb_ref.commit();
cb_ref.wait_until_completed();
@@ -505,10 +515,10 @@ fn test_topp_sampling_statistical_large() {
fn perf_topp_128k_vocab() {
use std::time::Instant;
- use rand::{Rng, SeedableRng, rngs::StdRng};
+ use rand::{RngExt, SeedableRng, rngs::StdRng};
// ---- Metal context ----
- let context = match create_test_context() {
+ let context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping perf test: {}", e);
@@ -523,7 +533,7 @@ fn perf_topp_128k_vocab() {
// ---- Kernel ----
let kernel =
- SamplingKernel::new(&context, KernelDataType::Float32, BATCH, VOCAB)
+ SamplingKernel::<Metal>::new(&context, DataType::F32, BATCH, VOCAB)
.expect("Failed to create sampling kernel");
// ---- Build random logits ----
@@ -553,7 +563,7 @@ fn perf_topp_128k_vocab() {
let output_buf = context
.device
.new_buffer(
- BATCH * std::mem::size_of::<u32>() ,
+ BATCH * std::mem::size_of::<u32>(),
metal::MTLResourceOptions::STORAGE_MODE_SHARED,
)
.expect("Failed to create buffer");
@@ -564,9 +574,12 @@ fn perf_topp_128k_vocab() {
.command_buffer()
.expect("Failed to create command buffer");
let cb = cb_ref.to_owned();
+ let compute_encoder = cb
+ .new_compute_command_encoder()
+ .expect("Failed to create compute encoder");
kernel
- .encode(
+ .encode_with_encoder(
&logits_buf,
Some(&seeds_buf),
0,
@@ -581,9 +594,10 @@ fn perf_topp_128k_vocab() {
},
BATCH,
VOCAB,
- &cb,
+ &compute_encoder,
)
.expect("encode");
+ compute_encoder.end_encoding();
// Time both host-side and GPU execution
let host_timer = Instant::now();
@@ -621,10 +635,10 @@ fn perf_topp_128k_vocab() {
fn perf_argmax_128k_vocab_with_strategy(strategy: ArgmaxStrategy) {
use std::time::Instant;
- use rand::{Rng, SeedableRng, rngs::StdRng};
+ use rand::{RngExt, SeedableRng, rngs::StdRng};
// ---- Metal context ----
- let context = match create_test_context() {
+ let context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping argmax perf test: {}", e);
@@ -637,9 +651,9 @@ fn perf_argmax_128k_vocab_with_strategy(strategy: ArgmaxStrategy) {
const VOCAB: usize = 128000; // 128K
// ---- Kernel ----
- let kernel = SamplingKernel::new_with_strategy(
+ let kernel = SamplingKernel::<Metal>::new_with_strategy(
&context,
- KernelDataType::Float32,
+ DataType::F32,
BATCH,
VOCAB,
strategy,
@@ -673,7 +687,7 @@ fn perf_argmax_128k_vocab_with_strategy(strategy: ArgmaxStrategy) {
let output_buf = context
.device
.new_buffer(
- BATCH * std::mem::size_of::<u32>() ,
+ BATCH * std::mem::size_of::<u32>(),
metal::MTLResourceOptions::STORAGE_MODE_SHARED,
)
.expect("Failed to create buffer");
@@ -684,9 +698,12 @@ fn perf_argmax_128k_vocab_with_strategy(strategy: ArgmaxStrategy) {
.command_buffer()
.expect("Failed to create command buffer");
let cb = cb_ref.to_owned();
+ let compute_encoder = cb
+ .new_compute_command_encoder()
+ .expect("Failed to create compute encoder");
kernel
- .encode(
+ .encode_with_encoder(
&logits_buf,
Some(&seeds_buf),
0,
@@ -696,9 +713,10 @@ fn perf_argmax_128k_vocab_with_strategy(strategy: ArgmaxStrategy) {
SamplingMethod::Greedy,
BATCH,
VOCAB,
- &cb,
+ &compute_encoder,
)
.expect("encode");
+ compute_encoder.end_encoding();
// Time both host-side and GPU execution
let host_timer = Instant::now();
@@ -756,7 +774,7 @@ fn perf_argmax_128k_vocab_with_strategy(strategy: ArgmaxStrategy) {
#[test]
fn test_categorical_sampling() {
- let context = match create_test_context() {
+ let context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping categorical test: {}", e);
@@ -767,9 +785,9 @@ fn test_categorical_sampling() {
let batch_size = 2;
let vocab_size = 4;
- let kernel = SamplingKernel::new(
+ let kernel = SamplingKernel::<Metal>::new(
&context,
- KernelDataType::Float32,
+ DataType::F32,
batch_size,
vocab_size,
)
@@ -786,7 +804,7 @@ fn test_categorical_sampling() {
let output_buffer = context
.device
.new_buffer(
- batch_size * std::mem::size_of::<u32>() ,
+ batch_size * std::mem::size_of::<u32>(),
MTLResourceOptions::STORAGE_MODE_SHARED,
)
.expect("Failed to create buffer");
@@ -820,9 +838,12 @@ fn test_categorical_sampling() {
.command_buffer()
.expect("Failed to create command buffer");
let command_buffer = command_buffer_ref.to_owned();
+ let compute_encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute encoder");
kernel
- .encode(
+ .encode_with_encoder(
&logits_buffer,
Some(&seeds_buffer),
0,
@@ -837,9 +858,10 @@ fn test_categorical_sampling() {
},
batch_size,
vocab_size,
- &command_buffer,
+ &compute_encoder,
)
.expect("Categorical sampling should succeed");
+ compute_encoder.end_encoding();
command_buffer_ref.commit();
command_buffer_ref.wait_until_completed();
@@ -897,9 +919,9 @@ fn test_categorical_sampling() {
#[test]
fn test_categorical_sampling_statistical() {
- use rand::{Rng, SeedableRng, rngs::StdRng};
+ use rand::{RngExt, SeedableRng, rngs::StdRng};
- let context = match create_test_context() {
+ let context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping categorical statistical test: {}", e);
@@ -913,7 +935,7 @@ fn test_categorical_sampling_statistical() {
const TOLERANCE: f32 = 0.05; // 5% tolerance
let kernel =
- SamplingKernel::new(&context, KernelDataType::Float32, BATCH, VOCAB)
+ SamplingKernel::<Metal>::new(&context, DataType::F32, BATCH, VOCAB)
.expect("Failed to create sampling kernel");
// Generate random logits
@@ -946,7 +968,7 @@ fn test_categorical_sampling_statistical() {
let output_buffer = context
.device
.new_buffer(
- BATCH * std::mem::size_of::<u32>() ,
+ BATCH * std::mem::size_of::<u32>(),
MTLResourceOptions::STORAGE_MODE_SHARED,
)
.expect("Failed to create buffer");
@@ -978,9 +1000,12 @@ fn test_categorical_sampling_statistical() {
.command_buffer()
.expect("Failed to create command buffer");
let command_buffer = command_buffer_ref.to_owned();
+ let compute_encoder = command_buffer
+ .new_compute_command_encoder()
+ .expect("Failed to create compute encoder");
kernel
- .encode(
+ .encode_with_encoder(
&logits_buffer,
Some(&seeds_buffer),
0,
@@ -995,9 +1020,10 @@ fn test_categorical_sampling_statistical() {
},
BATCH,
VOCAB,
- &command_buffer,
+ &compute_encoder,
)
.expect("encode");
+ compute_encoder.end_encoding();
command_buffer_ref.commit();
command_buffer_ref.wait_until_completed();
@@ -1044,9 +1070,9 @@ fn test_categorical_sampling_statistical() {
fn perf_categorical_128k_vocab() {
use std::time::Instant;
- use rand::{Rng, SeedableRng, rngs::StdRng};
+ use rand::{RngExt, SeedableRng, rngs::StdRng};
- let context = match create_test_context() {
+ let context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping categorical perf test: {}", e);
@@ -1058,7 +1084,7 @@ fn perf_categorical_128k_vocab() {
const VOCAB: usize = 128000; // 128K
let kernel =
- SamplingKernel::new(&context, KernelDataType::Float32, BATCH, VOCAB)
+ SamplingKernel::<Metal>::new(&context, DataType::F32, BATCH, VOCAB)
.expect("Failed to create sampling kernel");
// Build random logits
@@ -1088,7 +1114,7 @@ fn perf_categorical_128k_vocab() {
let output_buf = context
.device
.new_buffer(
- BATCH * std::mem::size_of::<u32>() ,
+ BATCH * std::mem::size_of::<u32>(),
metal::MTLResourceOptions::STORAGE_MODE_SHARED,
)
.expect("Failed to create buffer");
@@ -1099,9 +1125,12 @@ fn perf_categorical_128k_vocab() {
.command_buffer()
.expect("Failed to create command buffer");
let cb = cb_ref.to_owned();
+ let compute_encoder = cb
+ .new_compute_command_encoder()
+ .expect("Failed to create compute encoder");
kernel
- .encode(
+ .encode_with_encoder(
&logits_buf,
Some(&seeds_buf),
0,
@@ -1116,9 +1145,10 @@ fn perf_categorical_128k_vocab() {
},
BATCH,
VOCAB,
- &cb,
+ &compute_encoder,
)
.expect("encode");
+ compute_encoder.end_encoding();
let host_timer = Instant::now();
cb_ref.commit();
@@ -1154,7 +1184,7 @@ fn perf_categorical_128k_vocab() {
#[test]
fn test_temperature_gpu_cpu_match() {
- let context = match create_test_context() {
+ let context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping temperature gpu cpu match test: {}", e);
@@ -1168,8 +1198,9 @@ fn test_temperature_gpu_cpu_match() {
const RTOL: f32 = 1e-6;
const ATOL: f32 = 1e-6;
- let kernel = TemperatureKernel::new(&context, KernelDataType::Float32)
- .expect("Failed to create temperature kernel");
+ let kernel =
+ TemperatureMetalKernel::new(&context, KernelDataType::Float32.into())
+ .expect("Failed to create temperature kernel");
let logits: Vec<f32> = (0..BATCH * VOCAB)
.map(|i| ((i * 37 % 1000) as f32 - 500.0) * 0.01)
@@ -1186,7 +1217,7 @@ fn test_temperature_gpu_cpu_match() {
let processed_buffer = context
.device
.new_buffer(
- logits.len() * std::mem::size_of::<f32>() ,
+ logits.len() * std::mem::size_of::<f32>(),
MTLResourceOptions::STORAGE_MODE_SHARED,
)
.expect("Failed to create buffer");
@@ -1244,7 +1275,7 @@ fn test_temperature_gpu_cpu_match() {
fn test_topk_gpu_cpu_match() {
use rand::{SeedableRng, rngs::StdRng};
- let context = match create_test_context() {
+ let context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping topk gpu cpu match test: {}", e);
@@ -1256,7 +1287,7 @@ fn test_topk_gpu_cpu_match() {
const VOCAB: usize = 1024;
const TOPK: u32 = 16;
- let kernel = TopKKernel::new(&context, KernelDataType::Float32)
+ let kernel = TopKMetalKernel::new(&context, KernelDataType::Float32.into())
.expect("Failed to create topk kernel");
let mut rng = StdRng::seed_from_u64(42);
@@ -1276,7 +1307,7 @@ fn test_topk_gpu_cpu_match() {
let processed_buffer = context
.device
.new_buffer(
- logits.len() * std::mem::size_of::<f32>() ,
+ logits.len() * std::mem::size_of::<f32>(),
MTLResourceOptions::STORAGE_MODE_SHARED,
)
.expect("Failed to create buffer");
@@ -1330,7 +1361,7 @@ fn test_topk_gpu_cpu_match() {
fn test_topp_gpu_cpu_match() {
use rand::{SeedableRng, rngs::StdRng};
- let context = match create_test_context() {
+ let context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping topp gpu cpu match test: {}", e);
@@ -1342,7 +1373,7 @@ fn test_topp_gpu_cpu_match() {
const VOCAB: usize = 1024;
const TOPP: f32 = 0.9;
- let kernel = TopPKernel::new(&context, KernelDataType::Float32)
+ let kernel = TopPMetalKernel::new(&context, KernelDataType::Float32.into())
.expect("Failed to create topp kernel");
let mut rng = StdRng::seed_from_u64(42);
@@ -1362,7 +1393,7 @@ fn test_topp_gpu_cpu_match() {
let processed_buffer = context
.device
.new_buffer(
- logits.len() * std::mem::size_of::<f32>() ,
+ logits.len() * std::mem::size_of::<f32>(),
MTLResourceOptions::STORAGE_MODE_SHARED,
)
.expect("Failed to create buffer");
@@ -1418,9 +1449,9 @@ fn test_topp_gpu_cpu_match() {
#[test]
fn test_minp_gpu_cpu_match() {
- use rand::{Rng, SeedableRng, rngs::StdRng};
+ use rand::{RngExt, SeedableRng, rngs::StdRng};
- let context = match create_test_context() {
+ let context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping minp gpu cpu match test: {}", e);
@@ -1432,7 +1463,7 @@ fn test_minp_gpu_cpu_match() {
const VOCAB: usize = 1024;
const MINP: f32 = 0.1;
- let kernel = MinPKernel::new(&context, KernelDataType::Float32)
+ let kernel = MinPMetalKernel::new(&context, KernelDataType::Float32.into())
.expect("Failed to create minp kernel");
let mut rng = StdRng::seed_from_u64(42);
@@ -1452,7 +1483,7 @@ fn test_minp_gpu_cpu_match() {
let processed_buffer = context
.device
.new_buffer(
- logits.len() * std::mem::size_of::<f32>() ,
+ logits.len() * std::mem::size_of::<f32>(),
MTLResourceOptions::STORAGE_MODE_SHARED,
)
.expect("Failed to create buffer");
@@ -1505,7 +1536,7 @@ fn test_minp_sampling_exact_match(
use rand::{SeedableRng, rngs::StdRng};
- let context = match create_test_context() {
+ let context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
eprintln!("Skipping minp exact match test: {}", e);
@@ -1513,9 +1544,9 @@ fn test_minp_sampling_exact_match(
},
};
- let kernel = SamplingKernel::new(
+ let kernel = SamplingKernel::<Metal>::new(
&context,
- KernelDataType::Float32,
+ DataType::F32,
batch_size,
vocab_size,
)
@@ -1549,7 +1580,7 @@ fn test_minp_sampling_exact_match(
let output_buf = context
.device
.new_buffer(
- batch_size * std::mem::size_of::<u32>() ,
+ batch_size * std::mem::size_of::<u32>(),
metal::MTLResourceOptions::STORAGE_MODE_SHARED,
)
.expect("Failed to create buffer");
@@ -1582,8 +1613,11 @@ fn test_minp_sampling_exact_match(
.command_buffer()
.expect("Failed to create command buffer");
let cb = cb_ref.to_owned();
+ let compute_encoder = cb
+ .new_compute_command_encoder()
+ .expect("Failed to create compute encoder");
kernel
- .encode(
+ .encode_with_encoder(
&logits_buf,
Some(&seeds_buf),
0,
@@ -1598,9 +1632,10 @@ fn test_minp_sampling_exact_match(
},
batch_size,
vocab_size,
- &cb,
+ &compute_encoder,
)
.expect("encode");
+ compute_encoder.end_encoding();
cb_ref.commit();
cb_ref.wait_until_completed();
@@ -1649,7 +1684,7 @@ fn test_minp_sampling_match_large() {
#[test]
fn test_gumbel_gpu_cpu_match() {
- let context = match create_test_context() {
+ let context = match MTLContext::new() {
Ok(ctx) => ctx,
Err(e) => {
println!("Skipping gumbel gpu cpu match test: {}", e);
@@ -1662,8 +1697,9 @@ fn test_gumbel_gpu_cpu_match() {
const RTOL: f32 = 0.01;
const ATOL: f32 = 1e-6;
- let kernel = GumbelKernel::new(&context, KernelDataType::Float32)
- .expect("Failed to create gumbel kernel");
+ let kernel =
+ GumbelMetalKernel::new(&context, KernelDataType::Float32.into())
+ .expect("Failed to create gumbel kernel");
let logits = vec![0.0f32; BATCH * VOCAB];
let seeds: Vec<u64> = (0_u64..BATCH as u64).collect();
@@ -1687,7 +1723,7 @@ fn test_gumbel_gpu_cpu_match() {
let gumbel_logits_buffer = context
.device
.new_buffer(
- BATCH * VOCAB * std::mem::size_of::<f32>() ,
+ BATCH * VOCAB * std::mem::size_of::<f32>(),
MTLResourceOptions::STORAGE_MODE_SHARED,
)
.expect("Failed to create buffer");
diff --git a/crates/uzu/tests/ssd_prefill_kernel_test.rs b/crates/uzu/tests/ssd_prefill_kernel_test.rs
index 0781e963b..5659e7364 100644
--- a/crates/uzu/tests/ssd_prefill_kernel_test.rs
+++ b/crates/uzu/tests/ssd_prefill_kernel_test.rs
@@ -5,18 +5,19 @@ use std::mem::size_of;
use bytemuck;
use metal::{
MTLBlitCommandEncoder, MTLCommandBuffer, MTLCommandEncoder,
- MTLCommandQueue, MTLDevice, MTLDeviceExt, MTLResourceOptions,
+ MTLCommandQueue, MTLDeviceExt, MTLResourceOptions,
};
-use uzu::{
- backends::metal::{
+use uzu::backends::{
+ common::Context,
+ metal::{
KernelDataType, MTLBuffer, MTLContext, ProtocolObject,
kernel::ssm::{
- SSDPrefillArguments, SSDPrefillKernel, SSDPrefillMode,
+ SSDPrefillArguments, SSDPrefillKernels, SSDPrefillMode,
conv1d_scan::{Conv1dScanArguments, Conv1dScanKernel},
},
},
- config::Activation,
};
+use uzu::config::Activation;
const STORAGE_MODE: MTLResourceOptions =
MTLResourceOptions::STORAGE_MODE_SHARED;
@@ -34,12 +35,6 @@ fn softplus_f32(x: f32) -> f32 {
}
}
-fn create_context() -> Option<MTLContext> {
- let device = <dyn MTLDevice>::system_default()?;
- let command_queue = device.new_command_queue()?;
- MTLContext::new(device, command_queue).ok()
-}
-
fn write_buffer(
buf: &ProtocolObject<dyn MTLBuffer>,
data: &[f32],
@@ -223,7 +218,7 @@ impl SSDPrefillFixture {
fn run_prefill_kernel_mode(
ctx: &MTLContext,
- kernel: &SSDPrefillKernel,
+ kernel: &SSDPrefillKernels,
fixture: &SSDPrefillFixture,
mode: SSDPrefillMode,
) -> (Vec<f32>, Vec<f32>, Option<(Vec<f32>, Vec<f32>, Vec<f32>, Vec<f32>)>) {
@@ -301,7 +296,7 @@ fn run_prefill_kernel_mode(
let encoder = command_buffer
.new_compute_command_encoder()
.expect("Failed to create compute encoder");
- kernel.encode(&encoder, args, mode).unwrap();
+ kernel.encode(&encoder, args, mode);
encoder.end_encoding();
command_buffer.commit();
command_buffer.wait_until_completed();
@@ -432,11 +427,11 @@ fn run_conv_scan_once(
}
fn assert_deterministic_for_mode(mode: SSDPrefillMode) {
- let Some(ctx) = create_context() else {
+ let Some(ctx) = MTLContext::new().ok() else {
eprintln!("Skipping SSD prefill determinism test: no Metal device");
return;
};
- let kernel = SSDPrefillKernel::new(&ctx, KernelDataType::Float32).unwrap();
+ let kernel = SSDPrefillKernels::new(&ctx, KernelDataType::Float32).unwrap();
let fixture = SSDPrefillFixture::new();
let (y_a, state_a, _) =
@@ -449,11 +444,11 @@ fn assert_deterministic_for_mode(mode: SSDPrefillMode) {
}
fn assert_matches_cpu_reference(mode: SSDPrefillMode) {
- let Some(ctx) = create_context() else {
+ let Some(ctx) = MTLContext::new().ok() else {
eprintln!("Skipping SSD prefill reference test: no Metal device");
return;
};
- let kernel = SSDPrefillKernel::new(&ctx, KernelDataType::Float32).unwrap();
+ let kernel = SSDPrefillKernels::new(&ctx, KernelDataType::Float32).unwrap();
let fixture = SSDPrefillFixture::new();
let (y_ref, state_ref) = ssd_prefill_cpu_reference(
@@ -536,7 +531,7 @@ fn ssd_prefill_single_pass_matches_cpu_reference() {
#[test]
fn conv1d_scan_is_deterministic() {
- let Some(ctx) = create_context() else {
+ let Some(ctx) = MTLContext::new().ok() else {
eprintln!("Skipping conv1d scan determinism test: no Metal device");
return;
};
diff --git a/crates/uzu/tests/ssm_update_test.rs b/crates/uzu/tests/ssm_update_test.rs
index 8076b124c..0aa225d4c 100644
--- a/crates/uzu/tests/ssm_update_test.rs
+++ b/crates/uzu/tests/ssm_update_test.rs
@@ -3,12 +3,13 @@
use bytemuck;
use half::bf16;
use metal::{
- MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue, MTLDevice,
+ MTLBuffer, MTLCommandBuffer, MTLCommandEncoder, MTLCommandQueue,
MTLDeviceExt, MTLResourceOptions,
};
-use uzu::backends::metal::{
- KernelDataType, MTLContext,
- kernel::ssm::{SSDUpdateArguments, SSDUpdateKernel},
+use uzu::backends::{
+ common::Context,
+ common::kernel::SSDUpdateKernel,
+ metal::{KernelDataType, MTLContext, kernel::dsl::SSDUpdateMetalKernel},
};
#[allow(dead_code)]
@@ -175,15 +176,9 @@ fn ssd_update_ref_bf16(
(y, next_state)
}
-fn create_context() -> Option<MTLContext> {
- let device = <dyn MTLDevice>::system_default()?;
- let command_queue = device.new_command_queue()?;
- MTLContext::new(device, command_queue).ok()
-}
-
#[test]
fn ssd_update_with_z_bf16() {
- let Some(ctx) = create_context() else {
+ let Some(ctx) = MTLContext::new().ok() else {
eprintln!("Skipping: no Metal device");
return;
};
@@ -219,10 +214,10 @@ fn ssd_update_with_z_bf16() {
let y_exp_f32: Vec<f32> = y_exp.iter().map(|&v| v.to_f32()).collect();
let ns_exp_f32: Vec<f32> = ns_exp.iter().map(|&v| v.to_f32()).collect();
- let x_strides = [h * dh, dh, 1usize];
- let dt_strides = [h, 1usize];
- let cb_strides = [g * n, n, 1usize];
- let state_strides = [h * dh * n, dh * n, n, 1usize];
+ let x_strides = [(h * dh) as u32, dh as u32, 1u32];
+ let dt_strides = [h as u32, 1u32];
+ let cb_strides = [(g * n) as u32, n as u32, 1u32];
+ let state_strides = [(h * dh * n) as u32, (dh * n) as u32, n as u32, 1u32];
let x_buf = ctx
.device
@@ -288,7 +283,9 @@ fn ssd_update_with_z_bf16() {
)
.expect("Failed to create buffer");
- let kernel = SSDUpdateKernel::new(&ctx, KernelDataType::BFloat16).unwrap();
+ let kernel =
+ SSDUpdateMetalKernel::new(&ctx, KernelDataType::BFloat16.into())
+ .unwrap();
let cb_ref = ctx
.command_queue
.command_buffer()
@@ -297,31 +294,27 @@ fn ssd_update_with_z_bf16() {
let enc = cb
.new_compute_command_encoder()
.expect("Failed to create compute encoder");
- kernel
- .encode(
- &enc,
- SSDUpdateArguments {
- x: &x_buf,
- dt: &dt_buf,
- b: &b_buf,
- c: &c_buf,
- d: &d_buf,
- z: &z_buf,
- state: &state_buf,
- y: &y_buf,
- next_state: &ns_buf,
- group_size: (h / g) as i32,
- state_size: n as i32,
- x_strides,
- dt_strides,
- cb_strides,
- state_strides,
- b_size: bsz,
- h_size: h,
- dh_size: dh,
- },
- )
- .unwrap();
+ kernel.encode(
+ &x_buf,
+ &dt_buf,
+ &b_buf,
+ &c_buf,
+ &d_buf,
+ &z_buf,
+ &state_buf,
+ &y_buf,
+ &ns_buf,
+ (h / g) as u32,
+ n as u32,
+ x_strides.as_slice(),
+ dt_strides.as_slice(),
+ cb_strides.as_slice(),
+ state_strides.as_slice(),
+ bsz as u32,
+ h as u32,
+ dh as u32,
+ &enc,
+ );
enc.end_encoding();
cb_ref.commit();
cb_ref.wait_until_completed();
diff --git a/crates/uzu/tests/trie_test.rs b/crates/uzu/tests/trie_test.rs
index 49a9fb099..c5fcb2cf5 100644
--- a/crates/uzu/tests/trie_test.rs
+++ b/crates/uzu/tests/trie_test.rs
@@ -1,7 +1,7 @@
-use std::collections::{HashMap, HashSet};
+use std::collections::HashMap;
use uzu::{
- language_model::rng::DerivableSeed,
+ language_model::rng::PRng,
speculators::empty_speculator::EmptySpeculator,
trie::{TrieCreationConfig, TrieNode},
};
@@ -37,14 +37,13 @@ fn test_trie_manual_sprout() {
#[test]
fn test_trie_from_speculator_sprout() {
- let mut trie_rng = DerivableSeed::new(42);
- let mut ref_rng = DerivableSeed::new(42);
+ let rng = PRng::new(42);
let speculator = EmptySpeculator;
let trie_root = TrieNode::from_speculator(
&[0],
- &mut trie_rng,
+ &rng,
None,
&speculator,
&TrieCreationConfig {
@@ -53,12 +52,12 @@ fn test_trie_from_speculator_sprout() {
10,
);
- verify_sprout(&trie_root, ref_rng.next());
+ verify_sprout(&trie_root, rng.derive(0));
}
fn verify_stick(
trie_root: &TrieNode,
- mut expected_seeds: HashSet<u64>,
+ rng: &PRng,
) {
let flat_trie = trie_root.linearize();
@@ -75,39 +74,37 @@ fn verify_stick(
let position = flat_trie.index(cur_node).unwrap();
assert_eq!(token_ids[position], 0);
assert_eq!(token_positions[position], 0);
- assert!(expected_seeds.remove(&token_seeds[position]));
+ assert_eq!(token_seeds[position], rng.derive(0));
for i in 1..10 {
cur_node = cur_node.get(i as u64).unwrap();
assert_eq!(cur_node.token(), i as u64);
- assert!(expected_seeds.remove(&cur_node.seed()));
+ assert_eq!(cur_node.seed(), rng.derive(i));
let position = flat_trie.index(cur_node).unwrap();
assert_eq!(token_ids[position], cur_node.token());
assert_eq!(token_positions[position], i as usize);
- assert_eq!(token_seeds[position], cur_node.seed());
+ assert_eq!(token_seeds[position], rng.derive(i));
}
-
- assert!(expected_seeds.is_empty());
}
#[test]
fn test_trie_manual_stick() {
- let mut trie_root = TrieNode::new(0, None, 0);
+ let rng = PRng::new(0);
+ let mut trie_root = TrieNode::new(0, None, rng.derive(0));
let mut trie_leaf = &mut trie_root;
- for (token, seed) in std::iter::zip(1..10, 1..10) {
- trie_leaf.add(TrieNode::new(token, None, seed)).unwrap();
- trie_leaf = trie_leaf.get_mut(token).unwrap();
+ for i in 1..10u64 {
+ trie_leaf.add(TrieNode::new(i, None, rng.derive(i))).unwrap();
+ trie_leaf = trie_leaf.get_mut(i).unwrap();
}
- verify_stick(&trie_root, (0..10).collect());
+ verify_stick(&trie_root, &rng);
}
#[test]
fn test_trie_from_speculator_stick() {
- let mut trie_rng = DerivableSeed::new(42);
- let mut ref_rng = DerivableSeed::new(42);
+ let rng = PRng::new(42);
let mut hs = HashMap::new();
for i in 1..10 {
@@ -120,7 +117,7 @@ fn test_trie_from_speculator_stick() {
let trie_root = TrieNode::from_speculator(
&[0],
- &mut trie_rng,
+ &rng,
None,
&speculator,
&TrieCreationConfig {
@@ -129,15 +126,12 @@ fn test_trie_from_speculator_stick() {
10,
);
- verify_stick(
- &trie_root,
- std::iter::from_fn(|| Some(ref_rng.next())).take(10).collect(),
- );
+ verify_stick(&trie_root, &rng);
}
fn verify_bush(
trie_root: &TrieNode,
- mut expected_seeds: HashSet<u64>,
+ rng: &PRng,
) {
let flat_trie = trie_root.linearize();
@@ -152,40 +146,38 @@ fn verify_bush(
let root_position = flat_trie.index(&trie_root).unwrap();
assert_eq!(token_ids[root_position], 0);
assert_eq!(token_positions[root_position], 0);
- assert!(expected_seeds.remove(&token_seeds[root_position]));
+ assert_eq!(token_seeds[root_position], rng.derive(0));
for leaf_token in [1, 2, 3] {
let leaf = trie_root.get(leaf_token).unwrap();
assert_eq!(leaf.token(), leaf_token);
- assert!(expected_seeds.remove(&leaf.seed()));
+ assert_eq!(leaf.seed(), rng.derive(1));
let position = flat_trie.index(&leaf).unwrap();
assert_eq!(token_ids[position], leaf.token());
assert_eq!(token_positions[position], 1);
- assert_eq!(token_seeds[position], leaf.seed());
+ assert_eq!(token_seeds[position], rng.derive(1));
}
-
- assert!(expected_seeds.is_empty());
}
#[test]
fn test_trie_manual_bush() {
- let mut trie_root = TrieNode::new(0, None, 0);
+ let rng = PRng::new(0);
+ let mut trie_root = TrieNode::new(0, None, rng.derive(0));
- assert!(trie_root.add(TrieNode::new(1, None, 1)).is_ok());
- assert!(trie_root.add(TrieNode::new(1, None, 1)).is_err());
+ assert!(trie_root.add(TrieNode::new(1, None, rng.derive(1))).is_ok());
+ assert!(trie_root.add(TrieNode::new(1, None, rng.derive(1))).is_err());
assert!(trie_root.add(TrieNode::new(1, None, 10)).is_err());
- assert!(trie_root.add(TrieNode::new(2, None, 2)).is_ok());
- assert!(trie_root.add(TrieNode::new(3, None, 3)).is_ok());
+ assert!(trie_root.add(TrieNode::new(2, None, rng.derive(1))).is_ok());
+ assert!(trie_root.add(TrieNode::new(3, None, rng.derive(1))).is_ok());
- verify_bush(&trie_root, (0..4).into_iter().collect());
+ verify_bush(&trie_root, &rng);
}
#[test]
fn test_trie_from_speculator_bush() {
- let mut trie_rng = DerivableSeed::new(42);
- let mut ref_rng = DerivableSeed::new(42);
+ let rng = PRng::new(0);
let mut hs = HashMap::new();
hs.insert(
@@ -198,7 +190,7 @@ fn test_trie_from_speculator_bush() {
let trie_root = TrieNode::from_speculator(
&[0],
- &mut trie_rng,
+ &rng,
None,
&speculator,
&TrieCreationConfig {
@@ -207,13 +199,13 @@ fn test_trie_from_speculator_bush() {
10,
);
- verify_bush(
- &trie_root,
- std::iter::from_fn(|| Some(ref_rng.next())).take(4).collect(),
- );
+ verify_bush(&trie_root, &rng);
}
-fn verify_tree(trie_root: &TrieNode) {
+fn verify_tree(
+ trie_root: &TrieNode,
+ rng: &PRng,
+) {
let flat_trie = trie_root.linearize();
assert_eq!(flat_trie.len(), 7);
@@ -227,44 +219,43 @@ fn verify_tree(trie_root: &TrieNode) {
let root_position = flat_trie.index(&trie_root).unwrap();
assert_eq!(token_ids[root_position], 0);
assert_eq!(token_positions[root_position], 0);
- assert_eq!(token_seeds[root_position], 0);
+ assert_eq!(token_seeds[root_position], rng.derive(0));
- for (mid_token, mid_seed) in [(1, 1), (2, 1), (3, 2)] {
- let leaf = trie_root.get(mid_token).unwrap();
- let position = flat_trie.index(&leaf).unwrap();
+ for mid_token in [1, 2, 3] {
+ let node = trie_root.get(mid_token).unwrap();
+ let position = flat_trie.index(&node).unwrap();
assert_eq!(token_ids[position], mid_token);
assert_eq!(token_positions[position], 1);
- assert_eq!(token_seeds[position], mid_seed);
+ assert_eq!(token_seeds[position], rng.derive(1));
}
- for (mid_token, leaf_token, leaf_seed) in
- [(2, 10, 15), (3, 20, 25), (3, 21, 26)]
- {
+ for (mid_token, leaf_token) in [(2, 10), (3, 20), (3, 21)] {
let leaf = trie_root.get(mid_token).unwrap().get(leaf_token).unwrap();
let leaf_position = flat_trie.index(leaf).unwrap();
assert_eq!(token_ids[leaf_position], leaf_token);
assert_eq!(token_positions[leaf_position], 2);
- assert_eq!(token_seeds[leaf_position], leaf_seed);
+ assert_eq!(token_seeds[leaf_position], rng.derive(2));
}
}
#[test]
fn test_trie_manual_tree() {
- let mut trie_root = TrieNode::new(0, None, 0);
+ let rng = PRng::new(0);
+ let mut trie_root = TrieNode::new(0, None, rng.derive(0));
- assert!(trie_root.add(TrieNode::new(1, None, 1)).is_ok());
- assert!(trie_root.add(TrieNode::new(1, None, 1)).is_err());
+ assert!(trie_root.add(TrieNode::new(1, None, rng.derive(1))).is_ok());
+ assert!(trie_root.add(TrieNode::new(1, None, rng.derive(1))).is_err());
assert!(trie_root.add(TrieNode::new(1, None, 10)).is_err());
- assert!(trie_root.add(TrieNode::new(2, None, 1)).is_ok());
- assert!(trie_root.add(TrieNode::new(3, None, 2)).is_ok());
+ assert!(trie_root.add(TrieNode::new(2, None, rng.derive(1))).is_ok());
+ assert!(trie_root.add(TrieNode::new(3, None, rng.derive(1))).is_ok());
let mid_b = trie_root.get_mut(2).unwrap();
- assert!(mid_b.add(TrieNode::new(10, None, 15)).is_ok());
+ assert!(mid_b.add(TrieNode::new(10, None, rng.derive(2))).is_ok());
let mid_c = trie_root.get_mut(3).unwrap();
- assert!(mid_c.add(TrieNode::new(20, None, 25)).is_ok());
- assert!(mid_c.add(TrieNode::new(21, None, 26)).is_ok());
+ assert!(mid_c.add(TrieNode::new(20, None, rng.derive(2))).is_ok());
+ assert!(mid_c.add(TrieNode::new(21, None, rng.derive(2))).is_ok());
- verify_tree(&trie_root)
+ verify_tree(&trie_root, &rng)
}