The 80B Qwen3-Coder-Next on a 16 GB Mac, about 13 tok/s, fully offline. A mixture-of-experts inference engine for Apple silicon.
Rust
0.5.0: gpt-oss-120b - the family, its squeeze, --stand-ins and --flash-reads
1picassoai64e4ff67f75d
diff --git a/Cargo.lock b/Cargo.lockindex 2389b56..1782c09 100644--- a/Cargo.lock+++ b/Cargo.lock@@ -158,7 +158,6 @@ checksum = "1fd0f2584146f6f2ef48085050886acf353beff7305ebd1ae69500e27c67f64b"[[package]]name = "candle-core"version = "0.11.0"-source = "git+https://github.com/huggingface/candle?rev=5ba5d5b468b5b1df40e82dd3d556987bedeea041#5ba5d5b468b5b1df40e82dd3d556987bedeea041"dependencies = ["byteorder","candle-metal-kernels",@@ -290,7 +289,7 @@ checksum = "1c133bc6a41be0d194c306b5506d15e6feeea7b1d6604bd3f8310dfb2ca96486"[[package]]name = "coachwhip"-version = "0.4.1"+version = "0.5.0"dependencies = ["anyhow","candle-core",@@ -303,7 +302,7 @@ dependencies = [[[package]]name = "coachwhip-engine"-version = "0.4.1"+version = "0.5.0"dependencies = ["candle-core","candle-metal-kernels",@@ -317,7 +316,7 @@ dependencies = [[[package]]name = "coachwhip-prepare"-version = "0.4.1"+version = "0.5.0"dependencies = ["anyhow","byteorder",@@ -327,7 +326,7 @@ dependencies = [[[package]]name = "coachwhip-server"-version = "0.4.1"+version = "0.5.0"dependencies = ["anyhow","candle-core",diff --git a/Cargo.toml b/Cargo.tomlindex b1f7916..b37b471 100644--- a/Cargo.toml+++ b/Cargo.toml@@ -3,7 +3,7 @@ members = ["crates/engine", "crates/server", "crates/cli", "crates/prepare"]resolver = "2"[workspace.package]-version = "0.4.1"+version = "0.5.0"edition = "2021"license = "Apache-2.0"publish = false@@ -28,3 +28,4 @@ debug = false[patch."https://github.com/huggingface/candle"]candle-metal-kernels = { path = "vendor/candle-metal-kernels" }+candle-core = { path = "vendor/candle-core" }diff --git a/README.md b/README.mdindex c458912..87ab156 100644--- a/README.md+++ b/README.md@@ -9,9 +9,18 @@**Run mixture-of-experts models that are bigger than your memory.**-**A 122B model on a 16 GB Mac.** Qwen3.5-122B-A10B, squeezed once on your own Mac from the publisher's file; the 80B and the 35B run too. Coachwhip never needs the whole model in memory, because a mixture-of-experts model never needs the whole model for one word.+**OpenAI's gpt-oss-120b on a 16 GB Mac, at about 10 tok/s.** Squeezed once on your own Mac from the official file. Qwen3.5-122B-A10B, the 80B and the 35B run too. Coachwhip never needs the whole model in memory, because a mixture-of-experts model never needs the whole model for one word.-Fast mode on a 16 GB Mac mini M4, bank 40:+gpt-oss-120b on a 16 GB Mac mini M4, writing a TypeScript task (about 4,000 words of thinking and code, temperature 0), scored on 20 hidden checks:++| Mode | Speed | Score |+|---|---|---|+| Fast mode, `--stand-ins` | about 10 tok/s | 4.0 of 5 |+| The model's own 4 experts | about 4.2 tok/s | 4.8 of 5 |++How to set it up is in [gpt-oss-120b](#gpt-oss-120b).++Qwen3.5-122B-A10B, fast mode on the same Mac, bank 40:| Tokens | Speed | Coachwhip uses | Left for macOS and your apps ||---|---|---|---|@@ -20,6 +29,8 @@ Fast mode on a 16 GB Mac mini M4, bank 40:One Rust binary. Everything on the GPU. Nothing leaves your machine.+**New in 0.5.0** ([release notes](docs/releases/0.5.0.md)): **gpt-oss-120b**, OpenAI's open model, thinking included, at about 10 tok/s in fast mode; set up in [gpt-oss-120b](#gpt-oss-120b).+**New in 0.4.1** ([release notes](docs/releases/0.4.1.md)): **install in under a minute.** The installer downloads a ready-built Coachwhip; nothing to compile.**New in 0.4.0** ([release notes](docs/releases/0.4.0.md)): **Qwen3.5-122B-A10B on a 16 GB Mac**, squeezed once on your own Mac from the publisher's file; how to set it up is in [The 122B](#the-122b). Coachwhip now keeps itself inside your Mac's memory: it picks the bank that fits and warns you when macOS is short. Every model's numbers, how they were measured and a quality check per file: [docs/MODELS.md](docs/MODELS.md).@@ -71,6 +82,7 @@ Coachwhip is an engine for MoE models, in GGUF format.| Model | Architecture | Status ||---|---|---|+| [gpt-oss-120b](https://huggingface.co/openai/gpt-oss-120b) | gpt-oss | Tested, after `prepare` (see gpt-oss-120b). Thinks before it answers. || [Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) | qwen35moe | Tested, after `prepare` (see The 122B). || [Qwen3-Coder-30B-A3B-Instruct](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct) | qwen3moe | Tested. || [Qwen3-Coder-Next](https://huggingface.co/Qwen/Qwen3-Coder-Next) (80B) | qwen3next | Tested. See below for the file to use. |@@ -83,16 +95,16 @@ Speed and a quality check for each model file, measured on a 16 GB Mac: [docs/MOTry other MoE models and tell us what happens. If a model's architecture is not supported yet, Coachwhip says so, and an issue is the fastest way to move it up the list.-**Which GGUF file.** Coachwhip reads the standard GGUF quantisations (Q3_K, Q4_K, Q5_K, Q6_K, Q8_0, F16). Some newer conversions store a few tensors in MXFP4 or fuse the attention weights into one tensor; Coachwhip does not read those yet. For Qwen3-Coder-Next, [MaziyarPanahi's Q3_K_M and Q4_K_M](https://huggingface.co/MaziyarPanahi/Qwen3-Coder-Next-GGUF) load as is.+**Which GGUF file.** Coachwhip reads the standard GGUF quantisations (Q3_K, Q4_K, Q5_K, Q6_K, Q8_0, F16). gpt-oss ships in MXFP4, which `prepare` reads and squeezes. Some newer Qwen conversions store a few tensors in MXFP4 or fuse the attention weights into one tensor; Coachwhip does not read those yet. For Qwen3-Coder-Next, [MaziyarPanahi's Q3_K_M and Q4_K_M](https://huggingface.co/MaziyarPanahi/Qwen3-Coder-Next-GGUF) load as is.## What you need-| | 122B | 80B | 30B |-|---|---|---|---|-| Mac | Apple silicon, M4 or later recommended; M1 to M3 run it slower. Tested on a Mac mini M4. | the same | the same |-| Memory | 16 GB | 16 GB; 24 GB has room to spare | 16 GB |-| Disk | 105 GB free while squeezing, 46 GB once you delete the download; 15 MB for Coachwhip | 39 GB (Q3_K_M) or 49 GB (Q4_K_M) for the model, 15 MB for Coachwhip | 19 GB for the model, 15 MB for Coachwhip |-| macOS | Tested on macOS 26 | the same | the same |+| | gpt-oss-120b | 122B | 80B | 30B |+|---|---|---|---|---|+| Mac | Apple silicon, M4 or later recommended; M1 to M3 run it slower. Tested on a Mac mini M4. | the same | the same | the same |+| Memory | 16 GB | 16 GB | 16 GB; 24 GB has room to spare | 16 GB |+| Disk | 106 GB free while squeezing, 43 GB once you delete the download; 15 MB for Coachwhip | 105 GB free while squeezing, 46 GB once you delete the download; 15 MB for Coachwhip | 39 GB (Q3_K_M) or 49 GB (Q4_K_M) for the model, 15 MB for Coachwhip | 19 GB for the model, 15 MB for Coachwhip |+| macOS | Tested on macOS 26 | the same | the same | the same |## Install@@ -110,6 +122,8 @@ To build from source instead, run it with `COACHWHIP_FROM_SOURCE=1` in front ofCoachwhip is the engine; the model is yours. Any GGUF of a supported model works (see Models).+**gpt-oss-120b:** download the official MXFP4 file and squeeze it once on your Mac, as in [gpt-oss-120b](#gpt-oss-120b).+**Qwen3.5-122B-A10B:** download the publisher's file and squeeze it once on your Mac, as in [The 122B](#the-122b).**Qwen3-Coder-Next, the 80B coder, in its Q3_K_M file:** download the [Q3_K_M GGUF](https://huggingface.co/MaziyarPanahi/Qwen3-Coder-Next-GGUF) (38 GB) and its [tokenizer.json](https://huggingface.co/Qwen/Qwen3-Coder-Next), then start it with a bank of 70:@@ -122,11 +136,45 @@ The Q3_K_M file writes faster than Q4_K_M on a 16 GB Mac, and the two scored the| Model | GGUF file | Tokenizer | Bank on 16 GB ||---|---|---|---|+| gpt-oss-120b, after `prepare` | [ggml-org MXFP4](https://huggingface.co/ggml-org/gpt-oss-120b-GGUF) | [tokenizer.json](https://huggingface.co/openai/gpt-oss-120b) | 25 || Qwen3.5-122B-A10B, after `prepare` | [mradermacher Q3_K_M](https://huggingface.co/mradermacher/Qwen3.5-122B-A10B-GGUF) | [tokenizer.json](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) | 40 || Qwen3-Coder-Next (80B) | [MaziyarPanahi Q3_K_M](https://huggingface.co/MaziyarPanahi/Qwen3-Coder-Next-GGUF) | [tokenizer.json](https://huggingface.co/Qwen/Qwen3-Coder-Next) | 70 || Qwen3-Coder-Next (80B), the 4-bit file | [MaziyarPanahi Q4_K_M](https://huggingface.co/MaziyarPanahi/Qwen3-Coder-Next-GGUF) | [tokenizer.json](https://huggingface.co/Qwen/Qwen3-Coder-Next) | 56 || Qwen3-Coder-30B-A3B, a smaller download | [unsloth Q4_K_M](https://huggingface.co/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF) | [tokenizer.json](https://huggingface.co/Qwen/Qwen3-Coder-30B-A3B-Instruct) | 44 |+## gpt-oss-120b++OpenAI's gpt-oss-120b: 117 billion parameters, 4 of its 128 experts for each word in each of its 36 layers. It thinks before it answers, and the chat page shows the thinking.++1. Download the official [MXFP4 GGUF](https://huggingface.co/ggml-org/gpt-oss-120b-GGUF) (63.4 GB) and the model's [tokenizer.json](https://huggingface.co/openai/gpt-oss-120b).+2. Squeeze it. You need about **106 GB free** while it runs: the 63 GB download and the 43 GB squeezed file sit side by side.++```sh+~/coachwhip/target/release/coachwhip-prepare ~/Downloads/gpt-oss-120b-MXFP4.gguf ~/Downloads/gpt-oss-120b.Q2A.gguf+```++ The squeeze takes every expert matrix to Q2_K; everything else in the file is copied as it is. Nothing is downloaded from us.++3. Delete the download. Coachwhip only uses the squeezed file:++```sh+rm ~/Downloads/gpt-oss-120b-MXFP4.gguf+```++4. Start it in one of two modes. Coachwhip picks the bank that fits your Mac (25 on 16 GB):++```sh+# the model's own 4 experts per word, every one read for real+~/coachwhip/coachwhip.sh ~/Downloads/gpt-oss-120b.Q2A.gguf ~/Downloads/tokenizer.json 25 8090 --parallel --flash-reads++# fast: stand-ins for the light picks that are not in the bank+~/coachwhip/coachwhip.sh ~/Downloads/gpt-oss-120b.Q2A.gguf ~/Downloads/tokenizer.json 25 8090 --parallel --flash-reads --stand-ins+```++**Fast mode changes the answers.** A word's first expert is always read for real, and so is its second whenever the router gives it at least 0.3 of the weight. A lighter pick that is not already in the bank is replaced by the best expert that is, by the router's own scores, instead of being read from the SSD. Measured on one coding task, it wrote about 2.4 times as fast and scored 4.0 of 5 against 4.8. It is a choice, never the default.++Every setup we measured, what makes it fast and how the scores were taken: [docs/GPT-OSS.md](docs/GPT-OSS.md).+## The 122BQwen3.5-122B-A10B is the biggest model Coachwhip runs on 16 GB, and it needs one extra step: its experts are too big to stream fast as they come, so Coachwhip squeezes the file once, on your Mac.@@ -170,7 +218,7 @@ Its speed and memory are in the table at the top.The last number is the bank (44 if you leave it out). The chat opens in your browser. Type, press Send, and the answer streams as it is written. Follow-up questions read only your new words; press **New chat** to start again. Stop Coachwhip with Ctrl+C.-**Thinking.** A model with a thinking mode (Qwen3.6) shows a **let it think first** box on the page. Off, the model answers straight away; on, it reasons first and you watch it. Either way Coachwhip only puts the model's own template text at the start of its turn, nothing else changes, and each word comes at the same speed; thinking makes the reply longer, not slower per word. In the terminal it is `--think`; through the API it is always off, so coding tools get the answer. The 80B coder has no thinking mode, and the box is not shown for it.+**Thinking.** gpt-oss always thinks first; the page shows its thinking in a box above the answer, and the API returns only the answer. A model with a thinking mode (Qwen3.6) shows a **let it think first** box on the page. Off, the model answers straight away; on, it reasons first and you watch it. Either way Coachwhip only puts the model's own template text at the start of its turn, nothing else changes, and each word comes at the same speed; thinking makes the reply longer, not slower per word. In the terminal it is `--think`; through the API it is always off, so coding tools get the answer. The 80B coder has no thinking mode, and the box is not shown for it.**From your coding tools:** while Coachwhip runs, it also speaks the OpenAI chat API, so any tool that takes an OpenAI base URL can use the model.@@ -215,19 +263,22 @@ cd ~/coachwhip| `--prefetch 10` | Experts guessed and read in ahead, per layer; 0 turns the guess off || `--ahead 4` | How many layers ahead the guess looks || `--think` | Let a thinking model reason before it answers (`--prompt` runs) |+| `--parallel` | The GPU waits on a shared event for its experts; faster on big models, same answers |+| `--flash-reads` | Read experts straight from the SSD, past the page cache; same answers |+| `--stand-ins` | gpt-oss fast mode: stand-ins for the light picks not in the bank; the answers change || `--profile` | Print where the time went after each answer || `--help` | Everything else |## Roadmap-- More MoE families: Mixtral, Qwen2-MoE, OLMoE, DeepSeek, gpt-oss, GLM.-- MXFP4 and fused-attention GGUF files.+- More MoE families: Mixtral, Qwen2-MoE, OLMoE, DeepSeek, GLM, Qwen3.8.+- Fused-attention GGUF files.- An adapter that reads your own repository, on your own machine.- More platforms.## Credits-Built on [Candle](https://github.com/huggingface/candle). `crates/engine/src/model.rs` is derived from Candle's Qwen3 model, and the fused expert kernel is ggml's `mul_mv_id`, which ships inside Candle. The block-at-a-time linear attention follows [Gated Delta Networks](https://arxiv.org/abs/2412.06464) (Yang, Kautz and Hatamizadeh, ICLR 2025).+Built on [Candle](https://github.com/huggingface/candle); `vendor/candle-core` is its core crate with an MXFP4 kernel for Metal. gpt-oss-120b is [OpenAI's](https://huggingface.co/openai/gpt-oss-120b), under Apache-2.0. `crates/engine/src/model.rs` is derived from Candle's Qwen3 model, and the fused expert kernel is ggml's `mul_mv_id`, which ships inside Candle. The block-at-a-time linear attention follows [Gated Delta Networks](https://arxiv.org/abs/2412.06464) (Yang, Kautz and Hatamizadeh, ICLR 2025).## Licencediff --git a/crates/cli/Cargo.toml b/crates/cli/Cargo.tomlindex 03eed86..35fecf0 100644--- a/crates/cli/Cargo.toml+++ b/crates/cli/Cargo.toml@@ -1,6 +1,6 @@[package]name = "coachwhip"-description = "Run mixture-of-experts models bigger than your memory on a Mac: an 80B model on 16 GB. The experts stream from the SSD into a small GPU bank."+description = "Run mixture-of-experts models bigger than your memory on a Mac: a 122B model on 16 GB. The experts stream from the SSD into a small GPU bank."version.workspace = trueedition.workspace = truelicense.workspace = truediff --git a/crates/cli/src/main.rs b/crates/cli/src/main.rsindex cecd02f..d60b1d2 100644--- a/crates/cli/src/main.rs+++ b/crates/cli/src/main.rs@@ -1,4 +1,4 @@-//! Coachwhip: run mixture-of-experts models that are bigger than the memory, on a Mac. An 80B+//! Coachwhip: run mixture-of-experts models that are bigger than the memory, on a Mac. A 122B//! model runs on 16 GB: the experts stream from the SSD into a small bank on the GPU, and the//! ones the next layer will probably need are fetched while the current layer computes.@@ -14,7 +14,7 @@ use std::path::PathBuf;use tokenizers::Tokenizer;#[derive(Parser, Debug)]-#[command(version, about = "Run mixture-of-experts models bigger than your memory on a Mac: an 80B model on 16 GB")]+#[command(version, about = "Run mixture-of-experts models bigger than your memory on a Mac: a 122B model on 16 GB")]struct Args {/// The model, as a GGUF file: Qwen3 MoE or Qwen3-Next (Q4_K_M tested).#[arg(long)]@@ -28,6 +28,10 @@ struct Args {#[arg(long, conflicts_with = "prompt")]chat: Option<u16>,+ /// Lab: read the prompt as raw text, no chat template, and decode greedily.+ #[arg(long, hide = true)]+ raw: bool,+/// Answer one prompt and exit.#[arg(long)]prompt: Option<String>,@@ -94,6 +98,18 @@ struct Args {/// the next layers' experts from the state it predicts. Fewer reads; the answers do not change.#[arg(long)]register: bool,++ /// Stand-ins (gpt-oss): a light pick that is not in the bank is replaced by the best expert+ /// that is, by the router's own scores. The first pick is always read for real, and the second+ /// whenever the router gives it at least 0.3 of the weight. Fewer reads; the answers change,+ /// so check them for your use.+ #[arg(long)]+ stand_ins: bool,++ /// Flash reads: experts are read straight from the SSD, past the page cache, which has nothing+ /// to give back on a 16 GB Mac. The answers do not change.+ #[arg(long)]+ flash_reads: bool,}fn main() -> Result<()> {@@ -113,25 +129,37 @@ fn main() -> Result<()> {if args.register {std::env::set_var("COACHWHIP_REGISTER", "1");}+ if args.stand_ins {+ std::env::set_var("COACHWHIP_STANDIN", "1");+ if std::env::var("COACHWHIP_STANDIN_W").is_err() {+ std::env::set_var("COACHWHIP_STANDIN_W", "0.3");+ }+ }+ if args.flash_reads {+ std::env::set_var("COACHWHIP_NOCACHE", "1");+ }let device = Device::new_metal(0)?;// A hot path belongs to one model: its routes live in a folder named after the model file.let model_name = args.model.file_stem().map(|s| s.to_string_lossy().into_owned()).unwrap_or_else(|| "model".into());let routes = args.routes.join(&model_name);let learned = args.learned.join(&model_name);std::fs::create_dir_all(&learned)?;- // The 122B (48-layer qwen35moe) guesses best at 6 ahead by 2; the others at 10 by 4.- let big = {+ // The 122B (48-layer qwen35moe) guesses best at 6 ahead by 2, gpt-oss at 2 by 2 (its 13 MB+ // experts make a wrong guess dear); the others at 10 by 4.+ let (big, oss) = {let mut f = std::fs::File::open(&args.model)?;let ct = candle::quantized::gguf_file::Content::read(&mut f)?;let arch = ct.metadata.get("general.architecture").and_then(|v| v.to_string().ok().cloned()).unwrap_or_default();let layers = ct.metadata.get(&format!("{arch}.block_count")).and_then(|v| v.to_u32().ok()).unwrap_or(0);- arch == "qwen35moe" && layers >= 48+ (arch == "qwen35moe" && layers >= 48, arch == "gpt-oss")};+ // gpt-oss guesses about half the experts it uses per word (13 MB each makes a wrong guess dear).+ let oss_guess = args.experts.map_or(2, |k| (k / 2).max(1));let (prefetch, ahead) = match (args.prefetch, args.ahead) {(Some(p), Some(a)) => (p, a),- (Some(p), None) => (p, if big { 2 } else { 4 }),- (None, Some(a)) => (if big { 6 } else { 10 }, a),- (None, None) => if big { (6, 2) } else { (10, 4) },+ (Some(p), None) => (p, if big || oss { 2 } else { 4 }),+ (None, Some(a)) => (if oss { oss_guess } else if big { 6 } else { 10 }, a),+ (None, None) => if oss { (oss_guess, 2) } else if big { (6, 2) } else { (10, 4) },};let settings = experts::Settings {bank: args.bank,@@ -163,6 +191,45 @@ fn main() -> Result<()> {let Some(prompt) = args.prompt else {anyhow::bail!("give --chat <port> or --prompt <text>")};+ if args.raw {+ let ids = tokenizer.encode(prompt.as_str(), false).map_err(anyhow::Error::msg)?.get_ids().to_vec();+ eprintln!("coachwhip: raw prompt, {} tokens", ids.len());+ model.clear_kv_cache();+ // Read the prompt in 512-token pieces, as the chat does; the last piece goes through the loop.+ let mut offset = 0usize;+ let last = ids.len() - (ids.len() - 1) % 512 - 1;+ for piece in ids[..last].chunks(512) {+ model.forward(&candle::Tensor::new(piece, &device)?.unsqueeze(0)?, offset)?;+ offset += piece.len();+ }+ let mut input = candle::Tensor::new(&ids[last..], &device)?.unsqueeze(0)?;+ let mut outs = Vec::new();+ let t0 = std::time::Instant::now();+ for step in 0..args.max_tokens {+ let logits: Vec<f32> = model.forward(&input, offset)?.flatten_all()?.to_dtype(candle::DType::F32)?.to_vec1()?;+ offset += input.dim(1)?;+ let m = logits.iter().cloned().fold(f32::NEG_INFINITY, f32::max);+ let lse = m + logits.iter().map(|v| (v - m).exp()).sum::<f32>().ln();+ let mut idx: Vec<usize> = (0..logits.len()).collect();+ idx.sort_by(|&a, &b| logits[b].partial_cmp(&logits[a]).unwrap());+ if step == 0 {+ let top: Vec<(String, f32)> = idx[..3].iter().map(|&i| (tokenizer.decode(&[i as u32], false).unwrap_or_default(), logits[i] - lse)).collect();+ eprintln!("coachwhip: first token top-3 {top:?} after {:.1} s", t0.elapsed().as_secs_f64());+ }+ if std::env::var("COACHWHIP_POOLLOG").is_ok() && step % 25 == 0 {+ if let candle::Device::Metal(m) = &device {+ let (pooled, in_use, count) = m.pool_stats();+ eprintln!("coachwhip: pool at word {step}: {:.2} GB pooled in {count} buffers, {:.2} GB in use", pooled as f64 / 1e9, in_use as f64 / 1e9);+ }+ }+ let next = idx[0] as u32;+ outs.push(next);+ input = candle::Tensor::new(&[next], &device)?.unsqueeze(0)?;+ }+ println!("{:?}", tokenizer.decode(&outs, false).map_err(anyhow::Error::msg)?);+ eprintln!("coachwhip: {} tokens in {:.1} s", outs.len(), t0.elapsed().as_secs_f64());+ return Ok(());+ }let mut out = std::io::stdout();let mut emit = |t: &str| -> Result<()> {out.write_all(t.as_bytes())?;diff --git a/crates/engine/src/experts.rs b/crates/engine/src/experts.rsindex f372619..e3cb3be 100644--- a/crates/engine/src/experts.rs+++ b/crates/engine/src/experts.rs@@ -141,10 +141,13 @@ fn wrangler_fit(ct: &gguf_file::Content, metal: &MetalDevice, layers: usize, exp.map(|(_, i)| (i.shape.elem_count() / i.ggml_dtype.block_size() * i.ggml_dtype.type_size()) as f64).sum();// Keys and values for a full chat, f32, on the layers that keep them.- let interval = md_u(&format!("{arch}.full_attention_interval")).unwrap_or(1.0).max(1.0);+ // gpt-oss alternates full and 128-token sliding layers; only the full ones keep a whole chat.+ let interval = if arch == "gpt-oss" { 2.0 } else { md_u(&format!("{arch}.full_attention_interval")).unwrap_or(1.0).max(1.0) };let n_kv = md_u(&format!("{arch}.attention.head_count_kv")).unwrap_or(1.0);let hd = md_u(&format!("{arch}.attention.key_length")).unwrap_or(128.0);- let kv = (layers as f64 / interval).floor() * 2.0 * n_kv * hd * 4.0 * WRANGLER_CONTEXT;+ // gpt-oss keeps its cache in f16 (2 bytes a value); the others in f32.+ let bytes = if arch == "gpt-oss" { 2.0 } else { 4.0 };+ let kv = (layers as f64 / interval).floor() * 2.0 * n_kv * hd * bytes * WRANGLER_CONTEXT;let room = limit - WRANGLER_MARGIN - WRANGLER_WORKING - dense - kv;let fit = (room / (layers as f64 * expert_bytes)).floor().max(8.0) as usize;eprintln!(@@ -392,6 +395,8 @@ impl Default for Register {#[derive(Default)]struct Stats {+ /// Lab: weaker picks swapped for a resident expert (`COACHWHIP_STANDIN`).+ stand_ins: u64,hits: u64,loads: u64,load_seconds: f64,@@ -1172,7 +1177,7 @@ impl ExpertStore {let gate = x.apply_op2_no_bwd(&slots, &g)?;let up = x.apply_op2_no_bwd(&slots, &u)?;let inter = gate.dim(2)?;- let hidden = (candle_nn::ops::silu(&gate)? * &up)?.reshape((1, inter))?;+ let hidden = (candle_nn::ops::silu(&gate)? * &up)?.reshape((1, ()))?;let down = hidden.apply_op2_no_bwd(&slots, &d)?; // (1, 1, h)down.flatten_all()?.to_dtype(DType::F32)}@@ -2004,6 +2009,9 @@ impl ExpertStore {s.prefetch_copies,s.prefetch_hits);+ if s.stand_ins > 0 {+ eprintln!("coachwhip: stand-ins: {} weaker picks swapped for a resident expert", s.stand_ins);+ }eprintln!("coachwhip: waited {:.2} s on guesses still landing; dropped {} stale guessed reads; {} layers read fresh because the bank was crowded{}",s.prefetch_wait_seconds, s.stale_dropped, s.crowded,@@ -2126,7 +2134,7 @@ impl CustomOp2 for MoeMatvec {device.device(),&encoder,device.kernels(),- self.dtype.into(),+ candle_metal_kernels::GgmlDType::try_from(self.dtype)?,(self.n, self.k, self.slots, self.bytes),self.bank.buffer(),x.buffer(),@@ -2143,6 +2151,41 @@ impl CustomOp2 for MoeMatvec {}}+/// Lab (`COACHWHIP_GUESSLOG=file`): one line per word and layer, for racing guessers offline:+/// `layer ahead | picks as expert:weight | the router-ahead's top 32 as expert:probability`.+fn guess_log(path: &str, layer: usize, ahead: usize, picks: &[u32], top_w: &Tensor, scores: &[f32]) -> Result<()> {+ use std::io::Write as _;+ let w: Vec<f32> = top_w.flatten_all()?.to_dtype(DType::F32)?.to_vec1()?;+ let m = scores.iter().cloned().fold(f32::NEG_INFINITY, f32::max);+ let z: f32 = scores.iter().map(|s| (s - m).exp()).sum();+ let mut ranked: Vec<(usize, f32)> = scores.iter().enumerate().map(|(e, s)| (e, (s - m).exp() / z)).collect();+ ranked.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap_or(std::cmp::Ordering::Equal));+ let picks: Vec<String> = picks.iter().zip(w.iter()).map(|(e, w)| format!("{e}:{w:.4}")).collect();+ let guess: Vec<String> = ranked.iter().take(32).map(|(e, p)| format!("{e}:{p:.5}")).collect();+ let mut f = std::fs::OpenOptions::new().create(true).append(true).open(path)?;+ writeln!(f, "{layer} {ahead} | {} | {}", picks.join(" "), guess.join(" "))?;+ Ok(())+}++/// Lab (`COACHWHIP_STANDIN_MAX`, default 0.3): a weaker pick is swapped for a stand-in only when+/// the router gave it less than this share of the word.+/// Lab (`COACHWHIP_STANDIN_KEEP`): how many of the top picks are always read for real; the default+/// is half the picks (at least 1).+fn stand_in_keep(k: usize) -> usize {+ std::env::var("COACHWHIP_STANDIN_KEEP").ok().and_then(|v| v.parse().ok()).unwrap_or(k / 2).clamp(1, k.max(1))+}++/// Lab (`COACHWHIP_STANDIN_W=<w>`): with more than two picks, the second stays real when its weight is+/// at least w, else only the first does. Off: `COACHWHIP_STANDIN_KEEP` rules.+fn stand_in_w() -> Option<f32> {+ static V: std::sync::OnceLock<Option<f32>> = std::sync::OnceLock::new();+ *V.get_or_init(|| std::env::var("COACHWHIP_STANDIN_W").ok().and_then(|v| v.parse().ok()))+}++fn stand_in_limit() -> f32 {+ std::env::var("COACHWHIP_STANDIN_MAX").ok().and_then(|v| v.parse().ok()).unwrap_or(0.3)+}+/// Router scores with every expert not `allowed` pushed far below the rest, so `route` picks/// only among the allowed ones (S6 draft: the experts already in the bank).pub fn restrict(logits: &Tensor, allowed: impl Fn(u32) -> bool) -> Result<Tensor> {@@ -2164,6 +2207,57 @@ pub fn route(logits: &Tensor, top_k: usize, norm: bool) -> Result<(Tensor, TensoOk((top_ids, top_w))}+/// What a family does between an expert's matrices: the activation, and any per-expert biases.+/// The core runs the matrices; the family owns this.+pub enum ExpertAct {+ /// Qwen: down(silu(gate x) * up x).+ Silu,+ /// gpt-oss: gate and up get their biases, gate is capped at `limit`, up is clamped to+ /// ±`limit`, hidden = (up + 1) * gate * sigmoid(alpha * gate), then down gets its bias.+ /// Biases are (experts, width), looked up by the picked expert ids.+ /// `pad`: the width down's matrix reads; a squeezed file pads rows to whole 256-weight+ /// blocks, so the hidden state gets zeros up to it (0 = no padding).+ ClampedSwiglu { alpha: f64, limit: f32, gate_b: Tensor, up_b: Tensor, down_b: Tensor, pad: usize },+}++impl ExpertAct {+ /// gate, up: (n, k, inter); ids: the expert of each (n, k) position, asked for only when needed.+ pub fn hidden(&self, gate: &Tensor, up: &Tensor, ids: impl FnOnce() -> Result<Tensor>) -> Result<Tensor> {+ match self {+ Self::Silu => candle_nn::ops::silu(gate)? * up,+ Self::ClampedSwiglu { alpha, limit, gate_b, up_b, pad, .. } => {+ let ids = ids()?;+ let gate = (gate + Self::rows(gate_b, &ids, gate)?)?.clamp(f32::NEG_INFINITY, *limit)?;+ let up = (up + Self::rows(up_b, &ids, up)?)?.clamp(-*limit, *limit)?;+ let glu = (&gate * candle_nn::ops::sigmoid(&(&gate * *alpha)?)?)?;+ let hidden = ((up + 1.0)? * glu)?;+ let width = hidden.dim(candle::D::Minus1)?;+ if *pad > width {+ hidden.pad_with_zeros(candle::D::Minus1, 0, pad - width)+ } else {+ Ok(hidden)+ }+ }+ }+ }++ /// down: (n, k, h), before the routing weights are applied.+ pub fn finish(&self, down: Tensor, ids: impl FnOnce() -> Result<Tensor>) -> Result<Tensor> {+ match self {+ Self::Silu => Ok(down),+ Self::ClampedSwiglu { down_b, .. } => {+ let b = Self::rows(down_b, &ids()?, &down)?;+ down + b+ }+ }+ }++ fn rows(bias: &Tensor, ids: &Tensor, like: &Tensor) -> Result<Tensor> {+ let flat = ids.flatten_all()?.to_device(bias.device())?;+ bias.index_select(&flat, 0)?.reshape(like.shape())?.to_dtype(like.dtype())?.to_device(like.device())+ }+}+/// A MoE layer whose experts live in the ExpertStore instead of on the device.pub struct StreamedMoe {pub layer: usize,@@ -2178,9 +2272,55 @@ pub struct StreamedMoe {pub top_k: usize,pub norm_topk_prob: bool,pub prefetch_k: usize,+ pub act: ExpertAct,}impl StreamedMoe {+ /// Stand-ins on the parallel path, all on the GPU (2 experts a word): the first pick stays the+ /// router's best; the second is the best-scoring expert in the bank other than the first,+ /// which is the router's own second whenever that one is in the bank. Weights = the softmax+ /// over the two picks. The bank's map goes up as a small mask; nothing is read back.+ fn stand_in_gpu(&self, logits: &Tensor, top_ids: &Tensor, top_w: &Tensor) -> Result<(Tensor, Tensor)> {+ let e = logits.dim(D::Minus1)?;+ let resident: HashSet<u32> = self.store.resident_ready(self.layer).map(|m| m.into_keys().collect()).unwrap_or_default();+ let mask: Vec<f32> = (0..e as u32).map(|x| if resident.contains(&x) { 0.0 } else { -1e30 }).collect();+ let mask = Tensor::from_vec(mask, (1, e), logits.device())?.to_dtype(logits.dtype())?;+ if self.top_k > 2 {+ let k = self.top_k;+ let with_keep = |keep: usize| -> Result<Tensor> {+ let kept = top_ids.narrow(D::Minus1, 0, keep)?.contiguous()?;+ let cand = logits.broadcast_add(&mask)?.arg_sort_last_dim(false)?.narrow(D::Minus1, 0, k)?.contiguous()?;+ // Candidates already kept are pushed to the back, then the best k - keep of the rest are taken.+ let taken = cand.reshape((k, 1))?.broadcast_eq(&kept.reshape((1, keep))?)?.to_dtype(DType::F32)?.max_keepdim(1)?.reshape((1, k))?;+ let rank = (Tensor::arange(0f32, k as f32, logits.device())?.reshape((1, k))? + (taken * 1000.0)?)?;+ let pick = rank.arg_sort_last_dim(true)?.narrow(D::Minus1, 0, k - keep)?.contiguous()?;+ let ids = Tensor::cat(&[&kept, &cand.gather(&pick, D::Minus1)?], D::Minus1)?.contiguous()?;+ Ok(ids)+ };+ let ids = match stand_in_w() {+ Some(line) => {+ let heavy = top_w.narrow(D::Minus1, 1, 1)?.to_dtype(DType::F32)?.ge(line)?.broadcast_as((1, k))?.contiguous()?;+ heavy.where_cond(&with_keep(2)?, &with_keep(1)?)?+ }+ None => with_keep(stand_in_keep(k))?,+ };+ let w = candle_nn::ops::softmax_last_dim(&logits.gather(&ids, D::Minus1)?)?;+ return Ok((ids, w));+ }+ let ranked = logits.broadcast_add(&mask)?.arg_sort_last_dim(false)?.narrow(D::Minus1, 0, 2)?.contiguous()?;+ let first = top_ids.narrow(D::Minus1, 0, 1)?.contiguous()?;+ let m0 = ranked.narrow(D::Minus1, 0, 1)?.contiguous()?;+ let m1 = ranked.narrow(D::Minus1, 1, 1)?.contiguous()?;+ let stand_in = m0.eq(&first)?.where_cond(&m1, &m0)?;+ // Only a light second pick is swapped; a second pick that carries real weight is read for real.+ let own = top_ids.narrow(D::Minus1, 1, 1)?.contiguous()?;+ let light = top_w.narrow(D::Minus1, 1, 1)?.to_dtype(DType::F32)?.lt(stand_in_limit())?;+ let second = light.where_cond(&stand_in, &own)?;+ let ids = Tensor::cat(&[&first, &second], D::Minus1)?.contiguous()?;+ let w = candle_nn::ops::softmax_last_dim(&logits.gather(&ids, D::Minus1)?)?;+ Ok((ids, w))+ }+/// The plain path: one quantised matmul per expert, experts read fresh from the file.fn reference(&self, xs: &Tensor, ids: &[Vec<u32>], top_w: &Tensor) -> Result<Tensor> {let device = xs.device();@@ -2203,8 +2343,9 @@ impl StreamedMoe {let (gate, up, down) = (QMatMul::from_arc(g)?, QMatMul::from_arc(u)?, QMatMul::from_arc(d)?);let idx = Tensor::new(tokens.as_slice(), device)?;let x = xs.index_select(&idx, 0)?;- let hidden = (candle_nn::ops::silu(&gate.forward(&x)?)? * up.forward(&x)?)?;- let y = down.forward(&hidden)?;+ let picked = || Tensor::new(vec![experts[j]; tokens.len()], device);+ let hidden = self.act.hidden(&gate.forward(&x)?, &up.forward(&x)?, picked)?;+ let y = self.act.finish(down.forward(&hidden)?, picked)?;let w = Tensor::new(w.as_slice(), device)?.unsqueeze(1)?;parts[j] = Some((idx, y.broadcast_mul(&w)?));Ok(())@@ -2215,7 +2356,9 @@ impl StreamedMoe {if idx.is_empty() {return xs.zeros_like();}- xs.zeros_like()?.index_add(&Tensor::cat(&idx, 0)?, &Tensor::cat(&ys, 0)?, 0)+ // The output is as wide as down's rows, which a padded input can exceed.+ let ys = Tensor::cat(&ys, 0)?;+ Tensor::zeros((xs.dim(0)?, ys.dim(1)?), ys.dtype(), xs.device())?.index_add(&Tensor::cat(&idx, 0)?, &ys, 0)}pub fn forward(&self, xs: &Tensor) -> Result<Tensor> {@@ -2262,10 +2405,11 @@ impl StreamedMoe {let gate = xs.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 0, self.top_k, false))?;let up = xs.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 1, self.top_k, false))?;let inter = gate.dim(2)?;- let hidden = (candle_nn::ops::silu(&gate)? * &up)?.reshape((n * self.top_k, inter))?;+ let hidden = self.act.hidden(&gate, &up, || Ok(top_ids.clone()))?.reshape((n * self.top_k, ()))?;let down = hidden.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 2, self.top_k, true))?;+ let down = self.act.finish(down, || Ok(top_ids.clone()))?;let out = down.broadcast_mul(&top_w.unsqueeze(2)?)?.sum(1)?;- return out.reshape((b, s, h))?.to_dtype(dtype);+ return out.reshape((b, s, ()))?.to_dtype(dtype);}let logits = self.gate.forward(&xs)?;@@ -2280,6 +2424,8 @@ impl StreamedMoe {// the experts and signalled. The CPU moves straight on to the next layer.if n == 1 && self.store.s4.is_some() {let s4 = self.store.s4.as_ref().unwrap();+ // Lab (`COACHWHIP_STANDIN=1`, 2 experts a word): stand-ins chosen on the GPU.+ let (top_ids, top_w) = if self.top_k >= 2 && std::env::var("COACHWHIP_STANDIN").is_ok() { self.stand_in_gpu(&logits, &top_ids, &top_w)? } else { (top_ids, top_w) };if self.layer == 0 {self.store.ensure_all_banks()?;}@@ -2331,13 +2477,14 @@ impl StreamedMoe {let gate = xs.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 0, self.top_k, false))?;let up = xs.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 1, self.top_k, false))?;let inter = gate.dim(2)?;- let hidden = (candle_nn::ops::silu(&gate)? * &up)?.reshape((n * self.top_k, inter))?;+ let hidden = self.act.hidden(&gate, &up, || Ok(top_ids.clone()))?.reshape((n * self.top_k, ()))?;let down = hidden.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 2, self.top_k, true))?;+ let down = self.act.finish(down, || Ok(top_ids.clone()))?;let out = down.broadcast_mul(&top_w.unsqueeze(2)?)?.sum(1)?;if let Some(tx) = s4.tx.lock().unwrap().as_ref() {let _ = tx.send(S4Job { layer: self.layer, seq, n_ids: self.top_k, n_guess, prefetch_k: self.prefetch_k, ahead: self.ahead });}- return out.reshape((b, s, h))?.to_dtype(dtype);+ return out.reshape((b, s, ()))?.to_dtype(dtype);}// S6 measurement (profile mode, one word at a time): what a bank-only draft would pick.let draft_scores: Option<Vec<f32>> = if prof { Some(logits.flatten_all()?.to_dtype(DType::F32)?.to_vec1()?) } else { None };@@ -2357,6 +2504,37 @@ impl StreamedMoe {// Keep this wait if this code ever changes.let t_rb = Instant::now();let ids: Vec<Vec<u32>> = top_ids.to_vec2()?;+ // Lab (`COACHWHIP_STANDIN=1`): a word keeps its strongest pick (read from the SSD if it must+ // be), and each weaker pick that is not in the bank is swapped for the resident expert the+ // router scores highest for this word. The weights are the softmax over the final picks.+ let (ids, top_ids, top_w) = if n == 1 && std::env::var("COACHWHIP_STANDIN").is_ok() {+ let scores: Vec<f32> = logits.flatten_all()?.to_dtype(DType::F32)?.to_vec1()?;+ let resident: HashSet<u32> = self.store.resident(self.layer).into_iter().collect();+ let w0: Vec<f32> = top_w.flatten_all()?.to_dtype(DType::F32)?.to_vec1()?;+ let mut row = ids[0].clone();+ let keep = match (stand_in_w(), row.len() > 2) {+ (Some(line), true) => if w0[1] >= line { 2 } else { 1 },+ _ => stand_in_keep(row.len()),+ };+ for p in keep..row.len() {+ if !resident.contains(&row[p]) && (row.len() > 2 || w0[p] < stand_in_limit()) {+ let best = resident.iter().copied().filter(|e| !row.contains(e)).max_by(|a, b| scores[*a as usize].partial_cmp(&scores[*b as usize]).unwrap_or(std::cmp::Ordering::Equal));+ if let Some(e) = best {+ row[p] = e;+ self.store.stats.lock().unwrap().stand_ins += 1;+ }+ }+ }+ let m = row.iter().map(|&e| scores[e as usize]).fold(f32::NEG_INFINITY, f32::max);+ let ex: Vec<f32> = row.iter().map(|&e| (scores[e as usize] - m).exp()).collect();+ let z: f32 = ex.iter().sum();+ let k = row.len();+ let new_ids = Tensor::from_vec(row.clone(), (1, k), xs.device())?;+ let new_w = Tensor::from_vec(ex.iter().map(|v| v / z).collect::<Vec<f32>>(), (1, k), xs.device())?.to_dtype(top_w.dtype())?;+ (vec![row], new_ids, new_w)+ } else {+ (ids, top_ids, top_w)+ };let t_rb = t_rb.elapsed().as_secs_f64();// The GPU has drained: the misses the layer before used sharp can be squeezed now.if use_sharp() {@@ -2410,6 +2588,9 @@ impl StreamedMoe {};if let Some(logits) = early {let scores: Vec<f32> = logits.flatten_all()?.to_dtype(DType::F32)?.to_vec1()?;+ if let Ok(path) = std::env::var("COACHWHIP_GUESSLOG") {+ guess_log(&path, self.layer, self.ahead, &ids[0], &top_w, &scores)?;+ }if let Some(p_min) = q_rule() {// The probability rule: a candidate is read iff its router probability, calibrated// on the route dumps, says it is more likely needed than its bytes cost. On the@@ -2477,8 +2658,10 @@ impl StreamedMoe {let gate = xs.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 0, k, false))?;let up = xs.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 1, k, false))?;let inter = gate.dim(2)?;- let hidden = (candle_nn::ops::silu(&gate)? * &up)?.reshape((k, inter))?;+ let picked = || top_ids.index_select(&idx, 1);+ let hidden = self.act.hidden(&gate, &up, picked)?.reshape((k, ()))?;let down = hidden.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 2, k, true))?;+ let down = self.act.finish(down, picked)?;Ok(Some(down.broadcast_mul(&w.unsqueeze(2)?)?.sum(1)?))};let hit_pos: Vec<usize> = (0..row.len()).filter(|&p| !miss_set.contains(&row[p])).collect();@@ -2520,8 +2703,10 @@ impl StreamedMoe {let gate = xs.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 0, k, false))?;let up = xs.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 1, k, false))?;let inter = gate.dim(2)?;- let hidden = (candle_nn::ops::silu(&gate)? * &up)?.reshape((k, inter))?;- let mut down = hidden.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 2, k, true))?; // (1, k, h)+ let picked = || top_ids.index_select(&idx, 1);+ let hidden = self.act.hidden(&gate, &up, picked)?.reshape((k, ()))?;+ let down = hidden.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 2, k, true))?; // (1, k, h)+ let mut down = self.act.finish(down, picked)?;if repair_on() {let h = down.dim(2)?;let rep = self.store.repair.lock().unwrap();@@ -2546,8 +2731,10 @@ impl StreamedMoe {let gate = xs.apply_op2_no_bwd(&slots, &self.store.fresh_op(&parts[0], self.layer, 0, 1, false))?;let up = xs.apply_op2_no_bwd(&slots, &self.store.fresh_op(&parts[1], self.layer, 1, 1, false))?;let inter = gate.dim(2)?;- let hidden = (candle_nn::ops::silu(&gate)? * &up)?.reshape((1, inter))?;+ let picked = || top_ids.index_select(&idx, 1);+ let hidden = self.act.hidden(&gate, &up, picked)?.reshape((1, ()))?;let down = hidden.apply_op2_no_bwd(&slots, &self.store.fresh_op(&parts[2], self.layer, 2, 1, true))?;+ let down = self.act.finish(down, picked)?;add(down.broadcast_mul(&w.unsqueeze(2)?)?.sum(1)?)?;}match out {@@ -2570,8 +2757,10 @@ impl StreamedMoe {let gate = xs.apply_op2_no_bwd(&slots, &op(0, k, false))?;let up = xs.apply_op2_no_bwd(&slots, &op(1, k, false))?;let inter = gate.dim(2)?;- let hidden = (candle_nn::ops::silu(&gate)? * &up)?.reshape((k, inter))?;+ let picked = || top_ids.index_select(&idx, 1);+ let hidden = self.act.hidden(&gate, &up, picked)?.reshape((k, ()))?;let down = hidden.apply_op2_no_bwd(&slots, &op(2, k, true))?;+ let down = self.act.finish(down, picked)?;Ok(Some(down.broadcast_mul(&w.unsqueeze(2)?)?.sum(1)?))};let sharp_pos: Vec<usize> = (0..row.len()).filter(|&p| slot_of[&row[p]] & SHARP_FLAG != 0).collect();@@ -2587,8 +2776,9 @@ impl StreamedMoe {let gate = xs.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 0, self.top_k, false))?; // (n, k, inter)let up = xs.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 1, self.top_k, false))?;let inter = gate.dim(2)?;- let hidden = (candle_nn::ops::silu(&gate)? * &up)?.reshape((n * self.top_k, inter))?;+ let hidden = self.act.hidden(&gate, &up, || Ok(top_ids.clone()))?.reshape((n * self.top_k, ()))?;let down = hidden.apply_op2_no_bwd(&slots, &self.store.bank_op(self.layer, 2, self.top_k, true))?; // (n, k, h)+ let down = self.act.finish(down, || Ok(top_ids.clone()))?;if n == 1 && shaped() && self.store.requant.is_some() {self.store.note_shape(self.layer, &xs, &hidden)?;}@@ -2607,6 +2797,6 @@ impl StreamedMoe {st.maths_seconds += maths_started.elapsed().as_secs_f64();}}- out.reshape((b, s, h))?.to_dtype(dtype)+ out.reshape((b, s, ()))?.to_dtype(dtype)}}diff --git a/crates/engine/src/lib.rs b/crates/engine/src/lib.rsindex 096ea41..dc0e13f 100644--- a/crates/engine/src/lib.rs+++ b/crates/engine/src/lib.rs@@ -6,4 +6,5 @@ pub mod gpu_sync;pub mod requant;pub mod model;pub mod model_next;+pub mod model_gptoss;mod mv_id;diff --git a/crates/engine/src/model.rs b/crates/engine/src/model.rsindex b80ccb3..0d0a01f 100644--- a/crates/engine/src/model.rs+++ b/crates/engine/src/model.rs@@ -168,6 +168,7 @@ impl Qwen3Moe {let p = format!("blk.{layer}");let gate = gg.tensor(&format!("{p}.ffn_gate_inp.weight"))?.dequantize(device)?.to_dtype(DType::F32)?;let moe = StreamedMoe {+ act: crate::experts::ExpertAct::Silu,layer,gate: Linear::new(gate, None),next_gate: None,@@ -249,6 +250,7 @@ impl Qwen3Moe {pub enum Model {Qwen3Moe(Qwen3Moe),Qwen3Next(crate::model_next::Qwen3Next),+ GptOss(crate::model_gptoss::GptOss),}impl Model {@@ -264,6 +266,7 @@ impl Model {// f16 for attention: bf16 garbles long answers in this model."qwen3moe" => Ok(Self::Qwen3Moe(Qwen3Moe::load(path, device, DType::F16, settings)?)),"qwen3next" | "qwen35moe" => Ok(Self::Qwen3Next(crate::model_next::Qwen3Next::load(path, device, settings, arch.as_str())?)),+ "gpt-oss" => Ok(Self::GptOss(crate::model_gptoss::GptOss::load(path, device, settings)?)),other => candle::bail!("this model's architecture is \"{other}\", which Coachwhip does not support yet (today: qwen3moe, qwen3next). \Open an issue at https://github.com/1picassoai/coachwhip/issues so it moves up the list."@@ -275,6 +278,7 @@ impl Model {match self {Self::Qwen3Moe(m) => m.forward(x, offset),Self::Qwen3Next(m) => m.forward(x, offset),+ Self::GptOss(m) => m.forward(x, offset),}}@@ -282,6 +286,7 @@ impl Model {match self {Self::Qwen3Moe(m) => m.clear_kv_cache(),Self::Qwen3Next(m) => m.clear_kv_cache(),+ Self::GptOss(m) => m.clear_kv_cache(),}}@@ -289,11 +294,17 @@ impl Model {match self {Self::Qwen3Moe(m) => m.report(),Self::Qwen3Next(m) => m.report(),+ Self::GptOss(m) => m.report(),}}+ /// True for a model whose chat format is OpenAI's harmony (gpt-oss) rather than ChatML.+ pub fn harmony(&self) -> bool {+ matches!(self, Self::GptOss(_))+ }+pub fn thinks(&self) -> bool {- self.think().thinks()+ self.harmony() || self.think().thinks()}/// S6 measurement (Qwen3-Next family only): see `Qwen3Next::speculate_measure`.@@ -301,6 +312,7 @@ impl Model {match self {Self::Qwen3Moe(_) => Ok(None),Self::Qwen3Next(m) => m.speculate_measure(last, offset, k).map(Some),+ Self::GptOss(_) => Ok(None),}}@@ -309,6 +321,7 @@ impl Model {match self {Self::Qwen3Moe(_) => Ok(false),Self::Qwen3Next(m) => m.foresee(last, offset, steps),+ Self::GptOss(_) => Ok(false),}}@@ -317,6 +330,7 @@ impl Model {match self {Self::Qwen3Moe(_) => Ok(None),Self::Qwen3Next(m) => m.speculate(last, offset, k),+ Self::GptOss(_) => Ok(None),}}@@ -330,6 +344,7 @@ impl Model {match self {Self::Qwen3Moe(m) => m.think,Self::Qwen3Next(m) => m.think,+ Self::GptOss(m) => m.think,}}@@ -338,6 +353,7 @@ impl Model {match self {Self::Qwen3Moe(m) => m.progress = progress,Self::Qwen3Next(m) => m.progress = progress,+ Self::GptOss(m) => m.progress = progress,}}}diff --git a/crates/engine/src/model_gptoss.rs b/crates/engine/src/model_gptoss.rsnew file mode 100644index 0000000..36105f6--- /dev/null+++ b/crates/engine/src/model_gptoss.rs@@ -0,0 +1,368 @@+//! gpt-oss (OpenAI's mixture of experts). The family owns attention (sliding windows and sinks),+//! the biases and YaRN; the experts stream through the core like every other family, with the+//! family's clamped activation plugged into the core's hook. Experts are MXFP4.+use crate::experts::{ExpertAct, ExpertStore, Settings, StreamedMoe};+use crate::model::{Progress, ThinkOpen};+use candle::quantized::{ggml_file::qtensor_from_ggml, gguf_file, GgmlDType};+use candle::{DType, Device, Module, Result, Tensor};+use candle_nn::Linear;+use candle_transformers::models::quantized_qwen3::Gguf;+use candle_transformers::models::with_tracing::QMatMul;+use candle_transformers::quantized_nn::RmsNorm;+use std::os::unix::fs::FileExt;+use std::path::Path;+use std::sync::Arc;++const SWIGLU_LIMIT: f32 = 7.0;+const SWIGLU_ALPHA: f64 = 1.702;++struct Layer {+ attn_norm: RmsNorm,+ post_norm: RmsNorm,+ wq: QMatMul,+ bq: Tensor,+ wk: QMatMul,+ bk: Tensor,+ wv: QMatMul,+ bv: Tensor,+ wo: QMatMul,+ bo: Tensor,+ sinks: Tensor,+ moe: StreamedMoe,+ sliding: bool,+ k_cache: Option<Tensor>,+ v_cache: Option<Tensor>,+ /// The position of the first key the cache still holds (a sliding layer drops old ones).+ kv_start: usize,+}++/// The word table, packed as the file stores it (Q8_0), on the CPU; rows are unpacked when asked for.+struct Table {+ bytes: Vec<u8>,+ dtype: GgmlDType,+ row_bytes: usize,+ hidden: usize,+}++impl Table {+ fn rows(&self, ids: &[u32]) -> Result<Tensor> {+ let mut buf = Vec::with_capacity(ids.len() * self.row_bytes);+ for &id in ids {+ let at = id as usize * self.row_bytes;+ buf.extend_from_slice(&self.bytes[at..at + self.row_bytes]);+ }+ qtensor_from_ggml(self.dtype, &buf, vec![ids.len(), self.hidden], &Device::Cpu)?.dequantize(&Device::Cpu)+ }+}++pub struct GptOss {+ table: Table,+ layers: Vec<Layer>,+ norm: RmsNorm,+ output: QMatMul,+ store: Arc<ExpertStore>,+ device: Device,+ heads: usize,+ kv_heads: usize,+ head_dim: usize,+ window: usize,+ /// The experts' input width in the file (wider than the model when squeezed and padded).+ xpad: usize,+ cos: Tensor,+ sin: Tensor,+ pub think: ThinkOpen,+ pub progress: Option<Progress>,+}++/// YaRN rotary frequencies, as the reference computes them (truncate = false), with the+/// attention factor folded into cos and sin.+fn yarn_tables(dim: usize, base: f64, factor: f64, orig: f64, beta_fast: f64, beta_slow: f64, max_pos: usize, device: &Device) -> Result<(Tensor, Tensor)> {+ let half = dim / 2;+ let corr = |rot: f64| (dim as f64 * (orig / (rot * 2.0 * std::f64::consts::PI)).ln()) / (2.0 * base.ln());+ let low = corr(beta_fast).max(0.0);+ let mut high = corr(beta_slow).min(dim as f64 - 1.0);+ if (high - low).abs() < 1e-9 {+ high += 0.001;+ }+ let mscale = 0.1 * factor.ln() + 1.0;+ let mut inv = Vec::with_capacity(half);+ for i in 0..half {+ let pos_freq = base.powf((2 * i) as f64 / dim as f64);+ let extrap = 1.0 / pos_freq;+ let interp = 1.0 / (factor * pos_freq);+ let ramp = ((i as f64 - low) / (high - low)).clamp(0.0, 1.0);+ let extrap_factor = 1.0 - ramp;+ inv.push(interp * (1.0 - extrap_factor) + extrap * extrap_factor);+ }+ let mut c = Vec::with_capacity(max_pos * half);+ let mut s = Vec::with_capacity(max_pos * half);+ for p in 0..max_pos {+ for f in &inv {+ let a = p as f64 * f;+ c.push((a.cos() * mscale) as f32);+ s.push((a.sin() * mscale) as f32);+ }+ }+ Ok((Tensor::from_vec(c, (max_pos, half), device)?, Tensor::from_vec(s, (max_pos, half), device)?))+}+++impl GptOss {+ pub fn load(path: &Path, device: &Device, settings: &Settings) -> Result<Self> {+ let mut file = std::fs::File::open(path)?;+ let ct = gguf_file::Content::read(&mut file).map_err(|e| e.with_path(path))?;+ let md_u = |k: &str| -> Result<usize> { ct.metadata.get(k).ok_or_else(|| candle::Error::Msg(format!("missing {k}")))?.to_u32().map(|v| v as usize) };+ let md_f = |k: &str| -> Result<f64> { ct.metadata.get(k).ok_or_else(|| candle::Error::Msg(format!("missing {k}")))?.to_f32().map(|v| v as f64) };+ let block_count = md_u("gpt-oss.block_count")?;+ let heads = md_u("gpt-oss.attention.head_count")?;+ let kv_heads = md_u("gpt-oss.attention.head_count_kv")?;+ let head_dim = md_u("gpt-oss.attention.key_length")?;+ let window = md_u("gpt-oss.attention.sliding_window")?;+ // Fast mode (`--experts n`, `COACHWHIP_TOPK=n`): fewer experts per word than the model was trained with.+ let trained_k = md_u("gpt-oss.expert_used_count")?;+ let top_k = match std::env::var("COACHWHIP_TOPK").ok().and_then(|v| v.parse::<usize>().ok()) {+ Some(k) if k >= 1 && k < trained_k => k,+ _ => trained_k,+ };+ let eps = md_f("gpt-oss.attention.layer_norm_rms_epsilon")?;+ let (cos, sin) = yarn_tables(+ head_dim,+ md_f("gpt-oss.rope.freq_base")?,+ md_f("gpt-oss.rope.scaling.factor")?,+ md_u("gpt-oss.rope.scaling.original_context_length")? as f64,+ md_f("gpt-oss.rope.scaling.yarn_beta_fast")?,+ md_f("gpt-oss.rope.scaling.yarn_beta_slow")?,+ 16384,+ device,+ )?;+ let template = ct.metadata.get("tokenizer.chat_template").and_then(|t| t.to_string().ok().cloned());+ let think = ThinkOpen::from_template(template.as_deref());++ // A squeezed file pads expert rows to whole 256-weight blocks; the input to the experts+ // (and to the router, which shares it) gets the same zeros.+ let cols = |m: &str| ct.tensor_infos.get(&format!("blk.0.ffn_{m}_exps.weight")).map(|i| i.shape.dims()[2]).unwrap_or(0);+ let (xpad, hpad) = (cols("gate"), cols("down"));+ let store = ExpertStore::new(&ct, device, block_count, path, settings)?;+ let table = {+ let q = ct.tensor(&mut file, "token_embd.weight", &Device::Cpu)?;+ let hidden = *q.shape().dims().last().unwrap();+ let dtype = q.dtype();+ let row_bytes = hidden / dtype.block_size() * dtype.type_size();+ Table { bytes: q.data()?.into_owned(), dtype, row_bytes, hidden }+ };+ let mut gg = Gguf::new(ct, &mut file, device.clone());+ let dense = |gg: &mut Gguf<&mut std::fs::File>, name: &str| -> Result<Tensor> { gg.tensor(name)?.dequantize(device)?.to_dtype(DType::F32) };+ let mut layers = Vec::with_capacity(block_count);+ for i in 0..block_count {+ let p = format!("blk.{i}");+ let moe = StreamedMoe {+ act: ExpertAct::ClampedSwiglu {+ alpha: SWIGLU_ALPHA,+ limit: SWIGLU_LIMIT,+ gate_b: dense(&mut gg, &format!("{p}.ffn_gate_exps.bias"))?,+ up_b: dense(&mut gg, &format!("{p}.ffn_up_exps.bias"))?,+ down_b: dense(&mut gg, &format!("{p}.ffn_down_exps.bias"))?,+ pad: hpad,+ },+ layer: i,+ gate: Linear::new(+ {+ let w = dense(&mut gg, &format!("{p}.ffn_gate_inp.weight"))?;+ let width = w.dim(1)?;+ if xpad > width { w.pad_with_zeros(1, 0, xpad - width)? } else { w }+ },+ Some(dense(&mut gg, &format!("{p}.ffn_gate_inp.bias"))?),+ ),+ next_gate: None,+ gate1: None,+ ahead: settings.ahead.max(1),+ store: store.clone(),+ top_k,+ norm_topk_prob: true,+ prefetch_k: settings.prefetch,+ };+ layers.push(Layer {+ attn_norm: gg.rms_norm(&format!("{p}.attn_norm.weight"), eps)?,+ post_norm: gg.rms_norm(&format!("{p}.post_attention_norm.weight"), eps)?,+ wq: gg.qmatmul(&format!("{p}.attn_q.weight"))?,+ bq: dense(&mut gg, &format!("{p}.attn_q.bias"))?,+ wk: gg.qmatmul(&format!("{p}.attn_k.weight"))?,+ bk: dense(&mut gg, &format!("{p}.attn_k.bias"))?,+ wv: gg.qmatmul(&format!("{p}.attn_v.weight"))?,+ bv: dense(&mut gg, &format!("{p}.attn_v.bias"))?,+ wo: gg.qmatmul(&format!("{p}.attn_output.weight"))?,+ bo: dense(&mut gg, &format!("{p}.attn_output.bias"))?,+ sinks: dense(&mut gg, &format!("{p}.attn_sinks.weight"))?,+ moe,+ // The reference's layer_types start with sliding_attention and alternate.+ sliding: i % 2 == 0,+ k_cache: None,+ v_cache: None,+ kv_start: 0,+ });+ }+ // The guess for a later layer comes from that layer's own router, run early.+ let ahead = settings.ahead.max(1);+ for i in ahead..layers.len() {+ layers[i - ahead].moe.next_gate = Some(layers[i].moe.gate.clone());+ if ahead >= 2 {+ layers[i - ahead].moe.gate1 = Some(layers[i - ahead + 1].moe.gate.clone());+ }+ }+ let norm = gg.rms_norm("output_norm.weight", eps)?;+ let output = gg.qmatmul("output.weight")?;+ eprintln!("coachwhip: gpt-oss, {block_count} layers, {heads} heads over {kv_heads}, sliding window {window}, {top_k} experts per word");+ let m = Self { table, layers, norm, output, store, device: device.clone(), heads, kv_heads, head_dim, window, xpad, cos, sin, think, progress: None };+ if std::env::var("COACHWHIP_GPTOSS_CHECK").is_ok() {+ check_experts(path, 0, device)?;+ }+ Ok(m)+ }++ pub fn forward(&mut self, x: &Tensor, offset: usize) -> Result<Tensor> {+ let ids: Vec<u32> = x.flatten_all()?.to_vec1()?;+ if !self.store.drafting() {+ self.store.note_tokens(ids.clone());+ }+ let n = ids.len();+ let mut xs = self.table.rows(&ids)?.to_dtype(DType::F32)?.to_device(&self.device)?;+ let n_layers = self.layers.len();+ let (heads, kv_heads, head_dim, window) = (self.heads, self.kv_heads, self.head_dim, self.window);+ let group = heads / kv_heads;+ let scale = 1.0 / (head_dim as f64).sqrt();+ let cos = self.cos.narrow(0, offset, n)?;+ let sin = self.sin.narrow(0, offset, n)?;+ // Lab timing switches (text is junk with either on): skip the experts, or skip attention.+ let no_moe = std::env::var("COACHWHIP_GPTOSS_NOMOE").is_ok();+ let no_attn = std::env::var("COACHWHIP_GPTOSS_NOATTN").is_ok();+ for l in 0..n_layers {+ let layer = &mut self.layers[l];+ if !no_attn {+ // Attention with sinks; sliding layers see the last `window` tokens only.+ let h = layer.attn_norm.forward(&xs)?;+ // Heads first, (1, heads, n, d), so the rotation is candle's one fused kernel and the+ // cache grows along its own axis without a transpose per word.+ let q = layer.wq.forward(&h)?.broadcast_add(&layer.bq)?.reshape((1, n, heads, head_dim))?.transpose(1, 2)?.contiguous()?;+ let k = layer.wk.forward(&h)?.broadcast_add(&layer.bk)?.reshape((1, n, kv_heads, head_dim))?.transpose(1, 2)?.contiguous()?;+ let v = layer.wv.forward(&h)?.broadcast_add(&layer.bv)?.reshape((1, n, kv_heads, head_dim))?.transpose(1, 2)?.contiguous()?;+ let q = candle_nn::rotary_emb::rope(&q, &cos, &sin)?;+ let k = candle_nn::rotary_emb::rope(&k, &cos, &sin)?;+ // The cache holds keys and values in f16, as GGML does: half the memory, which the+ // Wrangler hands to the bank. They are widened back to f32 for the maths.+ let (k, v) = (k.to_dtype(DType::F16)?, v.to_dtype(DType::F16)?);+ let k = match &layer.k_cache { Some(c) => Tensor::cat(&[c, &k], 2)?, None => k };+ let v = match &layer.v_cache { Some(c) => Tensor::cat(&[c, &v], 2)?, None => v };+ // A sliding layer never looks further back than its window, so it keeps no more: one new+ // word needs exactly the last `window` keys (itself included), and then no mask at all.+ let keep = if n == 1 { window } else { window + n };+ let (k, v, first) = if layer.sliding && k.dim(2)? > keep {+ let drop = k.dim(2)? - keep;+ (k.narrow(2, drop, keep)?.contiguous()?, v.narrow(2, drop, keep)?.contiguous()?, layer.kv_start + drop)+ } else {+ (k, v, layer.kv_start)+ };+ layer.kv_start = first;+ layer.k_cache = Some(k.clone());+ layer.v_cache = Some(v.clone());+ let t = k.dim(2)?;+ // Grouped heads share their keys: q as (kv_heads, group * rows, d) against k as (kv_heads, t, d),+ // so nothing is copied out to all 64 heads.+ let kt = k.squeeze(0)?.to_dtype(DType::F32)?.t()?.contiguous()?;+ let vv = v.squeeze(0)?.to_dtype(DType::F32)?;+ // A long prompt is attended a block of rows at a time: its score table (heads x rows x+ // keys) then stays a few hundred MB instead of several GB on a 16 GB Mac.+ const ROWS: usize = 64;+ let mut parts = Vec::with_capacity(n.div_ceil(ROWS));+ let mut i0 = 0;+ while i0 < n {+ let rows = ROWS.min(n - i0);+ let qg = q.narrow(2, i0, rows)?.contiguous()?.reshape((kv_heads, group * rows, head_dim))?;+ let scores = (qg.matmul(&kt)? * scale)?.reshape((heads, rows, t))?;+ let scores = if n == 1 {+ scores+ } else {+ let mut mask = vec![0f32; rows * t];+ for i in 0..rows {+ let pos = offset + i0 + i;+ for j in 0..t {+ let at = first + j;+ if at > pos || (layer.sliding && at + window <= pos) {+ mask[i * t + j] = f32::NEG_INFINITY;+ }+ }+ }+ scores.broadcast_add(&Tensor::from_vec(mask, (1, rows, t), &self.device)?)?+ };+ let sinks = layer.sinks.reshape((heads, 1, 1))?.broadcast_as((heads, rows, 1))?.contiguous()?;+ let all = Tensor::cat(&[&scores, &sinks], 2)?;+ let probs = candle_nn::ops::softmax_last_dim(&all)?.narrow(2, 0, t)?.contiguous()?.reshape((kv_heads, group * rows, t))?;+ parts.push(probs.matmul(&vv)?.reshape((heads, rows, head_dim))?.transpose(0, 1)?.reshape((rows, heads * head_dim))?);+ i0 += rows;+ }+ let ctx = if parts.len() == 1 { parts.pop().unwrap() } else { Tensor::cat(&parts, 0)? };+ let o = layer.wo.forward(&ctx)?.broadcast_add(&layer.bo)?;+ xs = (xs + o)?;+ }+ if no_moe {+ continue;+ }+ // The experts, through the core.+ let post = layer.post_norm.forward(&xs)?;+ let width = post.dim(1)?;+ let post = if self.xpad > width { post.pad_with_zeros(1, 0, self.xpad - width)? } else { post };+ let post = post.unsqueeze(0)?;+ let routed = layer.moe.forward(&post)?.squeeze(0)?;+ xs = (xs + routed)?;+ if n > 1 {+ if let Some(p) = &mut self.progress {+ p(crate::model::Step::Layer(l + 1, n_layers));+ }+ }+ }+ let last = self.norm.forward(&xs.narrow(0, n - 1, 1)?)?;+ self.output.forward(&last)?.to_dtype(DType::F32)+ }++ pub fn clear_kv_cache(&mut self) {+ for l in &mut self.layers {+ l.k_cache = None;+ l.v_cache = None;+ l.kv_start = 0;+ }+ }++ pub fn report(&self) {+ self.store.report_and_reset();+ }+}++/// Lab, step 4 gate: every expert matrix of one layer, multiplied on `gpu`, against the same+/// matrix decoded to f32 and multiplied exactly on the CPU. Prints the worst relative error.+pub fn check_experts(path: &Path, layer: usize, gpu: &Device) -> Result<()> {+ let mut file = std::fs::File::open(path)?;+ let ct = gguf_file::Content::read(&mut file).map_err(|e| e.with_path(path))?;+ let mut worst = (0f32, 0usize, "gate");+ let mut count = 0;+ for name in ["gate", "up", "down"] {+ let info = &ct.tensor_infos[&format!("blk.{layer}.ffn_{name}_exps.weight")];+ let dims = info.shape.dims().to_vec();+ let (n, rows, cols) = (dims[0], dims[1], dims[2]);+ let bytes = rows * cols / info.ggml_dtype.block_size() * info.ggml_dtype.type_size();+ let x = Tensor::randn(0f32, 1f32, (3, cols), &Device::Cpu)?;+ count = n;+ for e in 0..n {+ let mut buf = vec![0u8; bytes];+ file.read_exact_at(&mut buf, ct.tensor_data_offset + info.offset + (e * bytes) as u64)?;+ let w = qtensor_from_ggml(info.ggml_dtype, &buf, vec![rows, cols], &Device::Cpu)?.dequantize(&Device::Cpu)?;+ let exact = x.matmul(&w.t()?)?;+ let q = QMatMul::from_weights(Arc::new(qtensor_from_ggml(info.ggml_dtype, &buf, vec![rows, cols], gpu)?))?;+ let got = q.forward(&x.to_device(gpu)?)?.to_device(&Device::Cpu)?;+ let err = (got - &exact)?.abs()?.max_all()?.to_scalar::<f32>()? / exact.abs()?.max_all()?.to_scalar::<f32>()?;+ if err > worst.0 {+ worst = (err, e, name);+ }+ }+ }+ eprintln!("coachwhip: expert check, layer {layer}, {count} experts x 3 matrices on {gpu:?}: worst relative error {:.2e} (expert {} {})", worst.0, worst.1, worst.2);+ Ok(())+}diff --git a/crates/engine/src/model_next.rs b/crates/engine/src/model_next.rsindex 1682237..c4b303e 100644--- a/crates/engine/src/model_next.rs+++ b/crates/engine/src/model_next.rs@@ -796,6 +796,7 @@ impl Qwen3Next {};let gate = dense(&mut gg, &format!("{p}.ffn_gate_inp.weight"))?;let moe = StreamedMoe {+ act: crate::experts::ExpertAct::Silu,layer,gate: Linear::new(gate, None),next_gate: None,diff --git a/crates/engine/src/mv_id.rs b/crates/engine/src/mv_id.rsindex 7f06413..aa2692a 100644--- a/crates/engine/src/mv_id.rs+++ b/crates/engine/src/mv_id.rs@@ -30,7 +30,7 @@ pub fn matvec_id(dst: &Buffer,) -> Result<(), MetalKernelError> {let (nth0, nth1, align) = match dtype {- GgmlDType::Q4_0 | GgmlDType::Q4_1 | GgmlDType::Q5_0 | GgmlDType::Q5_1 | GgmlDType::Q8_0 | GgmlDType::Q8_1 => (8, 8, 8),+ GgmlDType::Q4_0 | GgmlDType::Q4_1 | GgmlDType::Q5_0 | GgmlDType::Q5_1 | GgmlDType::Q8_0 | GgmlDType::Q8_1 | GgmlDType::Mxfp4 => (8, 8, 8),GgmlDType::Q2K => (2, 32, 8),GgmlDType::Q4K => (4, 8, 4),GgmlDType::Q3K | GgmlDType::Q5K => (2, 32, 4),@@ -43,6 +43,7 @@ pub fn matvec_id(GgmlDType::Q5_0 => "kernel_mul_mv_id_q5_0_f32",GgmlDType::Q5_1 => "kernel_mul_mv_id_q5_1_f32",GgmlDType::Q8_0 => "kernel_mul_mv_id_q8_0_f32",+ GgmlDType::Mxfp4 => "kernel_mul_mv_id_mxfp4_f32",GgmlDType::Q2K => "kernel_mul_mv_id_q2_K_f32",GgmlDType::Q3K => "kernel_mul_mv_id_q3_K_f32",GgmlDType::Q4K => "kernel_mul_mv_id_q4_K_f32",diff --git a/crates/prepare/src/main.rs b/crates/prepare/src/main.rsindex 32250e2..0cc5185 100644--- a/crates/prepare/src/main.rs+++ b/crates/prepare/src/main.rs@@ -18,7 +18,9 @@ fn target(name: &str, dtype: GgmlDType) -> GgmlDType {if name.ends_with(".ffn_gate_exps.weight") || name.ends_with(".ffn_up_exps.weight") {GgmlDType::Q2K} else if name.ends_with(".ffn_down_exps.weight") {- GgmlDType::Q3K+ // MXFP4 experts (gpt-oss) take down to Q2_K too: measured on the task-pool exam, the 120b+ // squeezed this way scores as the original. K-quant experts (Qwen) keep down at Q3_K.+ if dtype == GgmlDType::Mxfp4 || std::env::var("COACHWHIP_DOWN_Q2K").is_ok() { GgmlDType::Q2K } else { GgmlDType::Q3K }} else {dtype}@@ -92,9 +94,21 @@ fn dtype_code(d: GgmlDType) -> u32 {GgmlDType::Q6K => 14,GgmlDType::Q8K => 15,GgmlDType::BF16 => 30,+ GgmlDType::Mxfp4 => 39,}}+/// A K-quant reads rows in whole 256-weight blocks; a row that is not a multiple of 256 (gpt-oss:+/// 2880) is padded with zeros to the next one. The engine pads its input to match.+fn out_dims(dims: &[usize], from: GgmlDType, to: GgmlDType) -> Vec<usize> {+ let mut d = dims.to_vec();+ if to != from {+ let last = d.len() - 1;+ d[last] = d[last].div_ceil(to.block_size()) * to.block_size();+ }+ d+}+fn bytes_for(dtype: GgmlDType, elems: usize) -> usize {elems / dtype.block_size() * dtype.type_size()}@@ -134,7 +148,7 @@ fn main() -> Result<()> {changed += 1;}new_offsets.push(offset);- offset += bytes_for(to, info.shape.elem_count()) as u64;+ offset += bytes_for(to, out_dims(info.shape.dims(), info.ggml_dtype, to).iter().product()) as u64;offset = (offset + align - 1) / align * align;}eprintln!("coachwhip-prepare: {} tensors, {changed} to rewrite, {:.1} GB out", infos.len(), offset as f64 / 1e9);@@ -170,7 +184,7 @@ fn main() -> Result<()> {}for ((name, info), new_off) in infos.iter().zip(&new_offsets) {write_string(&mut w, name)?;- let dims = info.shape.dims();+ let dims = out_dims(info.shape.dims(), info.ggml_dtype, target(name, info.ggml_dtype));w.write_u32::<LittleEndian>(dims.len() as u32)?;for &d in dims.iter().rev() {w.write_u64::<LittleEndian>(d as u64)?;@@ -211,6 +225,21 @@ fn main() -> Result<()> {let data = unsafe { std::slice::from_raw_parts(dst_buf.contents() as *const u8, out_len) };w.write_all(data)?;written += out_len as u64;+ } else if out_dims(info.shape.dims(), info.ggml_dtype, to) != info.shape.dims() {+ // Padded rows: one expert at a time, so a big tensor never sits in memory as f32.+ let dims = info.shape.dims();+ let (n, rows, cols) = (dims[0], dims[1], dims[2]);+ let padded = out_dims(dims, info.ggml_dtype, to)[2];+ let per = bytes_for(info.ggml_dtype, rows * cols);+ for e in 0..n {+ let storage = QStorage::from_data(Cow::Owned(buf[e * per..(e + 1) * per].to_vec()), &Device::Cpu, info.ggml_dtype)?;+ let f = QTensor::new(storage, (rows, cols))?.dequantize(&Device::Cpu)?;+ let f = f.pad_with_zeros(1, 0, padded - cols)?;+ let q2 = QTensor::quantize(&f, to)?;+ let data = q2.data()?;+ w.write_all(&data)?;+ written += data.len() as u64;+ }} else {let storage = QStorage::from_data(Cow::Owned(buf), &Device::Cpu, info.ggml_dtype)?;let q = QTensor::new(storage, info.shape.clone())?;diff --git a/crates/server/src/chat.rs b/crates/server/src/chat.rsindex 8ba99d4..bcd48ba 100644--- a/crates/server/src/chat.rs+++ b/crates/server/src/chat.rs@@ -36,6 +36,66 @@ const SUMMARY_TOKENS: usize = 500;const SUMMARY_ASK: &str = "Summarise our conversation so far in at most 300 words, for someone who will continue it: what was asked, what was decided, and every name, file, function or fact the next answer will need. Plain text, no preamble.";+/// The start of every gpt-oss chat: the model's own default system message from its chat+/// template, at its default reasoning level.+const HARMONY_SYSTEM: &str = "<|start|>system<|message|>You are ChatGPT, a large language model trained by OpenAI.\nKnowledge cutoff: 2024-06\n\nReasoning: medium\n\n# Valid channels: analysis, commentary, final. Channel must be included for every message.<|end|>";++/// The chat format the model was trained on: ChatML (the Qwen families) or harmony (gpt-oss).+struct Fmt {+ harmony: bool,+}++impl Fmt {+ fn of(model: &Model) -> Self {+ Self { harmony: model.harmony() }+ }++ /// The words that end the assistant's turn. A stop word is never part of the answer.+ fn stops(&self, tokenizer: &Tokenizer) -> Vec<u32> {+ // Harmony: <|end|> only closes a message (the thinking ends with it); <|return|> ends the answer.+ let names: &[&str] = if self.harmony { &["<|return|>", "<|call|>"] } else { &["<|im_end|>"] };+ let vocab = tokenizer.get_vocab(true);+ names.iter().filter_map(|n| vocab.get(*n).copied()).collect()+ }++ /// What a new chat opens with, before its first turn.+ fn start(&self) -> &'static str {+ if self.harmony { HARMONY_SYSTEM } else { "" }+ }++ /// What closes the last answer before the next turn (the stop word itself was never read).+ fn close(&self) -> &'static str {+ if self.harmony { "<|end|>" } else { "<|im_end|>\n" }+ }++ fn user(&self, text: &str) -> String {+ if self.harmony { format!("<|start|>user<|message|>{text}<|end|>") } else { format!("<|im_start|>user\n{text}<|im_end|>\n") }+ }++ /// Opens the assistant's turn. gpt-oss with thinking on picks its own channel (analysis first);+ /// with thinking off it goes straight to its final channel.+ fn assistant(&self, model: &Model, think: bool) -> String {+ match (self.harmony, think) {+ (true, true) => "<|start|>assistant".to_string(),+ (true, false) => "<|start|>assistant<|channel|>final<|message|>".to_string(),+ _ => format!("<|im_start|>assistant\n{}", model.assistant_open(think)),+ }+ }+}++/// What the page shows of a gpt-oss answer that thinks: its analysis channel as a think block,+/// then its final channel. Format markers never show; one still arriving holds the text back.+fn harmony_view(raw: &str) -> String {+ let t = raw+ .replace("<|channel|>analysis<|message|>", "<think>\n")+ .replace("<|end|><|start|>assistant<|channel|>final<|message|>", "\n</think>\n\n")+ .replace("<|channel|>final<|message|>", "");+ match t.find("<|") {+ Some(i) => t[..i].to_string(),+ None => t,+ }+}+/// Separates the streamed text from the closing stats line.pub const STATS_MARK: &str = "\u{1}STATS:";@@ -115,23 +175,24 @@ pub fn generate(emit: &mut dyn FnMut(&str) -> Result<()>,) -> Result<Answer> {let think = think && model.thinks();- // The template's way of switching thinking on or off: what opens the assistant's turn.- let open = model.assistant_open(think);+ let fmt = Fmt::of(model);+ // The template's way of switching thinking on or off is in what opens the assistant's turn.+ let open = fmt.assistant(model, think);let encode = |text: String| -> Result<Vec<u32>> { Ok(tokenizer.encode(text, true).map_err(anyhow::Error::msg)?.get_ids().to_vec()) };- let eos = *tokenizer.get_vocab(true).get("<|im_end|>").unwrap();+ let stops = fmt.stops(tokenizer);let tell = progress.map(|p| Rc::new(RefCell::new(p)));// A follow-up closes the last answer and adds the new question; nothing earlier is read again.- let turn = format!("<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant\n{open}");+ let turn = format!("{}{open}", fmt.user(prompt));let mut tokens = Vec::new();let mut fresh = new_chat || session.pos == 0 || session.by_api;let mut summary = None;if !fresh {tokens.extend(session.pending);- tokens.extend(encode(format!("<|im_end|>\n{turn}"))?);+ tokens.extend(encode(format!("{}{turn}", fmt.close()))?);// Too long to go on: the model summarises the chat, and the summary opens a new one.if session.pos + tokens.len() + max_new > CONTEXT_LIMIT {- summary = Some(summarise(model, tokenizer, device, session, eos, &tell)?);+ summary = Some(summarise(model, tokenizer, device, session, &stops, &tell)?);fresh = true;}}@@ -139,8 +200,8 @@ pub fn generate(model.clear_kv_cache();*session = Session::default();tokens = encode(match &summary {- Some(s) => format!("<|im_start|>user\nSummary of our conversation so far, for context:\n{s}\n\nNow: {prompt}<|im_end|>\n<|im_start|>assistant\n{open}"),- None => turn,+ Some(s) => format!("{}{}{open}", fmt.start(), fmt.user(&format!("Summary of our conversation so far, for context:\n{s}\n\nNow: {prompt}"))),+ None => format!("{}{turn}", fmt.start()),})?;}@@ -175,7 +236,8 @@ pub fn generate_full(session.pending = None;}let tokens = prompt[session.fed.len()..].to_vec();- let a = answer(model, tokenizer, device, session, tokens, fresh, false, false, max_new, sampling, &None, emit);+ let think = model.harmony();+ let a = answer(model, tokenizer, device, session, tokens, fresh, false, think, max_new, sampling, &None, emit);session.by_api = true;a}@@ -196,7 +258,10 @@ fn answer(tell: &Option<Rc<RefCell<Progress>>>,emit: &mut dyn FnMut(&str) -> Result<()>,) -> Result<Answer> {- let eos = *tokenizer.get_vocab(true).get("<|im_end|>").unwrap();+ let fmt = Fmt::of(model);+ let stops = fmt.stops(tokenizer);+ // gpt-oss thinking: the channels are read from the format markers, so keep them when decoding.+ let channels = fmt.harmony && think;let mut sampler = LogitsProcessor::from_sampling(42, sampling.clone());let t0 = std::time::Instant::now();let logits = read_prompt(model, device, session.pos, &tokens, tell)?;@@ -206,10 +271,14 @@ fn answer(let read = t0.elapsed().as_secs_f64();// The answer so far; an end-of-turn word is never part of it.- let mut out = if next == eos { Vec::new() } else { vec![next] };+ let mut out = if stops.contains(&next) { Vec::new() } else { vec![next] };let mut sent = 0usize;let mut stream_text = |out: &[u32], sent: &mut usize| -> Result<()> {- let full = tokenizer.decode(out, true).map_err(anyhow::Error::msg)?;+ let full = if channels {+ harmony_view(&tokenizer.decode(out, false).map_err(anyhow::Error::msg)?)+ } else {+ tokenizer.decode(out, true).map_err(anyhow::Error::msg)?+ };// A token can be half of a multi-byte character; wait until the text is whole.if full.ends_with('\u{FFFD}') {return Ok(());@@ -244,12 +313,12 @@ fn answer(let speculate: usize = std::env::var("COACHWHIP_SPECULATE").ok().and_then(|v| v.parse().ok()).unwrap_or(0);let measure: usize = std::env::var("COACHWHIP_SPECULATE_MEASURE").ok().and_then(|v| v.parse().ok()).unwrap_or(0);let greedy = matches!(sampling, Sampling::ArgMax);- while out.len() < max_new && next != eos {+ while out.len() < max_new && !stops.contains(&next) {if speculate > 0 && greedy && max_new - out.len() > speculate {if let Some(seq) = model.speculate(next, base + out.len() - 1, speculate)? {for &w in &seq {next = w;- if w == eos {+ if stops.contains(&w) {break;}out.push(w);@@ -262,7 +331,7 @@ fn answer(if let Some((plain, _accepted)) = model.speculate_measure(next, base + out.len() - 1, measure)? {for &w in &plain {next = w;- if w == eos {+ if stops.contains(&w) {break;}out.push(w);@@ -276,7 +345,7 @@ fn answer(}let logits = model.forward(&Tensor::new(&[next], device)?.unsqueeze(0)?, base + out.len() - 1)?.squeeze(0)?;next = sampler.sample(&logits)?;- if next != eos {+ if !stops.contains(&next) {out.push(next);stream_text(&out, &mut sent)?;}@@ -284,7 +353,7 @@ fn answer(let write = t1.elapsed().as_secs_f64();let written = out.len();// Every word of the answer was read back, except the last one when the length limit cut it off.- let finished = next == eos;+ let finished = stops.contains(&next);let read_back = if finished { out.len() } else { out.len() - 1 };session.pos = base + read_back;session.pending = if finished { None } else { out.last().copied() };@@ -329,16 +398,17 @@ fn read_prompt(model: &mut Model, device: &Device, pos: usize, tokens: &[u32], t/// Asks the model, in the chat as it stands, for a summary of the chat; the chat is then thrown/// away and the summary opens the next one. Greedy, so it is the same every time.-fn summarise(model: &mut Model, tokenizer: &Tokenizer, device: &Device, session: &Session, eos: u32, tell: &Option<Rc<RefCell<Progress>>>) -> Result<String> {+fn summarise(model: &mut Model, tokenizer: &Tokenizer, device: &Device, session: &Session, stops: &[u32], tell: &Option<Rc<RefCell<Progress>>>) -> Result<String> {let mut tokens: Vec<u32> = session.pending.into_iter().collect();- let ask = format!("<|im_end|>\n<|im_start|>user\n{SUMMARY_ASK}<|im_end|>\n<|im_start|>assistant\n{}", model.assistant_open(false));+ let fmt = Fmt::of(model);+ let ask = format!("{}{}{}", fmt.close(), fmt.user(SUMMARY_ASK), fmt.assistant(model, false));tokens.extend(tokenizer.encode(ask, true).map_err(anyhow::Error::msg)?.get_ids().to_vec());let mut sampler = LogitsProcessor::from_sampling(42, Sampling::ArgMax);let logits = read_prompt(model, device, session.pos, &tokens, tell)?;let mut next = sampler.sample(&logits)?;let base = session.pos + tokens.len();let mut out = Vec::new();- while next != eos && out.len() < SUMMARY_TOKENS {+ while !stops.contains(&next) && out.len() < SUMMARY_TOKENS {out.push(next);if let Some(t) = tell {if out.len() % 8 == 0 {@@ -370,6 +440,22 @@ pub fn message_text(content: &serde_json::Value) -> String {}}+/// An OpenAI conversation in harmony (gpt-oss), ready for the assistant's turn. The client's own+/// system message becomes harmony's developer message; the model thinks before its final answer.+fn harmony_prompt(messages: &[serde_json::Value]) -> String {+ let mut text = HARMONY_SYSTEM.to_string();+ for m in messages {+ let content = message_text(&m["content"]);+ text.push_str(&match m["role"].as_str().unwrap_or("user") {+ "system" | "developer" => format!("<|start|>developer<|message|># Instructions\n\n{content}<|end|>"),+ "assistant" => format!("<|start|>assistant<|channel|>final<|message|>{content}<|end|>"),+ _ => format!("<|start|>user<|message|>{content}<|end|>"),+ });+ }+ text.push_str("<|start|>assistant");+ text+}+/// An OpenAI conversation in the model's chat format, ready for the assistant's turn; `open` is/// what the model's template puts at the start of that turn (see `Model::assistant_open`).pub fn chat_prompt(messages: &[serde_json::Value], open: &str) -> String {@@ -405,7 +491,8 @@ fn openai(mut s: TcpStream, body: &[u8], model: &mut Model, model_name: &str, to};let stream = q["stream"].as_bool().unwrap_or(false);let asked = q["max_completion_tokens"].as_u64().or(q["max_tokens"].as_u64());- let prompt = tokenizer.encode(chat_prompt(messages, model.assistant_open(false)), true).map_err(anyhow::Error::msg)?.get_ids().to_vec();+ let text = if model.harmony() { harmony_prompt(messages) } else { chat_prompt(messages, model.assistant_open(false)) };+ let prompt = tokenizer.encode(text, true).map_err(anyhow::Error::msg)?.get_ids().to_vec();let max_new = answer_budget(asked, max_new, prompt.len());let id = format!("chatcmpl-{}", std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.as_nanos()));let created = std::time::SystemTime::now().duration_since(std::time::UNIX_EPOCH).map_or(0, |d| d.as_secs());@@ -459,6 +546,11 @@ fn openai(mut s: TcpStream, body: &[u8], model: &mut Model, model_name: &str, to};println!(" api: read {} new tokens at {:.1} tok/s, wrote {} at {:.1} tok/s, chat now {} tokens", a.prompt_tokens, a.read_tps, a.written, a.write_tps, a.context);model.report();+ if model.harmony() {+ if let Some(i) = text.find("</think>") {+ text = text[i + "</think>".len()..].trim_start().to_string();+ }+ }let finish = if a.finished { "stop" } else { "length" };let usage = serde_json::json!({"prompt_tokens": prompt_len, "completion_tokens": a.written, "total_tokens": prompt_len + a.written});if stream {@@ -606,7 +698,14 @@ fn handle(mut s: TcpStream, model: &mut Model, model_name: &str, tokenizer: &Tokchunk(&mut s, format!("{STATS_MARK}{stats}").as_bytes())?;s.write_all(b"0\r\n\r\n")?;} else {- let page = if model.thinks() { PAGE.to_string() } else { PAGE.replace(THINK_BOX, "") };+ // gpt-oss is a reasoning model: its thinking switch starts ticked.+ let page = if model.harmony() {+ PAGE.replace(r#"id="think">"#, r#"id="think" checked>"#)+ } else if model.thinks() {+ PAGE.to_string()+ } else {+ PAGE.replace(THINK_BOX, "")+ };let page = page.replace("MODEL_NAME", &html_escape(model_name));write!(s, "HTTP/1.1 200 OK\r\nContent-Type: text/html; charset=utf-8\r\nContent-Length: {}\r\n\r\n", page.len())?;s.write_all(page.as_bytes())?;diff --git a/docs/GPT-OSS.md b/docs/GPT-OSS.mdnew file mode 100644index 0000000..4cf12de--- /dev/null+++ b/docs/GPT-OSS.md@@ -0,0 +1,38 @@+# gpt-oss-120b on a 16 GB Mac++OpenAI's gpt-oss-120b has 117 billion parameters and 128 experts in each of its 36 layers, of which it uses 4 for each word. On a Mac mini M4 with 16 GB, Coachwhip runs it at about 4.2 tok/s with every expert read for real, and at about 10 tok/s in fast mode.++All numbers here: Mac mini M4, 16 GB, macOS 26, the squeezed file (`prepare`, every expert matrix at Q2_K), `--parallel --flash-reads`, temperature 0. The task is one prompt, a generic cancellable async task pool in TypeScript with dependency resolution; the model thinks, then writes the code, about 4,000 words in all. The code is run against 20 hidden checks (ordering, the concurrency limit, failure cascades, cancellation, a task that ignores its abort signal, an empty list, and others), 4 checks to a point. A second version of the task, the same job worded as a build-pipeline runner, checks that a score is not a fluke.++## Speed and score, setup by setup++| Setup | Speed | Score | Second version |+|---|---|---|---|+| 4 experts, all read for real | 4.2 tok/s | 4.8 | |+| 3 experts, all read for real | 5.6 tok/s | 4.5 | |+| 4 experts, the first 2 real, stand-ins for the rest | 7.8 tok/s | 4.5 | 3.5 (7.9 tok/s) |+| **4 experts, stand-ins by weight (`--stand-ins`)** | **9.8 tok/s** | **4.0** | **4.0 (10.0 tok/s)** |+| 4 experts, only the first real | 9.6 tok/s | 4.5 | 0.8 (10.2 tok/s) |+| 2 experts | 7.1 tok/s | 0 (the code does not compile) | |++`--stand-ins` is the only setup near 10 tok/s that held its score on both versions of the task. Reading fewer experts (3 or 2) costs more than standing in for the light ones: the model was trained to use 4.++## What makes it fast++Speed on a 16 GB Mac is set by one thing: how many bytes each word reads from the SSD. Everything below cuts those bytes or hides them.++- **The squeeze.** `prepare` takes each expert from 13.2 MB (MXFP4) to 8.7 MB (Q2_K, rows padded to whole 256-weight blocks), so the bank on the GPU holds 25 experts a layer instead of 15. On this task the squeezed file scores the same as the original download.+- **The expert bank.** Recently used experts stay on the GPU and are reused; with every expert read for real, about 4 picks in 5 are already there.+- **The early guess.** While the GPU works on one layer, Coachwhip runs the router of a later layer on the same input and reads the experts it will want; for gpt-oss, 2 guesses a layer, 2 layers ahead.+- **Flash reads (`--flash-reads`).** Experts are read straight from the SSD, past macOS's page cache.+- **Stand-ins (`--stand-ins`).** A word's first pick is always read for real, and so is the second whenever the router gives it at least 0.3 of the weight. A lighter pick that is not in the bank is replaced by the best expert that is, by the router's own scores, and the weights are recomputed over the experts actually used. Bank hits go from about 78% to about 97%, and the bytes read per word fall by about seven-eighths.+- **Lean attention.** gpt-oss looks back only 128 words on every other layer; those layers keep only that window, and the shared keys are never copied out per head.++## Its chat format++gpt-oss speaks OpenAI's harmony format, with an analysis channel for its thinking and a final channel for the answer. The chat page shows the thinking in a box above the answer; the OpenAI-compatible API returns only the final answer.++## Not tested++- Fast mode on tasks other than the one above.+- Chats past 16K tokens, and Macs other than an M4 with 16 GB.diff --git a/docs/MODELS.md b/docs/MODELS.mdindex 1478450..5180e73 100644--- a/docs/MODELS.md+++ b/docs/MODELS.md@@ -9,6 +9,23 @@ What each model file does on a 16 GB Mac, measured the same way every time. Thes- **Speed:** the five coding prompts in [`bench/speed/prompts.json`](../bench/speed/prompts.json), each a fresh conversation capped at 300 tokens, run by [`bench/speed/run.py`](../bench/speed/run.py). "Writing" is answer tokens per second from the first token to the last, the same thing the chat shows under each answer; the table gives the mean over the five prompts, with the slowest and fastest in brackets.- **Quality:** 8 small coding tasks in [`bench/quality/tasks.json`](../bench/quality/tasks.json). Each answer's code is run against tests the model never sees ([`bench/quality/tests.py`](../bench/quality/tests.py)); a task passes only if every test does. Quality depends on the model file, not on Coachwhip's settings: the bank and the prefetch change how fast an answer comes, never what it says.+## gpt-oss-120b++OpenAI's open model (architecture `gpt-oss`): 36 layers, 128 experts of which 4 are used per word; it thinks before it answers. It runs from a **squeezed file** that `coachwhip-prepare` makes on your Mac from the official [MXFP4 GGUF](https://huggingface.co/ggml-org/gpt-oss-120b-GGUF) (63.4 GB): every expert matrix to Q2_K, rows padded to whole 256-weight blocks, the rest untouched; 42.6 GB written, about 106 GB free needed while both files exist. See the README, gpt-oss-120b.++Mac mini M4, 16 GB, `--parallel --flash-reads`, bank 25 (the Wrangler's pick), temperature 0. Measured on one coding task, a cancellable async task pool in TypeScript (about 4,000 words of thinking and code), against 20 hidden checks:++| Mode | Writing | Score of 5 | Second version of the task |+|---|---|---|---|+| Fast, `--stand-ins` | 9.8 tok/s | 4.0 | 4.0 at 10.0 tok/s |+| The model's own 4 experts | 4.2 tok/s | 4.8 | |++This model is measured on its own task, not the five-prompt speed set above: its thinking makes 300-token answers mostly thinking. Every setup we measured, and what makes it fast: [GPT-OSS.md](GPT-OSS.md).++- **Why the squeeze:** at MXFP4's 13.2 MB per expert the bank holds 15 experts a layer; at Q2_K's 8.7 MB it holds 25. On the task above the squeezed file scores the same as the original download.+- **Why fast mode changes answers:** a light pick that is not in the bank is replaced by the best expert that is. The first pick is always read for real, and the second whenever it carries at least 0.3 of the weight.+- Tokenizer: [tokenizer.json](https://huggingface.co/openai/gpt-oss-120b) from the model's own page.+## Qwen3.5-122B-A10BThe biggest model Coachwhip runs on 16 GB (architecture `qwen35moe`): 48 layers, 256 experts of which 8 are used per word. It runs from a **squeezed file** that `coachwhip-prepare` makes on your Mac from the publisher's Q3_K_M (59 GB): each expert's gate and up matrices to Q2_K on the GPU, its down matrix to Q3_K, the rest untouched; 45 GB written in 5 to 9 minutes on an M4, and about 105 GB free needed while both files exist. See the README, The 122B.diff --git a/docs/releases/0.5.0.md b/docs/releases/0.5.0.mdnew file mode 100644index 0000000..ff56981--- /dev/null+++ b/docs/releases/0.5.0.md@@ -0,0 +1,26 @@+# Coachwhip 0.5.0++Mac mini M4, 16 GB, measured; your Mac and your prompt will move the numbers.++## New model: gpt-oss-120b++- **OpenAI's gpt-oss-120b on a 16 GB Mac** (`gpt-oss`): 117 billion parameters, 4 of its 128 experts per word in each of 36 layers. It thinks before it answers, and the chat page shows the thinking.+- **Squeezed once on your own Mac** from the official MXFP4 file: `coachwhip-prepare` takes every expert matrix to Q2_K, rows padded to whole 256-weight blocks; everything else is copied as it is. 63.4 GB in, 42.6 GB out; about 106 GB free is needed while both exist. Nothing is downloaded from us.+- **Two modes.** The model's own 4 experts per word, every one read for real: about 4.2 tok/s. Fast mode, `--stand-ins`: about 10 tok/s. Both measured on the same coding task (a cancellable async task pool in TypeScript, about 4,000 words of thinking and code, temperature 0): the first scored 4.8 of 5 on our 20 hidden checks, fast mode 4.0 on two versions of the task.+- **Its own chat format.** gpt-oss speaks OpenAI's harmony format; the chat page and the API speak it to the model, and the API returns only the final answer.++## Faster++- **Stand-ins, `--stand-ins`.** When a light pick is not in the bank, Coachwhip uses the best expert that is, by the router's own scores, instead of reading one from the SSD. The first pick is always read for real, and the second whenever the router gives it at least 0.3 of the weight. Fewer reads per word; the answers change, so check them for your use.+- **Flash reads, `--flash-reads`.** Experts are read straight from the SSD, past macOS's page cache, which has nothing to give back on a 16 GB Mac. The answers do not change.++## Structure++- `crates/engine/src/model_gptoss.rs`: the gpt-oss family (attention sinks, a 128-word sliding window on every other layer, its clamped expert activation with biases). The expert maths is now a small hook per family, so the Qwen families are unchanged.+- Vendored `candle-core` with an MXFP4 kernel for Metal; its tests are left out.++## Not tested++- gpt-oss beyond the chat's 16K tokens, and fast mode on tasks other than the one above.+- Macs other than an M4 with 16 GB on macOS 26.+- `--stand-ins` on the Qwen models; it is meant for gpt-oss.diff --git a/vendor/candle-core/Cargo.toml b/vendor/candle-core/Cargo.tomlnew file mode 100644index 0000000..8972933--- /dev/null+++ b/vendor/candle-core/Cargo.toml@@ -0,0 +1,47 @@+[package]+name = "candle-core"+version = "0.11.0"+edition = "2021"+description = "Minimalist ML framework. Vendored at rev 5ba5d5b for Coachwhip with MXFP4 (candle PR #3899)."+repository = "https://github.com/huggingface/candle"+keywords = ["blas", "tensor", "machine-learning"]+categories = ["science"]+license = "MIT OR Apache-2.0"+readme = "README.md"+autobenches = false+autoexamples = false++[dependencies]+byteorder = "1.4.3"+candle-metal-kernels = { path = "../candle-metal-kernels", version = "0.11.0", optional = true }+objc2-metal = { version = "0.3.1", optional = true }+objc2-foundation = { version = "0.3.1", optional = true }+gemm = { version = "0.19.0", features = ["wasm-simd128-enable"] }+half = { version = "2.5.0", features = ["num-traits", "use-intrinsics", "rand_distr", "zerocopy"] }+float8 = { version = "0.7.0", features = ["num-traits", "rand_distr"] }+libc = { version = "0.2.147" }+libm = { version = "0.2.15" }+memmap2 = { version = "0.9.3", features = ["stable_deref_trait"] }+num-traits = "0.2.15"+num_cpus = "1.15.0"+rand = "0.9.0"+rand_distr = "0.5.1"+rayon = "1.7.0"+safetensors = "0.8.0"+thiserror = "2"+yoke = { version = "0.8.1", features = ["derive"] }+zerocopy = "0.8"+zip = { version = "8.6.0", default-features = false }+parking_lot = "0.12.5"++[target.'cfg(not(target_arch = "wasm32"))'.dependencies]+tokenizers = { version = "0.23.1", default-features = false, features = ["fancy-regex"] }++[features]+default = []+metal = [+ "dep:objc2-metal",+ "dep:objc2-foundation",+ "dep:candle-metal-kernels",+]+metal-debug-labels = ["metal", "candle-metal-kernels/debug-labels"]diff --git a/vendor/candle-core/LICENSE-APACHE b/vendor/candle-core/LICENSE-APACHEnew file mode 100644index 0000000..261eeb9--- /dev/null+++ b/vendor/candle-core/LICENSE-APACHE@@ -0,0 +1,201 @@+ Apache License+ Version 2.0, January 2004+ http://www.apache.org/licenses/++ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION++ 1. Definitions.++ "License" shall mean the terms and conditions for use, reproduction,+ and distribution as defined by Sections 1 through 9 of this document.++ "Licensor" shall mean the copyright owner or entity authorized by+ the copyright owner that is granting the License.++ "Legal Entity" shall mean the union of the acting entity and all+ other entities that control, are controlled by, or are under common+ control with that entity. For the purposes of this definition,+ "control" means (i) the power, direct or indirect, to cause the+ direction or management of such entity, whether by contract or+ otherwise, or (ii) ownership of fifty percent (50%) or more of the+ outstanding shares, or (iii) beneficial ownership of such entity.++ "You" (or "Your") shall mean an individual or Legal Entity+ exercising permissions granted by this License.++ "Source" form shall mean the preferred form for making modifications,+ including but not limited to software source code, documentation+ source, and configuration files.++ "Object" form shall mean any form resulting from mechanical+ transformation or translation of a Source form, including but+ not limited to compiled object code, generated documentation,+ and conversions to other media types.++ "Work" shall mean the work of authorship, whether in Source or+ Object form, made available under the License, as indicated by a+ copyright notice that is included in or attached to the work+ (an example is provided in the Appendix below).++ "Derivative Works" shall mean any work, whether in Source or Object+ form, that is based on (or derived from) the Work and for which the+ editorial revisions, annotations, elaborations, or other modifications+ represent, as a whole, an original work of authorship. For the purposes+ of this License, Derivative Works shall not include works that remain+ separable from, or merely link (or bind by name) to the interfaces of,+ the Work and Derivative Works thereof.++ "Contribution" shall mean any work of authorship, including+ the original version of the Work and any modifications or additions+ to that Work or Derivative Works thereof, that is intentionally+ submitted to Licensor for inclusion in the Work by the copyright owner+ or by an individual or Legal Entity authorized to submit on behalf of+ the copyright owner. For the purposes of this definition, "submitted"+ means any form of electronic, verbal, or written communication sent+ to the Licensor or its representatives, including but not limited to+ communication on electronic mailing lists, source code control systems,+ and issue tracking systems that are managed by, or on behalf of, the+ Licensor for the purpose of discussing and improving the Work, but+ excluding communication that is conspicuously marked or otherwise+ designated in writing by the copyright owner as "Not a Contribution."++ "Contributor" shall mean Licensor and any individual or Legal Entity+ on behalf of whom a Contribution has been received by Licensor and+ subsequently incorporated within the Work.++ 2. Grant of Copyright License. Subject to the terms and conditions of+ this License, each Contributor hereby grants to You a perpetual,+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable+ copyright license to reproduce, prepare Derivative Works of,+ publicly display, publicly perform, sublicense, and distribute the+ Work and such Derivative Works in Source or Object form.++ 3. Grant of Patent License. Subject to the terms and conditions of+ this License, each Contributor hereby grants to You a perpetual,+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable+ (except as stated in this section) patent license to make, have made,+ use, offer to sell, sell, import, and otherwise transfer the Work,+ where such license applies only to those patent claims licensable+ by such Contributor that are necessarily infringed by their+ Contribution(s) alone or by combination of their Contribution(s)+ with the Work to which such Contribution(s) was submitted. If You+ institute patent litigation against any entity (including a+ cross-claim or counterclaim in a lawsuit) alleging that the Work+ or a Contribution incorporated within the Work constitutes direct+ or contributory patent infringement, then any patent licenses+ granted to You under this License for that Work shall terminate+ as of the date such litigation is filed.++ 4. Redistribution. You may reproduce and distribute copies of the+ Work or Derivative Works thereof in any medium, with or without+ modifications, and in Source or Object form, provided that You+ meet the following conditions:++ (a) You must give any other recipients of the Work or+ Derivative Works a copy of this License; and++ (b) You must cause any modified files to carry prominent notices+ stating that You changed the files; and++ (c) You must retain, in the Source form of any Derivative Works+ that You distribute, all copyright, patent, trademark, and+ attribution notices from the Source form of the Work,+ excluding those notices that do not pertain to any part of+ the Derivative Works; and++ (d) If the Work includes a "NOTICE" text file as part of its+ distribution, then any Derivative Works that You distribute must+ include a readable copy of the attribution notices contained+ within such NOTICE file, excluding those notices that do not+ pertain to any part of the Derivative Works, in at least one+ of the following places: within a NOTICE text file distributed+ as part of the Derivative Works; within the Source form or+ documentation, if provided along with the Derivative Works; or,+ within a display generated by the Derivative Works, if and+ wherever such third-party notices normally appear. The contents+ of the NOTICE file are for informational purposes only and+ do not modify the License. You may add Your own attribution+ notices within Derivative Works that You distribute, alongside+ or as an addendum to the NOTICE text from the Work, provided+ that such additional attribution notices cannot be construed+ as modifying the License.++ You may add Your own copyright statement to Your modifications and+ may provide additional or different license terms and conditions+ for use, reproduction, or distribution of Your modifications, or+ for any such Derivative Works as a whole, provided Your use,+ reproduction, and distribution of the Work otherwise complies with+ the conditions stated in this License.++ 5. Submission of Contributions. Unless You explicitly state otherwise,+ any Contribution intentionally submitted for inclusion in the Work+ by You to the Licensor shall be under the terms and conditions of+ this License, without any additional terms or conditions.+ Notwithstanding the above, nothing herein shall supersede or modify+ the terms of any separate license agreement you may have executed+ with Licensor regarding such Contributions.++ 6. Trademarks. This License does not grant permission to use the trade+ names, trademarks, service marks, or product names of the Licensor,+ except as required for reasonable and customary use in describing the+ origin of the Work and reproducing the content of the NOTICE file.++ 7. Disclaimer of Warranty. Unless required by applicable law or+ agreed to in writing, Licensor provides the Work (and each+ Contributor provides its Contributions) on an "AS IS" BASIS,+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or+ implied, including, without limitation, any warranties or conditions+ of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A+ PARTICULAR PURPOSE. You are solely responsible for determining the+ appropriateness of using or redistributing the Work and assume any+ risks associated with Your exercise of permissions under this License.++ 8. Limitation of Liability. In no event and under no legal theory,+ whether in tort (including negligence), contract, or otherwise,+ unless required by applicable law (such as deliberate and grossly+ negligent acts) or agreed to in writing, shall any Contributor be+ liable to You for damages, including any direct, indirect, special,+ incidental, or consequential damages of any character arising as a+ result of this License or out of the use or inability to use the+ Work (including but not limited to damages for loss of goodwill,+ work stoppage, computer failure or malfunction, or any and all+ other commercial damages or losses), even if such Contributor+ has been advised of the possibility of such damages.++ 9. Accepting Warranty or Additional Liability. While redistributing+ the Work or Derivative Works thereof, You may choose to offer,+ and charge a fee for, acceptance of support, warranty, indemnity,+ or other liability obligations and/or rights consistent with this+ License. However, in accepting such obligations, You may act only+ on Your own behalf and on Your sole responsibility, not on behalf+ of any other Contributor, and only if You agree to indemnify,+ defend, and hold each Contributor harmless for any liability+ incurred by, or claims asserted against, such Contributor by reason+ of your accepting any such warranty or additional liability.++ END OF TERMS AND CONDITIONS++ APPENDIX: How to apply the Apache License to your work.++ To apply the Apache License to your work, attach the following+ boilerplate notice, with the fields enclosed by brackets "[]"+ replaced with your own identifying information. (Don't include+ the brackets!) The text should be enclosed in the appropriate+ comment syntax for the file format. We also recommend that a+ file or class name and description of purpose be included on the+ same "printed page" as the copyright notice for easier+ identification within third-party archives.++ Copyright [yyyy] [name of copyright owner]++ Licensed under the Apache License, Version 2.0 (the "License");+ you may not use this file except in compliance with the License.+ You may obtain a copy of the License at++ http://www.apache.org/licenses/LICENSE-2.0++ Unless required by applicable law or agreed to in writing, software+ distributed under the License is distributed on an "AS IS" BASIS,+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+ See the License for the specific language governing permissions and+ limitations under the License.diff --git a/vendor/candle-core/LICENSE-MIT b/vendor/candle-core/LICENSE-MITnew file mode 100644index 0000000..31aa793--- /dev/null+++ b/vendor/candle-core/LICENSE-MIT@@ -0,0 +1,23 @@+Permission is hereby granted, free of charge, to any+person obtaining a copy of this software and associated+documentation files (the "Software"), to deal in the+Software without restriction, including without+limitation the rights to use, copy, modify, merge,+publish, distribute, sublicense, and/or sell copies of+the Software, and to permit persons to whom the Software+is furnished to do so, subject to the following+conditions:++The above copyright notice and this permission notice+shall be included in all copies or substantial portions+of the Software.++THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF+ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED+TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A+PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT+SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR+IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER+DEALINGS IN THE SOFTWARE.diff --git a/vendor/candle-core/README.md b/vendor/candle-core/README.mdnew file mode 100644index 0000000..246f5ba--- /dev/null+++ b/vendor/candle-core/README.md@@ -0,0 +1,2 @@+# candle+Minimalist ML framework for Rustdiff --git a/vendor/candle-core/src/accelerate.rs b/vendor/candle-core/src/accelerate.rsnew file mode 100644index 0000000..d371d3b--- /dev/null+++ b/vendor/candle-core/src/accelerate.rs@@ -0,0 +1,476 @@+#![allow(dead_code)]+use libc::{c_char, c_double, c_float, c_int, c_long, c_ulong};++mod ffi {+ use super::*;+ extern "C" {+ // It would be nice to be able to switch to the NEWLAPACK version of the function but this+ // seems to trigger some link error. Available function names can be seen here:+ // /Library/Developer/CommandLineTools/SDKs/MacOSX13.3.sdk/System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate.tbd+ #[link_name = "sgemm_"]+ pub fn sgemm_ffi(+ transa: *const c_char,+ transb: *const c_char,+ m: *const c_int,+ n: *const c_int,+ k: *const c_int,+ alpha: *const c_float,+ a: *const c_float,+ lda: *const c_int,+ b: *const c_float,+ ldb: *const c_int,+ beta: *const c_float,+ c: *mut c_float,+ ldc: *const c_int,+ );+ #[link_name = "dgemm_"]+ pub fn dgemm_ffi(+ transa: *const c_char,+ transb: *const c_char,+ m: *const c_int,+ n: *const c_int,+ k: *const c_int,+ alpha: *const c_double,+ a: *const c_double,+ lda: *const c_int,+ b: *const c_double,+ ldb: *const c_int,+ beta: *const c_double,+ c: *mut c_double,+ ldc: *const c_int,+ );++ pub fn vvexpf(dst: *mut c_float, src: *const c_float, len: *const c_int);+ pub fn vvexp(dst: *mut c_double, src: *const c_double, len: *const c_int);+ pub fn vvsqrtf(dst: *mut c_float, src: *const c_float, len: *const c_int);+ pub fn vvsqrt(dst: *mut c_double, src: *const c_double, len: *const c_int);+ pub fn vvsinf(dst: *mut c_float, src: *const c_float, len: *const c_int);+ pub fn vvsin(dst: *mut c_double, src: *const c_double, len: *const c_int);+ pub fn vvcosf(dst: *mut c_float, src: *const c_float, len: *const c_int);+ pub fn vvcos(dst: *mut c_double, src: *const c_double, len: *const c_int);+ pub fn vvlogf(dst: *mut c_float, src: *const c_float, len: *const c_int);+ pub fn vvlog(dst: *mut c_double, src: *const c_double, len: *const c_int);+ pub fn vvtanhf(dst: *mut c_float, src: *const c_float, len: *const c_int);+ pub fn vvtanh(dst: *mut c_double, src: *const c_double, len: *const c_int);++ pub fn vDSP_vaddD(+ _: *const c_double,+ _: c_long,+ _: *const c_double,+ _: c_long,+ _: *mut c_double,+ _: c_long,+ _: c_ulong,+ );+ pub fn vDSP_vadd(+ _: *const c_float,+ _: c_long,+ _: *const c_float,+ _: c_long,+ _: *mut c_float,+ _: c_long,+ _: c_ulong,+ );+ pub fn vDSP_vsubD(+ _: *const c_double,+ _: c_long,+ _: *const c_double,+ _: c_long,+ _: *mut c_double,+ _: c_long,+ _: c_ulong,+ );+ pub fn vDSP_vsub(+ _: *const c_float,+ _: c_long,+ _: *const c_float,+ _: c_long,+ _: *mut c_float,+ _: c_long,+ _: c_ulong,+ );+ pub fn vDSP_vmulD(+ _: *const c_double,+ _: c_long,+ _: *const c_double,+ _: c_long,+ _: *mut c_double,+ _: c_long,+ _: c_ulong,+ );+ pub fn vDSP_vmul(+ _: *const c_float,+ _: c_long,+ _: *const c_float,+ _: c_long,+ _: *mut c_float,+ _: c_long,+ _: c_ulong,+ );+ pub fn vDSP_vdivD(+ _: *const c_double,+ _: c_long,+ _: *const c_double,+ _: c_long,+ _: *mut c_double,+ _: c_long,+ _: c_ulong,+ );+ pub fn vDSP_vdiv(+ _: *const c_float,+ _: c_long,+ _: *const c_float,+ _: c_long,+ _: *mut c_float,+ _: c_long,+ _: c_ulong,+ );+ pub fn vDSP_vminD(+ _: *const c_double,+ _: c_long,+ _: *const c_double,+ _: c_long,+ _: *mut c_double,+ _: c_long,+ _: c_ulong,+ );+ pub fn vDSP_vmin(+ _: *const c_float,+ _: c_long,+ _: *const c_float,+ _: c_long,+ _: *mut c_float,+ _: c_long,+ _: c_ulong,+ );+ pub fn vDSP_vmaxD(+ _: *const c_double,+ _: c_long,+ _: *const c_double,+ _: c_long,+ _: *mut c_double,+ _: c_long,+ _: c_ulong,+ );+ pub fn vDSP_vmax(+ _: *const c_float,+ _: c_long,+ _: *const c_float,+ _: c_long,+ _: *mut c_float,+ _: c_long,+ _: c_ulong,+ );+ }+}++#[allow(clippy::too_many_arguments)]+#[inline]+pub unsafe fn sgemm(+ transa: u8,+ transb: u8,+ m: i32,+ n: i32,+ k: i32,+ alpha: f32,+ a: &[f32],+ lda: i32,+ b: &[f32],+ ldb: i32,+ beta: f32,+ c: &mut [f32],+ ldc: i32,+) {+ ffi::sgemm_ffi(+ &(transa as c_char),+ &(transb as c_char),+ &m,+ &n,+ &k,+ &alpha,+ a.as_ptr(),+ &lda,+ b.as_ptr(),+ &ldb,+ &beta,+ c.as_mut_ptr(),+ &ldc,+ )+}++#[allow(clippy::too_many_arguments)]+#[inline]+pub unsafe fn dgemm(+ transa: u8,+ transb: u8,+ m: i32,+ n: i32,+ k: i32,+ alpha: f64,+ a: &[f64],+ lda: i32,+ b: &[f64],+ ldb: i32,+ beta: f64,+ c: &mut [f64],+ ldc: i32,+) {+ ffi::dgemm_ffi(+ &(transa as c_char),+ &(transb as c_char),+ &m,+ &n,+ &k,+ &alpha,+ a.as_ptr(),+ &lda,+ b.as_ptr(),+ &ldb,+ &beta,+ c.as_mut_ptr(),+ &ldc,+ )+}++#[inline]+pub fn vs_exp(a: &[f32], y: &mut [f32]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vvexpf(y.as_mut_ptr(), a.as_ptr(), &(a_len as i32)) }+}++#[inline]+pub fn vd_exp(a: &[f64], y: &mut [f64]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vvexp(y.as_mut_ptr(), a.as_ptr(), &(a_len as i32)) }+}++#[inline]+pub fn vs_sqrt(a: &[f32], y: &mut [f32]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vvsqrtf(y.as_mut_ptr(), a.as_ptr(), &(a_len as i32)) }+}++#[inline]+pub fn vd_sqrt(a: &[f64], y: &mut [f64]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vvsqrt(y.as_mut_ptr(), a.as_ptr(), &(a_len as i32)) }+}++#[inline]+pub fn vs_sin(a: &[f32], y: &mut [f32]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vvsinf(y.as_mut_ptr(), a.as_ptr(), &(a_len as i32)) }+}++#[inline]+pub fn vd_sin(a: &[f64], y: &mut [f64]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vvsin(y.as_mut_ptr(), a.as_ptr(), &(a_len as i32)) }+}+#[inline]+pub fn vs_cos(a: &[f32], y: &mut [f32]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vvcosf(y.as_mut_ptr(), a.as_ptr(), &(a_len as i32)) }+}++#[inline]+pub fn vd_cos(a: &[f64], y: &mut [f64]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vvcos(y.as_mut_ptr(), a.as_ptr(), &(a_len as i32)) }+}+#[inline]+pub fn vs_tanh(a: &[f32], y: &mut [f32]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vvtanhf(y.as_mut_ptr(), a.as_ptr(), &(a_len as i32)) }+}++#[inline]+pub fn vd_tanh(a: &[f64], y: &mut [f64]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vvtanh(y.as_mut_ptr(), a.as_ptr(), &(a_len as i32)) }+}++#[inline]+pub fn vs_ln(a: &[f32], y: &mut [f32]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vvlogf(y.as_mut_ptr(), a.as_ptr(), &(a_len as i32)) }+}++#[inline]+pub fn vd_ln(a: &[f64], y: &mut [f64]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vvlog(y.as_mut_ptr(), a.as_ptr(), &(a_len as i32)) }+}++#[inline]+pub fn vs_sqr(a: &[f32], y: &mut [f32]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ y.iter_mut().zip(a.iter()).for_each(|(y, a)| *y = *a * *a)+}++#[inline]+pub fn vd_sqr(a: &[f64], y: &mut [f64]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ y.iter_mut().zip(a.iter()).for_each(|(y, a)| *y = *a * *a)+}++#[inline]+pub fn vs_tanh_inplace(y: &mut [f32]) {+ unsafe { ffi::vvtanhf(y.as_mut_ptr(), y.as_ptr(), &(y.len() as i32)) }+}++#[inline]+pub fn vd_tanh_inplace(y: &mut [f64]) {+ unsafe { ffi::vvtanh(y.as_mut_ptr(), y.as_ptr(), &(y.len() as i32)) }+}++#[inline]+pub fn vs_exp_inplace(y: &mut [f32]) {+ unsafe { ffi::vvexpf(y.as_mut_ptr(), y.as_ptr(), &(y.len() as i32)) }+}++#[inline]+pub fn vd_exp_inplace(y: &mut [f64]) {+ unsafe { ffi::vvexp(y.as_mut_ptr(), y.as_ptr(), &(y.len() as i32)) }+}++#[inline]+pub fn vs_gelu(vs: &[f32], ys: &mut [f32]) {+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = (2.0f32 / std::f32::consts::PI).sqrt() * v * (1.0 + 0.044715 * v * v)+ }+ vs_tanh_inplace(ys);+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = 0.5 * v * (1.0 + *y)+ }+}++#[inline]+pub fn vd_gelu(vs: &[f64], ys: &mut [f64]) {+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = (2.0f64 / std::f64::consts::PI).sqrt() * v * (1.0 + 0.044715 * v * v)+ }+ vd_tanh_inplace(ys);+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = 0.5 * v * (1.0 + *y)+ }+}++#[inline]+pub fn vs_silu(vs: &[f32], ys: &mut [f32]) {+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = -v+ }+ vs_exp_inplace(ys);+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = v / (1.0 + *y)+ }+}++#[inline]+pub fn vd_silu(vs: &[f64], ys: &mut [f64]) {+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = -v+ }+ vd_exp_inplace(ys);+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = v / (1.0 + *y)+ }+}++macro_rules! binary_op {+ ($fn_name:ident, $ty:ty, $accelerate_name:ident) => {+ #[inline]+ pub fn $fn_name(a: &[$ty], b: &[$ty], y: &mut [$ty]) {+ let a_len = a.len();+ let b_len = b.len();+ let y_len = y.len();+ if a_len != y_len || b_len != y_len {+ panic!(+ "{} a,b,y len mismatch {a_len} {b_len} {y_len}",+ stringify!($fn_name)+ );+ }+ unsafe {+ // Weird quirk of accelerate, the rhs comes before the lhs.+ ffi::$accelerate_name(+ b.as_ptr(),+ 1,+ a.as_ptr(),+ 1,+ y.as_mut_ptr(),+ 1,+ a_len as u64,+ )+ }+ }+ };+}+binary_op!(vs_add, f32, vDSP_vadd);+binary_op!(vd_add, f64, vDSP_vaddD);+binary_op!(vs_sub, f32, vDSP_vsub);+binary_op!(vd_sub, f64, vDSP_vsubD);+binary_op!(vs_mul, f32, vDSP_vmul);+binary_op!(vd_mul, f64, vDSP_vmulD);+binary_op!(vs_div, f32, vDSP_vdiv);+binary_op!(vd_div, f64, vDSP_vdivD);+binary_op!(vs_max, f32, vDSP_vmax);+binary_op!(vd_max, f64, vDSP_vmaxD);+binary_op!(vs_min, f32, vDSP_vmin);+binary_op!(vd_min, f64, vDSP_vminD);diff --git a/vendor/candle-core/src/backend.rs b/vendor/candle-core/src/backend.rsnew file mode 100644index 0000000..d8ab2b5--- /dev/null+++ b/vendor/candle-core/src/backend.rs@@ -0,0 +1,174 @@+//! Traits to Define Backend Behavior+//!+use crate::op::{BinaryOpT, CmpOp, ReduceOp, UnaryOpT};+use crate::{CpuStorage, DType, Layout, Result, Shape};++pub trait BackendStorage: Sized {+ type Device: BackendDevice;++ fn try_clone(&self, _: &Layout) -> Result<Self>;++ fn dtype(&self) -> DType;++ fn device(&self) -> &Self::Device;++ // Maybe this should return a Cow instead so that no copy is done on the cpu case.+ fn to_cpu_storage(&self) -> Result<CpuStorage>;++ fn affine(&self, _: &Layout, _: f64, _: f64) -> Result<Self>;++ fn powf(&self, _: &Layout, _: f64) -> Result<Self>;++ fn elu(&self, _: &Layout, _: f64) -> Result<Self>;++ fn reduce_op(&self, _: ReduceOp, _: &Layout, _: &[usize]) -> Result<Self>;++ fn cmp(&self, _: CmpOp, _: &Self, _: &Layout, _: &Layout) -> Result<Self>;++ fn to_dtype(&self, _: &Layout, _: DType) -> Result<Self>;++ fn unary_impl<B: UnaryOpT>(&self, _: &Layout) -> Result<Self>;++ fn binary_impl<B: BinaryOpT>(&self, _: &Self, _: &Layout, _: &Layout) -> Result<Self>;++ fn where_cond(&self, _: &Layout, _: &Self, _: &Layout, _: &Self, _: &Layout) -> Result<Self>;++ fn conv1d(+ &self,+ _l: &Layout,+ _kernel: &Self,+ _kernel_l: &Layout,+ _params: &crate::conv::ParamsConv1D,+ ) -> Result<Self>;++ fn conv_transpose1d(+ &self,+ _l: &Layout,+ _kernel: &Self,+ _kernel_l: &Layout,+ _params: &crate::conv::ParamsConvTranspose1D,+ ) -> Result<Self>;++ fn conv2d(+ &self,+ _l: &Layout,+ _kernel: &Self,+ _kernel_l: &Layout,+ _params: &crate::conv::ParamsConv2D,+ ) -> Result<Self>;++ fn conv_transpose2d(+ &self,+ _l: &Layout,+ _kernel: &Self,+ _kernel_l: &Layout,+ _params: &crate::conv::ParamsConvTranspose2D,+ ) -> Result<Self>;++ fn avg_pool2d(&self, _: &Layout, _: (usize, usize), _: (usize, usize)) -> Result<Self>;+ fn max_pool2d(&self, _: &Layout, _: (usize, usize), _: (usize, usize)) -> Result<Self>;+ fn upsample_nearest1d(&self, _: &Layout, _: usize) -> Result<Self>;+ fn upsample_nearest2d(&self, _: &Layout, _: usize, _: usize) -> Result<Self>;+ fn upsample_bilinear2d(+ &self,+ _: &Layout,+ _: usize,+ _: usize,+ _: bool,+ _: Option<f64>,+ _: Option<f64>,+ ) -> Result<Self>;++ fn gather(&self, _: &Layout, _: &Self, _: &Layout, _: usize) -> Result<Self>;++ fn scatter_set(+ &mut self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: usize,+ ) -> Result<()>;++ fn scatter_add_set(+ &mut self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: usize,+ ) -> Result<()>;++ fn index_select(&self, _: &Self, _: &Layout, _: &Layout, _: usize) -> Result<Self>;+ fn index_add(+ &self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: usize,+ ) -> Result<Self>;++ fn matmul(+ &self,+ _: &Self,+ _: (usize, usize, usize, usize),+ _: &Layout,+ _: &Layout,+ ) -> Result<Self>;++ fn copy_strided_src(&self, _: &mut Self, _: usize, _: &Layout) -> Result<()>;++ #[allow(clippy::too_many_arguments)]+ // Similar to cudaMemcpy2D, though values are in elements and not in bytes.+ fn copy2d(+ &self,+ _: &mut Self,+ _d1: usize,+ _d2: usize,+ _src_stride1: usize,+ _dst_stride1: usize,+ _src_offset: usize,+ _dst_offset: usize,+ ) -> Result<()>;++ fn const_set(&mut self, _: crate::scalar::Scalar, _: &Layout) -> Result<()>;+}++pub trait BackendDevice: Sized + std::fmt::Debug + Clone {+ type Storage: BackendStorage;++ // TODO: Make the usize generic and part of a generic DeviceLocation.+ fn new(_: usize) -> Result<Self>;++ fn location(&self) -> crate::DeviceLocation;++ fn same_device(&self, _: &Self) -> bool;++ fn zeros_impl(&self, _shape: &Shape, _dtype: DType) -> Result<Self::Storage>;++ /// # Safety+ /// This function is unsafe as it doesn't initialize the underlying data store.+ /// The caller should ensure that the data is properly initialized as early as possible+ /// after this call.+ unsafe fn alloc_uninit(&self, _shape: &Shape, _dtype: DType) -> Result<Self::Storage>;++ fn storage_from_slice<T: crate::WithDType>(&self, _: &[T]) -> Result<Self::Storage>;++ fn storage_from_cpu_storage(&self, _: &CpuStorage) -> Result<Self::Storage>;++ fn storage_from_cpu_storage_owned(&self, _: CpuStorage) -> Result<Self::Storage>;++ fn rand_uniform(&self, _: &Shape, _: DType, _: f64, _: f64) -> Result<Self::Storage>;++ fn rand_normal(&self, _: &Shape, _: DType, _: f64, _: f64) -> Result<Self::Storage>;++ fn set_seed(&self, _: u64) -> Result<()>;+ fn get_current_seed(&self) -> Result<u64>;++ /// Synchronize should block until all the operations on the device are completed.+ fn synchronize(&self) -> Result<()>;+}diff --git a/vendor/candle-core/src/backprop.rs b/vendor/candle-core/src/backprop.rsnew file mode 100644index 0000000..7dc4114--- /dev/null+++ b/vendor/candle-core/src/backprop.rs@@ -0,0 +1,811 @@+//! Methods for backpropagation of gradients.+use crate::op::{BinaryOp, Op, ReduceOp, UnaryOp};+use crate::{Error, Result, Tensor, TensorId};+use std::collections::{hash_map::Entry, HashMap};++// arg has been reduced to node via reduce_dims, expand it back to arg.+// This has to handle keepdims.+fn broadcast_back(arg: &Tensor, node: &Tensor, reduced_dims: &[usize]) -> Result<Tensor> {+ if arg.rank() == node.rank() {+ // keepdim = true+ node.broadcast_as(arg.shape())+ } else {+ // keepdim = false+ // first expand the reduced dims.+ node.reshape(reduced_dims)?.broadcast_as(arg.shape())+ }+}++thread_local! {+ static CANDLE_GRAD_DO_NOT_DETACH: bool = {+ match std::env::var("CANDLE_GRAD_DO_NOT_DETACH") {+ Ok(s) => {+ !s.is_empty() && s != "0"+ },+ Err(_) => false,+ }+ }+}++impl Tensor {+ /// Return all the nodes that lead to this value in a topologically sorted vec, the first+ /// elements having dependencies on the latter ones, e.g. the first element if any is the+ /// argument.+ /// This assumes that the op graph is a DAG.+ pub fn sorted_nodes(&self) -> Vec<&Tensor> {+ // The vec of sorted nodes is passed as an owned value rather than a mutable reference+ // to get around some lifetime limitations.+ fn walk<'a>(+ node: &'a Tensor,+ nodes: Vec<&'a Tensor>,+ already_seen: &mut HashMap<TensorId, bool>,+ ) -> (bool, Vec<&'a Tensor>) {+ if let Some(&tg) = already_seen.get(&node.id()) {+ return (tg, nodes);+ }+ let mut track_grad = false;+ let mut nodes = if node.is_variable() {+ // Do not call recursively on the "leaf" nodes.+ track_grad = true;+ nodes+ } else if node.dtype().is_int() {+ nodes+ } else if let Some(op) = node.op() {+ match op {+ Op::IndexAdd(t1, t2, t3, _)+ | Op::Scatter(t1, t2, t3, _)+ | Op::ScatterAdd(t1, t2, t3, _)+ | Op::CustomOp3(t1, t2, t3, _)+ | Op::WhereCond(t1, t2, t3) => {+ let (tg, nodes) = walk(t1, nodes, already_seen);+ track_grad |= tg;+ let (tg, nodes) = walk(t2, nodes, already_seen);+ track_grad |= tg;+ let (tg, nodes) = walk(t3, nodes, already_seen);+ track_grad |= tg;+ nodes+ }+ Op::Conv1D {+ arg: lhs,+ kernel: rhs,+ ..+ }+ | Op::ConvTranspose1D {+ arg: lhs,+ kernel: rhs,+ ..+ }+ | Op::Conv2D {+ arg: lhs,+ kernel: rhs,+ ..+ }+ | Op::ConvTranspose2D {+ arg: lhs,+ kernel: rhs,+ ..+ }+ | Op::CustomOp2(lhs, rhs, _)+ | Op::Binary(lhs, rhs, _)+ | Op::Gather(lhs, rhs, _)+ | Op::IndexSelect(lhs, rhs, _)+ | Op::Matmul(lhs, rhs)+ | Op::SliceScatter0(lhs, rhs, _) => {+ let (tg, nodes) = walk(lhs, nodes, already_seen);+ track_grad |= tg;+ let (tg, nodes) = walk(rhs, nodes, already_seen);+ track_grad |= tg;+ nodes+ }+ Op::Cat(args, _) => args.iter().fold(nodes, |nodes, arg| {+ let (tg, nodes) = walk(arg, nodes, already_seen);+ track_grad |= tg;+ nodes+ }),+ Op::Affine { arg, mul, .. } => {+ if *mul == 0. {+ nodes+ } else {+ let (tg, nodes) = walk(arg, nodes, already_seen);+ track_grad |= tg;+ nodes+ }+ }+ Op::Unary(_node, UnaryOp::Ceil)+ | Op::Unary(_node, UnaryOp::Floor)+ | Op::Unary(_node, UnaryOp::Round)+ | Op::Unary(_node, UnaryOp::Sign) => nodes,+ Op::Reshape(node)+ | Op::UpsampleNearest1D { arg: node, .. }+ | Op::UpsampleNearest2D { arg: node, .. }+ | Op::UpsampleBilinear2D { arg: node, .. }+ | Op::AvgPool2D { arg: node, .. }+ | Op::MaxPool2D { arg: node, .. }+ | Op::Copy(node)+ | Op::Broadcast(node)+ | Op::Cmp(node, _)+ | Op::Reduce(node, ReduceOp::Min | ReduceOp::Sum | ReduceOp::Max, _)+ | Op::ToDevice(node)+ | Op::Transpose(node, _, _)+ | Op::Permute(node, _)+ | Op::Narrow(node, _, _, _)+ | Op::Unary(node, _)+ | Op::Elu(node, _)+ | Op::Powf(node, _)+ | Op::CustomOp1(node, _) => {+ let (tg, nodes) = walk(node, nodes, already_seen);+ track_grad |= tg;+ nodes+ }+ Op::ToDType(node) => {+ if node.dtype().is_float() {+ let (tg, nodes) = walk(node, nodes, already_seen);+ track_grad |= tg;+ nodes+ } else {+ nodes+ }+ }+ Op::Reduce(_, ReduceOp::ArgMin | ReduceOp::ArgMax, _) => nodes,+ }+ } else {+ nodes+ };+ already_seen.insert(node.id(), track_grad);+ if track_grad {+ nodes.push(node);+ }+ (track_grad, nodes)+ }+ let (_tg, mut nodes) = walk(self, vec![], &mut HashMap::new());+ nodes.reverse();+ nodes+ }++ pub fn backward(&self) -> Result<GradStore> {+ let sorted_nodes = self.sorted_nodes();+ let mut grads = GradStore::new();+ grads.insert(self, self.ones_like()?.contiguous()?);+ for node in sorted_nodes.iter() {+ if node.is_variable() {+ continue;+ }+ let grad = grads+ .remove(node)+ .expect("candle internal error - grad not populated");+ // https://github.com/huggingface/candle/issues/1241+ // Ideally, we would make these operations in place where possible to ensure that we+ // do not have to allocate too often. Here we just call `.detach` to avoid computing+ // the backprop graph of the backprop itself. This would be an issue for second order+ // derivatives but these are out of scope at the moment.+ let do_not_detach = CANDLE_GRAD_DO_NOT_DETACH.with(|b| *b);+ let grad = if do_not_detach { grad } else { grad.detach() };+ if let Some(op) = node.op() {+ match op {+ Op::Binary(lhs, rhs, BinaryOp::Add) => {+ let lhs_sum_grad = grads.or_insert(lhs)?;+ *lhs_sum_grad = lhs_sum_grad.add(&grad)?;+ let rhs_sum_grad = grads.or_insert(rhs)?;+ *rhs_sum_grad = rhs_sum_grad.add(&grad)?;+ }+ Op::Binary(lhs, rhs, BinaryOp::Sub) => {+ let lhs_sum_grad = grads.or_insert(lhs)?;+ *lhs_sum_grad = lhs_sum_grad.add(&grad)?;+ let rhs_sum_grad = grads.or_insert(rhs)?;+ *rhs_sum_grad = rhs_sum_grad.sub(&grad)?;+ }+ Op::Binary(lhs, rhs, BinaryOp::Mul) => {+ let lhs_grad = grad.mul(rhs)?;+ let lhs_sum_grad = grads.or_insert(lhs)?;+ *lhs_sum_grad = lhs_sum_grad.add(&lhs_grad)?;+ let rhs_grad = grad.mul(lhs)?;+ let rhs_sum_grad = grads.or_insert(rhs)?;+ *rhs_sum_grad = rhs_sum_grad.add(&rhs_grad)?;+ }+ Op::Binary(lhs, rhs, BinaryOp::Div) => {+ let lhs_grad = grad.div(rhs)?;+ let lhs_sum_grad = grads.or_insert(lhs)?;+ *lhs_sum_grad = lhs_sum_grad.add(&lhs_grad)?;+ let rhs_grad = grad.mul(lhs)?.div(&rhs.sqr()?)?;+ let rhs_sum_grad = grads.or_insert(rhs)?;+ *rhs_sum_grad = rhs_sum_grad.sub(&rhs_grad)?;+ }+ Op::Binary(lhs, rhs, BinaryOp::Minimum)+ | Op::Binary(lhs, rhs, BinaryOp::Maximum) => {+ let mask_lhs = node.eq(lhs)?.to_dtype(grad.dtype())?;+ let mask_rhs = node.eq(rhs)?.to_dtype(grad.dtype())?;++ // If both masks are 1 one the same point, we want to scale the+ // gradient by 0.5 rather than 1.+ let lhs_grad = mask_lhs.mul(&grad)?.div(&(&mask_rhs + 1.)?)?;+ let lhs_sum_grad = grads.or_insert(lhs)?;+ *lhs_sum_grad = lhs_sum_grad.add(&lhs_grad)?;++ let rhs_grad = mask_rhs.mul(&grad)?.div(&(&mask_lhs + 1.)?)?;+ let rhs_sum_grad = grads.or_insert(rhs)?;+ *rhs_sum_grad = rhs_sum_grad.add(&rhs_grad)?;+ }+ Op::WhereCond(pred, t, f) => {+ let zeros = grad.zeros_like()?;+ let t_sum_grad = grads.or_insert(t)?;+ let t_grad = pred.where_cond(&grad, &zeros)?;+ *t_sum_grad = t_sum_grad.add(&t_grad)?;+ let f_sum_grad = grads.or_insert(f)?;+ let f_grad = pred.where_cond(&zeros, &grad)?;+ *f_sum_grad = f_sum_grad.add(&f_grad)?;+ }+ Op::Conv1D {+ arg,+ kernel,+ padding,+ stride,+ dilation,+ } => {+ // The output height for conv_transpose1d is:+ // (l_in - 1) * stride - 2 * padding + dilation * (k_size - 1) + out_padding + 1+ let grad_l_in = grad.dim(2)?;+ let k_size = kernel.dim(2)?;+ let out_size =+ (grad_l_in - 1) * stride + dilation * (k_size - 1) + 1 - 2 * padding;+ let out_padding = arg.dim(2)? - out_size;+ let grad_arg = grad.conv_transpose1d(+ kernel,+ *padding,+ out_padding,+ *stride,+ *dilation,+ /* groups */ 1,+ )?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&grad_arg)?;++ let grad_kernel = arg+ .transpose(0, 1)?+ .conv1d(&grad.transpose(0, 1)?, *padding, *dilation, *stride, 1)?+ .transpose(0, 1)?;+ let sum_grad = grads.or_insert(kernel)?;+ let (_, _, k0) = kernel.dims3()?;+ let (_, _, g_k0) = grad_kernel.dims3()?;+ let grad_kernel = if g_k0 != k0 {+ grad_kernel.narrow(2, 0, k0)?+ } else {+ grad_kernel+ };+ *sum_grad = sum_grad.add(&grad_kernel)?;+ }+ Op::Conv2D {+ arg,+ kernel,+ padding,+ stride,+ dilation,+ } => {+ // The output height for conv_transpose2d is:+ // (i_h - 1) * stride - 2 * padding + dilation * (k_h - 1) + out_padding + 1+ let grad_h = grad.dim(2)?;+ let k_h = kernel.dim(2)?;+ let out_size =+ (grad_h - 1) * stride + dilation * (k_h - 1) + 1 - 2 * padding;+ let out_padding = arg.dim(2)? - out_size;+ let grad_arg = grad.conv_transpose2d(+ kernel,+ *padding,+ out_padding,+ *stride,+ *dilation,+ )?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&grad_arg)?;++ let grad_kernel = arg+ .transpose(0, 1)?+ .conv2d(&grad.transpose(0, 1)?, *padding, *dilation, *stride, 1)?+ .transpose(0, 1)?;+ let sum_grad = grads.or_insert(kernel)?;+ let (_, _, k0, k1) = kernel.dims4()?;+ let (_, _, g_k0, g_k1) = grad_kernel.dims4()?;+ let grad_kernel = if g_k0 != k0 || g_k1 != k1 {+ grad_kernel.narrow(2, 0, k0)?.narrow(3, 0, k1)?+ } else {+ grad_kernel+ };+ *sum_grad = sum_grad.add(&grad_kernel)?;+ }+ Op::ConvTranspose1D { .. } => Err(Error::BackwardNotSupported {+ op: "conv-transpose1d",+ })?,+ Op::ConvTranspose2D {+ arg,+ kernel,+ padding,+ stride,+ dilation,+ output_padding: _output_padding,+ } => {+ let grad_arg = grad.conv2d(kernel, *padding, *stride, *dilation, 1)?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&grad_arg)?;++ let grad_kernel = grad+ .transpose(0, 1)?+ .conv2d(&arg.transpose(0, 1)?, *padding, *dilation, *stride, 1)?+ .transpose(0, 1)?;+ let sum_grad = grads.or_insert(kernel)?;+ let (_, _, k0, k1) = kernel.dims4()?;+ let (_, _, g_k0, g_k1) = grad_kernel.dims4()?;+ let grad_kernel = if g_k0 != k0 || g_k1 != k1 {+ grad_kernel.narrow(2, 0, k0)?.narrow(3, 0, k1)?+ } else {+ grad_kernel+ };+ *sum_grad = sum_grad.add(&grad_kernel)?;+ }+ Op::AvgPool2D {+ arg,+ kernel_size,+ stride,+ } => {+ if kernel_size != stride {+ crate::bail!("backward not supported for avgpool2d if ksize {kernel_size:?} != stride {stride:?}")+ }+ let (_n, _c, h, w) = arg.dims4()?;+ let grad_arg = grad.upsample_nearest2d(h, w)?;+ let grad_arg =+ (grad_arg * (1f64 / (kernel_size.0 * kernel_size.1) as f64))?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&grad_arg)?;+ }+ Op::MaxPool2D {+ arg,+ kernel_size,+ stride,+ } => {+ if kernel_size != stride {+ crate::bail!("backward not supported for maxpool2d if ksize {kernel_size:?} != stride {stride:?}")+ }+ let (_n, _c, h, w) = arg.dims4()?;+ // For computing the max-pool gradient, we compute a mask where a 1 means+ // that the element is the maximum, then we apply this mask to the+ // upsampled gradient (taking into account that multiple max may exist so+ // we scale the gradient for this case).+ let node_upsampled = node.upsample_nearest2d(h, w)?;+ let mask = arg.eq(&node_upsampled)?.to_dtype(arg.dtype())?;+ let avg = mask.avg_pool2d_with_stride(*kernel_size, *stride)?;+ let grad_arg = ((grad * avg)?.upsample_nearest2d(h, w)? * mask)?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&grad_arg)?;+ }+ Op::UpsampleNearest1D { arg, target_size } => {+ let (_n, c, size) = arg.dims3()?;+ if target_size % size != 0 {+ crate::bail!("backward not supported for non integer upscaling factors")+ }+ let scale = target_size / size;++ let kernel = Tensor::ones((c, 1, scale), arg.dtype(), arg.device())?;+ let conv_sum = grad.conv1d(&kernel, 0, scale, 1, c)?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = conv_sum;+ }+ Op::UpsampleNearest2D {+ arg,+ target_h,+ target_w,+ } => {+ let (_n, c, h, w) = arg.dims4()?;+ if target_h % h != 0 || target_w % w != 0 {+ crate::bail!("backward not supported for non integer upscaling factors")+ }+ let scale_h = target_h / h;+ let scale_w = target_w / w;++ if scale_h != scale_w {+ crate::bail!("backward not supported for non uniform upscaling factors")+ };+ let kernel =+ Tensor::ones((c, 1, scale_h, scale_w), arg.dtype(), arg.device())?;+ let conv_sum = grad.conv2d(&kernel, 0, scale_h, 1, c)?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = conv_sum;+ }+ Op::UpsampleBilinear2D { .. } => {+ crate::bail!("backward not supported for upsample_bilinear2d")+ }+ Op::SliceScatter0(lhs, rhs, start_rhs) => {+ let rhs_sum_grad = grads.or_insert(rhs)?;+ let rhs_grad = grad.narrow(0, *start_rhs, rhs.dim(0)?)?;+ *rhs_sum_grad = rhs_sum_grad.add(&rhs_grad)?;++ let lhs_sum_grad = grads.or_insert(lhs)?;+ let lhs_grad = grad.slice_scatter0(&rhs.zeros_like()?, *start_rhs)?;+ *lhs_sum_grad = lhs_sum_grad.add(&lhs_grad)?+ }+ Op::Gather(arg, indexes, dim) => {+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.scatter_add(indexes, &grad, *dim)?;+ }+ Op::Scatter(init, indexes, src, dim) => {+ let init_sum_grad = grads.or_insert(init)?;+ *init_sum_grad = init_sum_grad.add(&grad)?;++ let src_grad = grad.gather(indexes, *dim)?;+ let src_sum_grad = grads.or_insert(src)?;+ *src_sum_grad = src_sum_grad.add(&src_grad)?;+ }+ Op::ScatterAdd(init, indexes, src, dim) => {+ let init_sum_grad = grads.or_insert(init)?;+ let mask = init.ones_like()?;+ let mask = mask.scatter(indexes, &mask.zeros_like()?, *dim)?;+ *init_sum_grad = init_sum_grad.add(&grad.mul(&mask)?)?;++ let src_grad = grad.gather(indexes, *dim)?;+ let src_sum_grad = grads.or_insert(src)?;+ *src_sum_grad = src_sum_grad.add(&src_grad)?;+ }+ Op::IndexAdd(init, indexes, src, dim) => {+ let init_sum_grad = grads.or_insert(init)?;+ *init_sum_grad = init_sum_grad.add(&grad)?;++ let src_grad = grad.index_select(indexes, *dim)?;+ let src_sum_grad = grads.or_insert(src)?;+ *src_sum_grad = src_sum_grad.add(&src_grad)?;+ }+ Op::IndexSelect(arg, indexes, dim) => {+ let sum_grad = grads.or_insert(arg)?;+ // index_select accepts strided ids but index_add requires contiguous ones.+ let indexes = indexes.contiguous()?;+ *sum_grad = sum_grad.index_add(&indexes, &grad, *dim)?;+ }+ Op::Matmul(lhs, rhs) => {+ // Skipping checks, the op went ok, we can skip+ // the matmul size checks for now.++ let lhs_grad = grad.matmul(&rhs.t()?)?;+ let lhs_sum_grad = grads.or_insert(lhs)?;+ *lhs_sum_grad = lhs_sum_grad.add(&lhs_grad)?;++ let rhs_grad = lhs.t()?.matmul(&grad)?;+ let rhs_sum_grad = grads.or_insert(rhs)?;+ *rhs_sum_grad = rhs_sum_grad.add(&rhs_grad)?;+ }+ Op::Cat(args, dim) => {+ let mut start_idx = 0;+ for arg in args {+ let len = arg.dims()[*dim];+ let arg_grad = grad.narrow(*dim, start_idx, len)?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&arg_grad)?;+ start_idx += len;+ }+ }+ Op::Broadcast(arg) => {+ let arg_dims = arg.dims();+ let node_dims = node.dims();+ // The number of dims that have been inserted on the left.+ let left_dims = node_dims.len() - arg_dims.len();+ let mut sum_dims: Vec<usize> = (0..left_dims).collect();+ for (dim, (node_dim, arg_dim)) in node_dims[left_dims..]+ .iter()+ .zip(arg_dims.iter())+ .enumerate()+ {+ if node_dim != arg_dim {+ sum_dims.push(dim + left_dims)+ }+ }++ let mut arg_grad = grad.sum_keepdim(sum_dims.as_slice())?;+ for _i in 0..left_dims {+ arg_grad = arg_grad.squeeze(0)?+ }+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&arg_grad.broadcast_as(sum_grad.dims())?)?;+ }+ Op::Reduce(arg, ReduceOp::Sum, reduced_dims) => {+ let grad = broadcast_back(arg, &grad, reduced_dims)?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&grad)?;+ }+ Op::Reduce(arg, ReduceOp::Max, reduced_dims) => {+ let node = broadcast_back(arg, node, reduced_dims)?;+ let grad = broadcast_back(arg, &grad, reduced_dims)?;+ let grad = node.eq(arg)?.to_dtype(grad.dtype())?.mul(&grad)?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&grad.broadcast_as(sum_grad.dims())?)?;+ }+ Op::Reduce(arg, ReduceOp::Min, reduced_dims) => {+ let node = broadcast_back(arg, node, reduced_dims)?;+ let grad = broadcast_back(arg, &grad, reduced_dims)?;+ let grad = node.eq(arg)?.to_dtype(grad.dtype())?.mul(&grad)?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&grad.broadcast_as(sum_grad.dims())?)?;+ }+ Op::ToDType(arg) => {+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&grad.to_dtype(arg.dtype())?)?+ }+ Op::Copy(arg) => {+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&grad)?+ }+ Op::Affine { arg, mul, .. } => {+ let arg_grad = grad.affine(*mul, 0.)?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&arg_grad)?+ }+ Op::Unary(arg, UnaryOp::Log) => {+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&(grad / arg)?)?+ }+ Op::Unary(arg, UnaryOp::Sin) => {+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&(&grad * arg.cos())?)?+ }+ Op::Unary(arg, UnaryOp::Cos) => {+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.sub(&(&grad * arg.sin())?)?+ }+ Op::Unary(arg, UnaryOp::Tanh) => {+ let sum_grad = grads.or_insert(arg)?;+ let minus_dtanh = (node.sqr()? - 1.)?;+ *sum_grad = sum_grad.sub(&(&grad * &minus_dtanh)?)?+ }+ Op::Unary(arg, UnaryOp::Abs) => {+ let sum_grad = grads.or_insert(arg)?;+ let ones = arg.ones_like()?;+ let abs_grad = arg.ge(&arg.zeros_like()?)?.where_cond(&ones, &ones.neg()?);+ *sum_grad = sum_grad.add(&(&grad * abs_grad)?)?+ }+ Op::Unary(arg, UnaryOp::Exp) => {+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&(&grad * *node)?)?+ }+ Op::Unary(arg, UnaryOp::Neg) => {+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.sub(&grad)?+ }+ Op::Unary(arg, UnaryOp::Recip) => {+ let sum_grad = grads.or_insert(arg)?;+ let grad = (grad / arg.sqr()?)?;+ *sum_grad = sum_grad.sub(&grad)?+ }+ &Op::Narrow(ref arg, dim, start_idx, len) => {+ let arg_dims = arg.dims();+ let left_pad = if start_idx == 0 {+ None+ } else {+ let mut dims = arg_dims.to_vec();+ dims[dim] = start_idx;+ Some(Tensor::zeros(dims, grad.dtype(), grad.device())?)+ };+ let right_pad = arg_dims[dim] - start_idx - len;+ let right_pad = if right_pad == 0 {+ None+ } else {+ let mut dims = arg_dims.to_vec();+ dims[dim] = right_pad;+ Some(Tensor::zeros(dims, grad.dtype(), grad.device())?)+ };+ let arg_grad = match (left_pad, right_pad) {+ (None, None) => grad,+ (Some(l), None) => Tensor::cat(&[&l, &grad], dim)?,+ (None, Some(r)) => Tensor::cat(&[&grad, &r], dim)?,+ (Some(l), Some(r)) => Tensor::cat(&[&l, &grad, &r], dim)?,+ };+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&arg_grad)?+ }+ Op::Unary(_, UnaryOp::Floor)+ | Op::Unary(_, UnaryOp::Round)+ | Op::Reduce(_, ReduceOp::ArgMin, _)+ | Op::Reduce(_, ReduceOp::ArgMax, _)+ | Op::Unary(_, UnaryOp::Sign)+ | Op::Cmp(_, _) => {}+ Op::Reshape(arg) => {+ let arg_grad = grad.reshape(arg.dims())?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&arg_grad)?+ }+ Op::Unary(_, UnaryOp::Ceil) => Err(Error::BackwardNotSupported { op: "ceil" })?,+ Op::Unary(arg, UnaryOp::Gelu) => {+ let sum_grad = grads.or_insert(arg)?;+ let cube = arg.powf(3.)?;+ let tanh = (0.0356774 * &cube + (0.797885 * arg)?)?.tanh()?;+ let gelu_grad = (((0.5 * &tanh)?+ + (0.0535161 * cube + (0.398942 * arg)?)? * (1. - tanh.powf(2.)?))?+ + 0.5)?;+ *sum_grad = sum_grad.add(&(&grad * gelu_grad)?)?+ }+ Op::Unary(arg, UnaryOp::Erf) => {+ let sum_grad = grads.or_insert(arg)?;+ // d/dx erf(x) = 2/sqrt(pi) * e^(-x^2)+ let erf_grad =+ (2. / std::f64::consts::PI.sqrt()) * (arg.sqr()?.neg()?).exp()?;+ *sum_grad = sum_grad.add(&(&grad * erf_grad)?)?+ }+ Op::Unary(arg, UnaryOp::GeluErf) => {+ let sum_grad = grads.or_insert(arg)?;+ // d/dx gelu_erf(x) = 0.5 + 0.398942 e^(-x^2/2) x + 0.5 erf(x/sqrt(2))+ let neg_half_square = (arg.sqr()?.neg()? / 2.)?;+ let scaled_exp_arg = (0.398942 * neg_half_square.exp()? * arg)?;+ let arg_scaled_sqrt = (arg / 2f64.sqrt())?;+ let erf_scaled_sqrt = (0.5 * arg_scaled_sqrt.erf()?)?;+ let gelu_erf_grad = (0.5 + scaled_exp_arg + erf_scaled_sqrt)?;+ *sum_grad = sum_grad.add(&(&grad * gelu_erf_grad)?)?;+ }+ Op::Unary(arg, UnaryOp::Relu) => {+ let sum_grad = grads.or_insert(arg)?;+ let relu_grad = arg.ge(&arg.zeros_like()?)?.to_dtype(arg.dtype())?;+ *sum_grad = sum_grad.add(&(&grad * relu_grad)?)?+ }+ Op::Unary(arg, UnaryOp::Silu) => {+ let sum_grad = grads.or_insert(arg)?;+ // d/dx silu = sigmoid(x) * (1 + x * (1 - sigmoid(x))) = sigmoid(x) * (1 - node) + node+ let sigmoid_arg = (arg.neg()?.exp()? + 1.)?.recip()?;+ let silu_grad = &sigmoid_arg * (1. - *node) + *node;+ *sum_grad = sum_grad.add(&(&grad * silu_grad)?)?+ }+ Op::Elu(arg, alpha) => {+ // d/dx elu(x) = 1 for x > 0, alpha * e^x for x <= 0+ let sum_grad = grads.or_insert(arg)?;+ let zeros = arg.zeros_like()?;+ let positive_mask = arg.gt(&zeros)?.to_dtype(arg.dtype())?;+ let negative_mask = arg.le(&zeros)?.to_dtype(arg.dtype())?;+ // node == alpha * (e^x - 1) for x <= 0, reuse it+ let negative_exp_mask = (negative_mask * (*node + *alpha))?;+ let combined_mask = (positive_mask + negative_exp_mask)?;+ *sum_grad = sum_grad.add(&(grad * combined_mask)?)?+ }+ Op::Powf(arg, e) => {+ let arg_grad = (&(grad * arg.powf(e - 1.)?)? * *e)?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&arg_grad)?+ }+ Op::CustomOp1(arg, c) => {+ if let Some(arg_grad) = c.bwd(arg, node, &grad)? {+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&arg_grad)?+ }+ }+ Op::CustomOp2(arg1, arg2, c) => {+ let (arg_grad1, arg_grad2) = c.bwd(arg1, arg2, node, &grad)?;+ if let Some(arg_grad1) = arg_grad1 {+ let sum_grad = grads.or_insert(arg1)?;+ *sum_grad = sum_grad.add(&arg_grad1)?+ }+ if let Some(arg_grad2) = arg_grad2 {+ let sum_grad = grads.or_insert(arg2)?;+ *sum_grad = sum_grad.add(&arg_grad2)?+ }+ }+ Op::CustomOp3(arg1, arg2, arg3, c) => {+ let (arg_grad1, arg_grad2, arg_grad3) =+ c.bwd(arg1, arg2, arg3, node, &grad)?;+ if let Some(arg_grad1) = arg_grad1 {+ let sum_grad = grads.or_insert(arg1)?;+ *sum_grad = sum_grad.add(&arg_grad1)?+ }+ if let Some(arg_grad2) = arg_grad2 {+ let sum_grad = grads.or_insert(arg2)?;+ *sum_grad = sum_grad.add(&arg_grad2)?+ }+ if let Some(arg_grad3) = arg_grad3 {+ let sum_grad = grads.or_insert(arg3)?;+ *sum_grad = sum_grad.add(&arg_grad3)?+ }+ }+ Op::Unary(arg, UnaryOp::Sqr) => {+ let arg_grad = arg.mul(&grad)?.affine(2., 0.)?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&arg_grad)?+ }+ Op::Unary(arg, UnaryOp::Sqrt) => {+ let arg_grad = grad.div(node)?.affine(0.5, 0.)?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&arg_grad)?+ }+ Op::ToDevice(arg) => {+ let sum_grad = grads.or_insert(arg)?;+ let arg_grad = grad.to_device(sum_grad.device())?;+ *sum_grad = sum_grad.add(&arg_grad)?+ }+ Op::Transpose(arg, dim1, dim2) => {+ let arg_grad = grad.transpose(*dim1, *dim2)?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&arg_grad)?+ }+ Op::Permute(arg, dims) => {+ let mut inv_dims = vec![0; dims.len()];+ for (i, &dim_idx) in dims.iter().enumerate() {+ inv_dims[dim_idx] = i+ }+ let arg_grad = grad.permute(inv_dims)?;+ let sum_grad = grads.or_insert(arg)?;+ *sum_grad = sum_grad.add(&arg_grad)?+ }+ };+ }+ }+ Ok(grads)+ }+}++/// A store for gradients, associating a tensor id to the corresponding gradient tensor, used for back propagation.+#[derive(Default, Debug)]+pub struct GradStore(HashMap<TensorId, Tensor>);++impl GradStore {+ /// Create a new gradient store+ fn new() -> Self {+ Self::default()+ }++ /// Get the gradient tensor corresponding to the given tensor id+ pub fn get_id(&self, id: TensorId) -> Option<&Tensor> {+ self.0.get(&id)+ }++ /// Get the gradient tensor associated with the given tensor+ pub fn get(&self, tensor: &Tensor) -> Option<&Tensor> {+ self.0.get(&tensor.id())+ }++ /// Remove the gradient tensor associated with the given tensor, returning it if it exists+ pub fn remove(&mut self, tensor: &Tensor) -> Option<Tensor> {+ self.0.remove(&tensor.id())+ }++ /// Insert a gradient tensor associated with the given tensor, returning the previous gradient tensor if it existed+ pub fn insert(&mut self, tensor: &Tensor, grad: Tensor) -> Option<Tensor> {+ self.0.insert(tensor.id(), grad)+ }++ /// Insert a gradient tensor associated with the given tensor id, returning the previous gradient tensor if it existed+ pub fn insert_id(&mut self, id: TensorId, grad: Tensor) -> Option<Tensor> {+ self.0.insert(id, grad)+ }++ /// Get the gradient tensor associated with the given tensor, or, if it does not exist,+ /// insert a tensor of zeroes, with the same shape and type as the given tensors and return it+ fn or_insert(&mut self, tensor: &Tensor) -> Result<&mut Tensor> {+ let grad = match self.0.entry(tensor.id()) {+ Entry::Occupied(entry) => entry.into_mut(),+ Entry::Vacant(entry) => {+ let grad = tensor.zeros_like()?;+ entry.insert(grad)+ }+ };+ Ok(grad)+ }++ /// Extend this gradient store with the contents of another.+ /// If an entry is already occupied, updates and ensures new tensor follows correct detach semantics.+ /// Otherwise simply inserts.+ pub fn extend(&mut self, other: Self) -> Result<()> {+ for (id, grad) in other.0 {+ match self.0.entry(id) {+ Entry::Occupied(mut entry) => {+ let new_grad = entry.get().add(&grad)?;++ let do_not_detach = CANDLE_GRAD_DO_NOT_DETACH.with(|b| *b);+ let new_grad = if do_not_detach {+ new_grad+ } else {+ new_grad.detach()+ };++ *entry.get_mut() = new_grad;+ }+ Entry::Vacant(entry) => {+ entry.insert(grad);+ }+ }+ }+ Ok(())+ }++ /// Get the tensor ids of the stored gradient tensors+ pub fn get_ids(&self) -> impl Iterator<Item = &TensorId> {+ self.0.keys()+ }+}diff --git a/vendor/candle-core/src/conv.rs b/vendor/candle-core/src/conv.rsnew file mode 100644index 0000000..115035e--- /dev/null+++ b/vendor/candle-core/src/conv.rs@@ -0,0 +1,387 @@+//! 1D and 2D Convolutions+//!+use crate::{op::BackpropOp, op::Op, Error, Result, Tensor};++#[derive(Debug, Clone, PartialEq, Eq)]+pub struct ParamsConv1D {+ pub(crate) b_size: usize,+ // Maybe we should have a version without l_in as this bit depends on the input and not only on+ // the weights.+ pub(crate) l_in: usize,+ pub(crate) c_out: usize,+ pub(crate) c_in: usize,+ pub(crate) k_size: usize,+ pub(crate) padding: usize,+ pub(crate) stride: usize,+ pub(crate) dilation: usize,+ pub(crate) cudnn_fwd_algo: Option<CudnnFwdAlgo>,+}++impl ParamsConv1D {+ pub(crate) fn l_out(&self) -> usize {+ (self.l_in + 2 * self.padding - self.dilation * (self.k_size - 1) - 1) / self.stride + 1+ }++ pub(crate) fn out_dims(&self) -> Vec<usize> {+ let l_out = self.l_out();+ vec![self.b_size, self.c_out, l_out]+ }+}++#[derive(Debug, Clone, PartialEq, Eq)]+pub struct ParamsConvTranspose1D {+ pub(crate) b_size: usize,+ pub(crate) l_in: usize,+ pub(crate) c_out: usize,+ pub(crate) c_in: usize,+ pub(crate) k_size: usize,+ pub(crate) padding: usize,+ pub(crate) output_padding: usize,+ pub(crate) stride: usize,+ pub(crate) dilation: usize,+}++impl ParamsConvTranspose1D {+ pub(crate) fn l_out(&self) -> usize {+ (self.l_in - 1) * self.stride - 2 * self.padding+ + self.dilation * (self.k_size - 1)+ + self.output_padding+ + 1+ }++ pub(crate) fn out_dims(&self) -> Vec<usize> {+ let l_out = self.l_out();+ vec![self.b_size, self.c_out, l_out]+ }+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]+pub enum CudnnFwdAlgo {+ ImplicitGemm,+ ImplicitPrecompGemm,+ Gemm,+ Direct,+ Fft,+ FftTiling,+ Winograd,+ WinogradNonFused,+ Count,+}++#[derive(Debug, Clone, PartialEq, Eq)]+pub struct ParamsConv2D {+ pub(crate) b_size: usize,+ pub(crate) i_h: usize,+ pub(crate) i_w: usize,+ pub(crate) k_h: usize,+ pub(crate) k_w: usize,+ pub(crate) c_out: usize,+ pub(crate) c_in: usize,+ pub(crate) padding: usize,+ pub(crate) stride: usize,+ pub(crate) dilation: usize,+ pub cudnn_fwd_algo: Option<CudnnFwdAlgo>,+}++impl ParamsConv2D {+ pub(crate) fn out_h(&self) -> usize {+ (self.i_h + 2 * self.padding - self.dilation * (self.k_h - 1) - 1) / self.stride + 1+ }++ pub(crate) fn out_w(&self) -> usize {+ (self.i_w + 2 * self.padding - self.dilation * (self.k_w - 1) - 1) / self.stride + 1+ }++ pub(crate) fn out_dims(&self) -> Vec<usize> {+ vec![self.b_size, self.c_out, self.out_h(), self.out_w()]+ }+}++#[derive(Debug, Clone, PartialEq, Eq)]+pub struct ParamsConvTranspose2D {+ pub(crate) b_size: usize,+ pub(crate) i_h: usize,+ pub(crate) i_w: usize,+ pub(crate) k_h: usize,+ pub(crate) k_w: usize,+ pub(crate) c_out: usize,+ pub(crate) c_in: usize,+ pub(crate) padding: usize,+ pub(crate) output_padding: usize,+ pub(crate) stride: usize,+ pub(crate) dilation: usize,+}++impl ParamsConvTranspose2D {+ pub(crate) fn out_h(&self) -> usize {+ (self.i_h - 1) * self.stride + self.dilation * (self.k_h - 1) + self.output_padding + 1+ - 2 * self.padding+ }++ pub(crate) fn out_w(&self) -> usize {+ (self.i_w - 1) * self.stride + self.dilation * (self.k_w - 1) + self.output_padding + 1+ - 2 * self.padding+ }++ pub(crate) fn out_dims(&self) -> Vec<usize> {+ vec![self.b_size, self.c_out, self.out_h(), self.out_w()]+ }+}++impl Tensor {+ fn conv1d_single_group(&self, kernel: &Self, params: &ParamsConv1D) -> Result<Self> {+ let storage =+ self.storage()+ .conv1d(self.layout(), &kernel.storage(), kernel.layout(), params)?;+ let op = BackpropOp::new2(self, kernel, |arg, kernel| Op::Conv1D {+ arg,+ kernel,+ padding: params.padding,+ stride: params.stride,+ dilation: params.dilation,+ });+ let out_dims = params.out_dims();+ Ok(crate::tensor::from_storage(storage, out_dims, op, false))+ }++ /// Applies a 1D convolution over the input tensor.+ pub fn conv1d(+ &self,+ kernel: &Self,+ padding: usize,+ stride: usize,+ dilation: usize,+ groups: usize,+ ) -> Result<Self> {+ self.conv1d_with_algo(kernel, padding, stride, dilation, groups, None)+ }++ /// Applies a 1D convolution over the input tensor.+ pub fn conv1d_with_algo(+ &self,+ kernel: &Self,+ padding: usize,+ stride: usize,+ dilation: usize,+ groups: usize,+ cudnn_fwd_algo: Option<CudnnFwdAlgo>,+ ) -> Result<Self> {+ let (c_out, c_in_k, k_size) = kernel.dims3()?;+ let (b_size, c_in, l_in) = self.dims3()?;+ if c_in != c_in_k * groups {+ Err(Error::Conv1dInvalidArgs {+ inp_shape: self.shape().clone(),+ k_shape: kernel.shape().clone(),+ padding,+ stride,+ msg: "the number of in-channels on the input doesn't match the kernel size",+ }+ .bt())?+ }++ let params = ParamsConv1D {+ b_size,+ l_in,+ c_out: c_out / groups,+ c_in: c_in / groups,+ k_size,+ padding,+ stride,+ dilation,+ cudnn_fwd_algo,+ };+ if groups == 1 {+ self.conv1d_single_group(kernel, ¶ms)+ } else {+ let blocks = self.chunk(groups, 1)?;+ let kernel = kernel.chunk(groups, 0)?;+ let blocks = blocks+ .iter()+ .zip(&kernel)+ .map(|(block, kernel)| block.conv1d_single_group(kernel, ¶ms))+ .collect::<Result<Vec<_>>>()?;+ Tensor::cat(&blocks, 1)+ }+ }++ fn conv_transpose1d_single_group(+ &self,+ kernel: &Self,+ params: &ParamsConvTranspose1D,+ ) -> Result<Self> {+ let storage = self.storage().conv_transpose1d(+ self.layout(),+ &kernel.storage(),+ kernel.layout(),+ params,+ )?;+ let op = BackpropOp::new2(self, kernel, |arg, kernel| Op::ConvTranspose1D {+ arg,+ kernel,+ padding: params.padding,+ output_padding: params.output_padding,+ stride: params.stride,+ dilation: params.dilation,+ });+ let out_dims = params.out_dims();+ Ok(crate::tensor::from_storage(storage, out_dims, op, false))+ }++ /// Applies a 1D transposed convolution over the input tensor.+ pub fn conv_transpose1d(+ &self,+ kernel: &Self,+ padding: usize,+ output_padding: usize,+ stride: usize,+ dilation: usize,+ groups: usize,+ ) -> Result<Self> {+ let (c_in_k, c_out, k_size) = kernel.dims3()?;+ let (b_size, c_in, l_in) = self.dims3()?;+ if c_in != c_in_k {+ crate::bail!("in_channel mismatch between input ({c_in}) and kernel ({c_in_k})")+ }+ if c_in % groups != 0 {+ crate::bail!("in_channel {c_in} is not divisible by the number of groups")+ }+ let params = ParamsConvTranspose1D {+ b_size,+ l_in,+ k_size,+ c_out,+ c_in: c_in / groups,+ padding,+ output_padding,+ stride,+ dilation,+ };+ if groups == 1 {+ self.conv_transpose1d_single_group(kernel, ¶ms)+ } else {+ let blocks = self.chunk(groups, 1)?;+ let kernel = kernel.chunk(groups, 0)?;+ let blocks = blocks+ .iter()+ .zip(&kernel)+ .map(|(block, kernel)| block.conv_transpose1d_single_group(kernel, ¶ms))+ .collect::<Result<Vec<_>>>()?;+ Tensor::cat(&blocks, 1)+ }+ }++ fn conv2d_single_group(&self, kernel: &Self, params: &ParamsConv2D) -> Result<Self> {+ let storage =+ self.storage()+ .conv2d(self.layout(), &kernel.storage(), kernel.layout(), params)?;+ let op = BackpropOp::new2(self, kernel, |arg, kernel| Op::Conv2D {+ arg,+ kernel,+ padding: params.padding,+ stride: params.stride,+ dilation: params.dilation,+ });+ let out_dims = params.out_dims();+ Ok(crate::tensor::from_storage(storage, out_dims, op, false))+ }++ /// Applies a 2D convolution over the input tensor.+ pub fn conv2d(+ &self,+ kernel: &Self,+ padding: usize,+ stride: usize,+ dilation: usize,+ groups: usize,+ ) -> Result<Self> {+ self.conv2d_with_algo(kernel, padding, stride, dilation, groups, None)+ }++ pub fn conv2d_with_algo(+ &self,+ kernel: &Self,+ padding: usize,+ stride: usize,+ dilation: usize,+ groups: usize,+ cudnn_fwd_algo: Option<CudnnFwdAlgo>,+ ) -> Result<Self> {+ let (b_size, c_in, i_h, i_w) = self.dims4()?;+ let (c_out, c_in_k, k_h, k_w) = kernel.dims4()?;+ if c_in != c_in_k * groups {+ crate::bail!(+ "in_channel mismatch between input ({c_in}, groups {groups}) and kernel ({c_in_k})"+ )+ }+ let params = ParamsConv2D {+ b_size,+ i_h,+ i_w,+ k_h,+ k_w,+ c_out: c_out / groups,+ c_in: c_in / groups,+ padding,+ stride,+ dilation,+ cudnn_fwd_algo,+ };+ if groups == 1 {+ self.conv2d_single_group(kernel, ¶ms)+ } else {+ let blocks = self.chunk(groups, 1)?;+ let kernel = kernel.chunk(groups, 0)?;+ let blocks = blocks+ .iter()+ .zip(&kernel)+ .map(|(block, kernel)| block.conv2d_single_group(kernel, ¶ms))+ .collect::<Result<Vec<_>>>()?;+ Tensor::cat(&blocks, 1)+ }+ }++ /// Applies a 2D transposed convolution over the input tensor.+ pub fn conv_transpose2d(+ &self,+ kernel: &Self,+ padding: usize,+ output_padding: usize,+ stride: usize,+ dilation: usize,+ ) -> Result<Self> {+ let (b_size, c_in, i_h, i_w) = self.dims4()?;+ let (c_in_k, c_out, k_h, k_w) = kernel.dims4()?;+ if c_in != c_in_k {+ crate::bail!("in_channel mismatch between input ({c_in}) and kernel ({c_in_k})")+ }+ let params = ParamsConvTranspose2D {+ b_size,+ i_h,+ i_w,+ k_h,+ k_w,+ c_out,+ c_in,+ padding,+ output_padding,+ stride,+ dilation,+ };+ let storage = self.storage().conv_transpose2d(+ self.layout(),+ &kernel.storage(),+ kernel.layout(),+ ¶ms,+ )?;+ let op = BackpropOp::new2(self, kernel, |arg, kernel| Op::ConvTranspose2D {+ arg,+ kernel,+ padding: params.padding,+ output_padding: params.output_padding,+ stride: params.stride,+ dilation: params.dilation,+ });+ let out_dims = params.out_dims();+ Ok(crate::tensor::from_storage(storage, out_dims, op, false))+ }+}diff --git a/vendor/candle-core/src/convert.rs b/vendor/candle-core/src/convert.rsnew file mode 100644index 0000000..38e7a7c--- /dev/null+++ b/vendor/candle-core/src/convert.rs@@ -0,0 +1,166 @@+//! Implement conversion traits for tensors+use crate::{DType, Device, Error, Tensor, WithDType};+use half::{bf16, f16, slice::HalfFloatSliceExt};+use std::convert::TryFrom;++impl<T: WithDType> TryFrom<&Tensor> for Vec<T> {+ type Error = Error;+ fn try_from(tensor: &Tensor) -> Result<Self, Self::Error> {+ tensor.to_vec1::<T>()+ }+}++impl<T: WithDType> TryFrom<&Tensor> for Vec<Vec<T>> {+ type Error = Error;+ fn try_from(tensor: &Tensor) -> Result<Self, Self::Error> {+ tensor.to_vec2::<T>()+ }+}++impl<T: WithDType> TryFrom<&Tensor> for Vec<Vec<Vec<T>>> {+ type Error = Error;+ fn try_from(tensor: &Tensor) -> Result<Self, Self::Error> {+ tensor.to_vec3::<T>()+ }+}++impl<T: WithDType> TryFrom<Tensor> for Vec<T> {+ type Error = Error;+ fn try_from(tensor: Tensor) -> Result<Self, Self::Error> {+ Vec::<T>::try_from(&tensor)+ }+}++impl<T: WithDType> TryFrom<Tensor> for Vec<Vec<T>> {+ type Error = Error;+ fn try_from(tensor: Tensor) -> Result<Self, Self::Error> {+ Vec::<Vec<T>>::try_from(&tensor)+ }+}++impl<T: WithDType> TryFrom<Tensor> for Vec<Vec<Vec<T>>> {+ type Error = Error;+ fn try_from(tensor: Tensor) -> Result<Self, Self::Error> {+ Vec::<Vec<Vec<T>>>::try_from(&tensor)+ }+}++impl<T: WithDType> TryFrom<&[T]> for Tensor {+ type Error = Error;+ fn try_from(v: &[T]) -> Result<Self, Self::Error> {+ Tensor::from_slice(v, v.len(), &Device::Cpu)+ }+}++impl<T: WithDType> TryFrom<Vec<T>> for Tensor {+ type Error = Error;+ fn try_from(v: Vec<T>) -> Result<Self, Self::Error> {+ let len = v.len();+ Tensor::from_vec(v, len, &Device::Cpu)+ }+}++macro_rules! from_tensor {+ ($typ:ident) => {+ impl TryFrom<&Tensor> for $typ {+ type Error = Error;++ fn try_from(tensor: &Tensor) -> Result<Self, Self::Error> {+ tensor.to_scalar::<$typ>()+ }+ }++ impl TryFrom<Tensor> for $typ {+ type Error = Error;++ fn try_from(tensor: Tensor) -> Result<Self, Self::Error> {+ $typ::try_from(&tensor)+ }+ }++ impl TryFrom<$typ> for Tensor {+ type Error = Error;++ fn try_from(v: $typ) -> Result<Self, Self::Error> {+ Tensor::new(v, &Device::Cpu)+ }+ }+ };+}++from_tensor!(f64);+from_tensor!(f32);+from_tensor!(f16);+from_tensor!(bf16);+from_tensor!(i64);+from_tensor!(i32);+from_tensor!(i16);+from_tensor!(u32);+from_tensor!(u8);++impl Tensor {+ pub fn write_bytes<W: std::io::Write>(&self, f: &mut W) -> crate::Result<()> {+ use byteorder::{LittleEndian, WriteBytesExt};++ let vs = self.flatten_all()?;+ match self.dtype() {+ DType::BF16 => {+ let vs = vs.to_vec1::<bf16>()?;+ for &v in vs.reinterpret_cast() {+ f.write_u16::<LittleEndian>(v)?+ }+ }+ DType::F16 => {+ let vs = vs.to_vec1::<f16>()?;+ for &v in vs.reinterpret_cast() {+ f.write_u16::<LittleEndian>(v)?+ }+ }+ DType::F32 => {+ // TODO: Avoid using a buffer when data is already on the CPU.+ for v in vs.to_vec1::<f32>()? {+ f.write_f32::<LittleEndian>(v)?+ }+ }+ DType::F64 => {+ for v in vs.to_vec1::<f64>()? {+ f.write_f64::<LittleEndian>(v)?+ }+ }+ DType::U32 => {+ for v in vs.to_vec1::<u32>()? {+ f.write_u32::<LittleEndian>(v)?+ }+ }+ DType::I16 => {+ for v in vs.to_vec1::<i16>()? {+ f.write_i16::<LittleEndian>(v)?+ }+ }+ DType::I32 => {+ for v in vs.to_vec1::<i32>()? {+ f.write_i32::<LittleEndian>(v)?+ }+ }+ DType::I64 => {+ for v in vs.to_vec1::<i64>()? {+ f.write_i64::<LittleEndian>(v)?+ }+ }+ DType::U8 => {+ let vs = vs.to_vec1::<u8>()?;+ f.write_all(&vs)?;+ }+ DType::F8E4M3 => {+ let vs = vs.to_vec1::<float8::F8E4M3>()?;+ for v in vs {+ f.write_u8(v.to_bits())?+ }+ }+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ return Err(crate::Error::UnsupportedDTypeForOp(self.dtype(), "write_bytes").bt())+ }+ }+ Ok(())+ }+}diff --git a/vendor/candle-core/src/cpu/avx.rs b/vendor/candle-core/src/cpu/avx.rsnew file mode 100644index 0000000..1aaa40e--- /dev/null+++ b/vendor/candle-core/src/cpu/avx.rs@@ -0,0 +1,391 @@+use super::{Cpu, CpuBF16, CpuF16};+#[cfg(target_arch = "x86")]+use core::arch::x86::*;+#[cfg(target_arch = "x86_64")]+use core::arch::x86_64::*;+use half::{bf16, f16};+use std::{is_x86_feature_detected, mem::transmute};++pub struct CurrentCpu {}++const STEP: usize = 32;+const EPR: usize = 8;+const ARR: usize = STEP / EPR;++impl Cpu for CurrentCpu {+ type Unit = __m256;+ type Array = [__m256; ARR];++ const STEP: usize = STEP;+ const EPR: usize = EPR;+ const ARR: usize = ARR;++ unsafe fn zero() -> Self::Unit {+ _mm256_setzero_ps()+ }++ unsafe fn zero_array() -> Self::Array {+ [Self::zero(); ARR]+ }++ unsafe fn from_f32(v: f32) -> Self::Unit {+ _mm256_set1_ps(v)+ }++ unsafe fn load(mem_addr: *const f32) -> Self::Unit {+ _mm256_loadu_ps(mem_addr)+ }++ unsafe fn vec_add(a: Self::Unit, b: Self::Unit) -> Self::Unit {+ _mm256_add_ps(a, b)+ }++ unsafe fn vec_fma(a: Self::Unit, b: Self::Unit, c: Self::Unit) -> Self::Unit {+ _mm256_add_ps(_mm256_mul_ps(b, c), a)+ }++ unsafe fn vec_store(mem_addr: *mut f32, a: Self::Unit) {+ _mm256_storeu_ps(mem_addr, a);+ }++ unsafe fn vec_reduce(mut x: Self::Array, y: *mut f32) {+ for i in 0..ARR / 2 {+ x[2 * i] = _mm256_add_ps(x[2 * i], x[2 * i + 1]);+ }+ for i in 0..ARR / 4 {+ x[4 * i] = _mm256_add_ps(x[4 * i], x[4 * i + 2]);+ }+ #[allow(clippy::reversed_empty_ranges)]+ for i in 0..ARR / 8 {+ x[8 * i] = _mm256_add_ps(x[8 * i], x[8 * i + 4]);+ }+ let t0 = _mm_add_ps(_mm256_castps256_ps128(x[0]), _mm256_extractf128_ps(x[0], 1));+ let t1 = _mm_hadd_ps(t0, t0);+ *y = _mm_cvtss_f32(_mm_hadd_ps(t1, t1));+ }+}++pub struct CurrentCpuF16 {}+impl CpuF16 for CurrentCpuF16 {+ type Unit = __m256;+ type Array = [__m256; ARR];++ const STEP: usize = STEP;+ const EPR: usize = EPR;+ const ARR: usize = ARR;++ unsafe fn zero() -> Self::Unit {+ _mm256_setzero_ps()+ }++ unsafe fn zero_array() -> Self::Array {+ [Self::zero(); ARR]+ }++ unsafe fn from_f32(v: f32) -> Self::Unit {+ _mm256_set1_ps(v)+ }++ #[cfg(target_feature = "f16c")]+ unsafe fn load(mem_addr: *const f16) -> Self::Unit {+ _mm256_cvtph_ps(_mm_loadu_si128(mem_addr as *const __m128i))+ }++ #[cfg(not(target_feature = "f16c"))]+ unsafe fn load(mem_addr: *const f16) -> Self::Unit {+ let mut tmp = [0.0f32; 8];+ for i in 0..8 {+ tmp[i] = (*mem_addr.add(i)).to_f32();+ }+ _mm256_loadu_ps(tmp.as_ptr())+ }++ unsafe fn vec_add(a: Self::Unit, b: Self::Unit) -> Self::Unit {+ _mm256_add_ps(a, b)+ }++ unsafe fn vec_fma(a: Self::Unit, b: Self::Unit, c: Self::Unit) -> Self::Unit {+ _mm256_add_ps(_mm256_mul_ps(b, c), a)+ }++ #[cfg(target_feature = "f16c")]+ unsafe fn vec_store(mem_addr: *mut f16, a: Self::Unit) {+ _mm_storeu_si128(mem_addr as *mut __m128i, _mm256_cvtps_ph(a, 0))+ }++ #[cfg(not(target_feature = "f16c"))]+ unsafe fn vec_store(mem_addr: *mut f16, a: Self::Unit) {+ let mut tmp = [0.0f32; 8];+ _mm256_storeu_ps(tmp.as_mut_ptr(), a);+ for i in 0..8 {+ *mem_addr.add(i) = f16::from_f32(tmp[i]);+ }+ }++ unsafe fn vec_reduce(mut x: Self::Array, y: *mut f32) {+ let mut offset = ARR >> 1;+ for i in 0..offset {+ x[i] = _mm256_add_ps(x[i], x[offset + i]);+ }+ offset >>= 1;+ for i in 0..offset {+ x[i] = _mm256_add_ps(x[i], x[offset + i]);+ }+ offset >>= 1;+ for i in 0..offset {+ x[i] = _mm256_add_ps(x[i], x[offset + i]);+ }+ let t0 = _mm_add_ps(_mm256_castps256_ps128(x[0]), _mm256_extractf128_ps(x[0], 1));+ let t1 = _mm_hadd_ps(t0, t0);+ *y = _mm_cvtss_f32(_mm_hadd_ps(t1, t1));+ }+}++pub struct CurrentCpuBF16 {}+impl CpuBF16 for CurrentCpuBF16 {+ type Unit = __m256;+ type Array = [__m256; ARR];++ const STEP: usize = STEP;+ const EPR: usize = EPR;+ const ARR: usize = ARR;++ unsafe fn zero() -> Self::Unit {+ _mm256_setzero_ps()+ }++ unsafe fn zero_array() -> Self::Array {+ [Self::zero(); ARR]+ }++ unsafe fn from_f32(v: f32) -> Self::Unit {+ _mm256_set1_ps(v)+ }++ unsafe fn load(mem_addr: *const bf16) -> Self::Unit {+ if is_x86_feature_detected!("avx512bf16") && is_x86_feature_detected!("avx512vl") {+ _mm256_cvtpbh_ps(transmute::<__m128i, __m128bh>(_mm_loadu_si128(+ mem_addr as *const __m128i,+ )))+ } else {+ let mut tmp = [0.0f32; 8];+ for i in 0..8 {+ tmp[i] = (*mem_addr.add(i)).to_f32();+ }+ _mm256_loadu_ps(tmp.as_ptr())+ }+ }++ unsafe fn vec_add(a: Self::Unit, b: Self::Unit) -> Self::Unit {+ _mm256_add_ps(a, b)+ }++ unsafe fn vec_fma(a: Self::Unit, b: Self::Unit, c: Self::Unit) -> Self::Unit {+ _mm256_add_ps(_mm256_mul_ps(b, c), a)+ }++ unsafe fn vec_store(mem_addr: *mut bf16, a: Self::Unit) {+ if is_x86_feature_detected!("avx512bf16") && is_x86_feature_detected!("avx512vl") {+ _mm_storeu_si128(+ mem_addr as *mut __m128i,+ transmute::<__m128bh, __m128i>(_mm256_cvtneps_pbh(a)),+ )+ } else {+ let mut tmp = [0.0f32; 8];+ _mm256_storeu_ps(tmp.as_mut_ptr(), a);+ for i in 0..8 {+ *mem_addr.add(i) = bf16::from_f32(tmp[i]);+ }+ }+ }++ unsafe fn vec_reduce(mut x: Self::Array, y: *mut f32) {+ let mut offset = ARR >> 1;+ for i in 0..offset {+ x[i] = _mm256_add_ps(x[i], x[offset + i]);+ }+ offset >>= 1;+ for i in 0..offset {+ x[i] = _mm256_add_ps(x[i], x[offset + i]);+ }+ offset >>= 1;+ for i in 0..offset {+ x[i] = _mm256_add_ps(x[i], x[offset + i]);+ }+ let t0 = _mm_add_ps(_mm256_castps256_ps128(x[0]), _mm256_extractf128_ps(x[0], 1));+ let t1 = _mm_hadd_ps(t0, t0);+ *y = _mm_cvtss_f32(_mm_hadd_ps(t1, t1));+ }+}++#[target_feature(enable = "avx2")]+pub(crate) unsafe fn vec_dot_f32(a_row: *const f32, b_row: *const f32, c: *mut f32, k: usize) {+ let mut sum = CurrentCpu::zero_array();+ let mut i = 0;+ while i + CurrentCpu::STEP <= k {+ for j in 0..CurrentCpu::ARR {+ sum[j] = CurrentCpu::vec_fma(+ sum[j],+ CurrentCpu::load(a_row.add(i + j * CurrentCpu::EPR)),+ CurrentCpu::load(b_row.add(i + j * CurrentCpu::EPR)),+ );+ }+ i += CurrentCpu::STEP;+ }+ CurrentCpu::vec_reduce(sum, c);+ while i < k {+ *c += *a_row.add(i) * (*b_row.add(i));+ i += 1;+ }+}++#[target_feature(enable = "avx2")]+pub(crate) unsafe fn vec_sum(row: *const f32, b: *mut f32, k: usize) {+ let np = k & !(CurrentCpu::STEP - 1);+ let mut sum = CurrentCpu::zero_array();+ let mut x = CurrentCpu::zero_array();+ for i in (0..np).step_by(CurrentCpu::STEP) {+ for j in 0..CurrentCpu::ARR {+ x[j] = CurrentCpu::load(row.add(i + j * CurrentCpu::EPR));+ sum[j] = CurrentCpu::vec_add(sum[j], x[j]);+ }+ }+ CurrentCpu::vec_reduce(sum, b);+ for i in np..k {+ *b += *row.add(i)+ }+}++#[target_feature(enable = "avx2")]+pub(crate) unsafe fn vec_dot_f16(a_row: *const f16, b_row: *const f16, c: *mut f32, k: usize) {+ let mut sumf = 0.0f32;+ let mut sum = CurrentCpuF16::zero_array();+ let mut steps_since_flush = 0usize;+ let mut i = 0;+ while i + CurrentCpuF16::STEP <= k {+ for j in 0..CurrentCpuF16::ARR {+ sum[j] = CurrentCpuF16::vec_fma(+ sum[j],+ CurrentCpuF16::load(a_row.add(i + j * CurrentCpuF16::EPR)),+ CurrentCpuF16::load(b_row.add(i + j * CurrentCpuF16::EPR)),+ );+ }+ steps_since_flush += 1;+ if steps_since_flush == CurrentCpuF16::FLUSH_INTERVAL {+ let mut partial = 0.0f32;+ CurrentCpuF16::vec_reduce(sum, &mut partial);+ sumf += partial;+ sum = CurrentCpuF16::zero_array();+ steps_since_flush = 0;+ }+ i += CurrentCpuF16::STEP;+ }+ let mut partial = 0.0f32;+ CurrentCpuF16::vec_reduce(sum, &mut partial);+ sumf += partial;+ while i < k {+ sumf += (*a_row.add(i)).to_f32() * (*b_row.add(i)).to_f32();+ i += 1;+ }+ *c = sumf;+}++#[target_feature(enable = "avx2")]+pub(crate) unsafe fn vec_dot_bf16(a_row: *const bf16, b_row: *const bf16, c: *mut f32, k: usize) {+ let mut sum = CurrentCpuBF16::zero_array();+ let mut i = 0;+ while i + CurrentCpuBF16::STEP <= k {+ for j in 0..CurrentCpuBF16::ARR {+ sum[j] = CurrentCpuBF16::vec_fma(+ sum[j],+ CurrentCpuBF16::load(a_row.add(i + j * CurrentCpuBF16::EPR)),+ CurrentCpuBF16::load(b_row.add(i + j * CurrentCpuBF16::EPR)),+ );+ }+ i += CurrentCpuBF16::STEP;+ }+ CurrentCpuBF16::vec_reduce(sum, c);+ while i < k {+ *c += (*a_row.add(i)).to_f32() * (*b_row.add(i)).to_f32();+ i += 1;+ }+}++#[target_feature(enable = "avx2")]+pub(crate) unsafe fn vec_add_f16(a_row: *const f16, b_row: *const f16, c: *mut f16, k: usize) {+ let mut i = 0;+ while i + CurrentCpuF16::STEP <= k {+ for j in 0..CurrentCpuF16::ARR {+ CurrentCpuF16::vec_store(+ c.add(i + j * CurrentCpuF16::EPR),+ CurrentCpuF16::vec_add(+ CurrentCpuF16::load(a_row.add(i + j * CurrentCpuF16::EPR)),+ CurrentCpuF16::load(b_row.add(i + j * CurrentCpuF16::EPR)),+ ),+ );+ }+ i += CurrentCpuF16::STEP;+ }+ for j in i..k {+ *c.add(j) = *a_row.add(j) + *b_row.add(j);+ }+}++#[target_feature(enable = "avx2")]+pub(crate) unsafe fn vec_add_bf16(a_row: *const bf16, b_row: *const bf16, c: *mut bf16, k: usize) {+ let mut i = 0;+ while i + CurrentCpuBF16::STEP <= k {+ for j in 0..CurrentCpuBF16::ARR {+ CurrentCpuBF16::vec_store(+ c.add(i + j * CurrentCpuBF16::EPR),+ CurrentCpuBF16::vec_add(+ CurrentCpuBF16::load(a_row.add(i + j * CurrentCpuBF16::EPR)),+ CurrentCpuBF16::load(b_row.add(i + j * CurrentCpuBF16::EPR)),+ ),+ );+ }+ i += CurrentCpuBF16::STEP;+ }+ for j in i..k {+ *c.add(j) = *a_row.add(j) + *b_row.add(j);+ }+}++#[target_feature(enable = "avx2")]+pub(crate) unsafe fn vec_scalar_add_f16(scalar: f16, xs: *const f16, ys: *mut f16, k: usize) {+ let sv = CurrentCpuF16::from_f32(scalar.to_f32());+ let mut i = 0;+ while i + CurrentCpuF16::STEP <= k {+ for j in 0..CurrentCpuF16::ARR {+ CurrentCpuF16::vec_store(+ ys.add(i + j * CurrentCpuF16::EPR),+ CurrentCpuF16::vec_add(CurrentCpuF16::load(xs.add(i + j * CurrentCpuF16::EPR)), sv),+ );+ }+ i += CurrentCpuF16::STEP;+ }+ for j in i..k {+ *ys.add(j) = *xs.add(j) + scalar;+ }+}++#[target_feature(enable = "avx2")]+pub(crate) unsafe fn vec_scalar_add_bf16(scalar: bf16, xs: *const bf16, ys: *mut bf16, k: usize) {+ let sv = CurrentCpuBF16::from_f32(scalar.to_f32());+ let mut i = 0;+ while i + CurrentCpuBF16::STEP <= k {+ for j in 0..CurrentCpuBF16::ARR {+ CurrentCpuBF16::vec_store(+ ys.add(i + j * CurrentCpuBF16::EPR),+ CurrentCpuBF16::vec_add(+ CurrentCpuBF16::load(xs.add(i + j * CurrentCpuBF16::EPR)),+ sv,+ ),+ );+ }+ i += CurrentCpuBF16::STEP;+ }+ for j in i..k {+ *ys.add(j) = *xs.add(j) + scalar;+ }+}diff --git a/vendor/candle-core/src/cpu/erf.rs b/vendor/candle-core/src/cpu/erf.rsnew file mode 100644index 0000000..bc99c89--- /dev/null+++ b/vendor/candle-core/src/cpu/erf.rs@@ -0,0 +1,319 @@+#![allow(clippy::excessive_precision)]+// Code taken from https://github.com/statrs-dev/statrs+//! Provides the [error](https://en.wikipedia.org/wiki/Error_function) and+//! related functions++mod evaluate {+ //! Provides functions that don't have a numerical solution and must+ //! be solved computationally (e.g. evaluation of a polynomial)++ /// evaluates a polynomial at `z` where `coeff` are the coefficients+ /// to a polynomial of order `k` where `k` is the length of `coeff` and the+ /// coeffecient+ /// to the `k`th power is the `k`th element in coeff. E.g. [3,-1,2] equates to+ /// `2z^2 - z + 3`+ ///+ /// # Remarks+ ///+ /// Returns 0 for a 0 length coefficient slice+ pub fn polynomial(z: f64, coeff: &[f64]) -> f64 {+ match coeff {+ [] => 0.0,+ [.., sum] => coeff.iter().rev().skip(1).fold(*sum, |sum, c| c + z * sum),+ }+ }+}+use std::f64;++/// `erf` calculates the error function at `x`.+pub fn erf_f64(x: f64) -> f64 {+ libm::erf(x)+}++pub fn erf_f32(x: f32) -> f32 {+ libm::erff(x)+}++/// `erf_inv` calculates the inverse error function+/// at `x`.+pub fn erf_inv(x: f64) -> f64 {+ if x == 0.0 {+ 0.0+ } else if x >= 1.0 {+ f64::INFINITY+ } else if x <= -1.0 {+ f64::NEG_INFINITY+ } else if x < 0.0 {+ erf_inv_impl(-x, 1.0 + x, -1.0)+ } else {+ erf_inv_impl(x, 1.0 - x, 1.0)+ }+}++/// `erfc` calculates the complementary error function+/// at `x`.+pub fn erfc_f64(x: f64) -> f64 {+ libm::erfc(x)+}++pub fn erfc_f32(x: f32) -> f32 {+ libm::erfcf(x)+}++/// `erfc_inv` calculates the complementary inverse+/// error function at `x`.+pub fn erfc_inv(x: f64) -> f64 {+ if x <= 0.0 {+ f64::INFINITY+ } else if x >= 2.0 {+ f64::NEG_INFINITY+ } else if x > 1.0 {+ erf_inv_impl(-1.0 + x, 2.0 - x, -1.0)+ } else {+ erf_inv_impl(1.0 - x, x, 1.0)+ }+}++// **********************************************************+// ********** Coefficients for erf_inv_impl polynomial ******+// **********************************************************++/// Polynomial coefficients for a numerator of `erf_inv_impl`+/// in the interval [0, 0.5].+const ERF_INV_IMPL_AN: &[f64] = &[+ -0.000508781949658280665617,+ -0.00836874819741736770379,+ 0.0334806625409744615033,+ -0.0126926147662974029034,+ -0.0365637971411762664006,+ 0.0219878681111168899165,+ 0.00822687874676915743155,+ -0.00538772965071242932965,+];++/// Polynomial coefficients for a denominator of `erf_inv_impl`+/// in the interval [0, 0.5].+const ERF_INV_IMPL_AD: &[f64] = &[+ 1.0,+ -0.970005043303290640362,+ -1.56574558234175846809,+ 1.56221558398423026363,+ 0.662328840472002992063,+ -0.71228902341542847553,+ -0.0527396382340099713954,+ 0.0795283687341571680018,+ -0.00233393759374190016776,+ 0.000886216390456424707504,+];++/// Polynomial coefficients for a numerator of `erf_inv_impl`+/// in the interval [0.5, 0.75].+const ERF_INV_IMPL_BN: &[f64] = &[+ -0.202433508355938759655,+ 0.105264680699391713268,+ 8.37050328343119927838,+ 17.6447298408374015486,+ -18.8510648058714251895,+ -44.6382324441786960818,+ 17.445385985570866523,+ 21.1294655448340526258,+ -3.67192254707729348546,+];++/// Polynomial coefficients for a denominator of `erf_inv_impl`+/// in the interval [0.5, 0.75].+const ERF_INV_IMPL_BD: &[f64] = &[+ 1.0,+ 6.24264124854247537712,+ 3.9713437953343869095,+ -28.6608180499800029974,+ -20.1432634680485188801,+ 48.5609213108739935468,+ 10.8268667355460159008,+ -22.6436933413139721736,+ 1.72114765761200282724,+];++/// Polynomial coefficients for a numerator of `erf_inv_impl`+/// in the interval [0.75, 1] with x less than 3.+const ERF_INV_IMPL_CN: &[f64] = &[+ -0.131102781679951906451,+ -0.163794047193317060787,+ 0.117030156341995252019,+ 0.387079738972604337464,+ 0.337785538912035898924,+ 0.142869534408157156766,+ 0.0290157910005329060432,+ 0.00214558995388805277169,+ -0.679465575181126350155e-6,+ 0.285225331782217055858e-7,+ -0.681149956853776992068e-9,+];++/// Polynomial coefficients for a denominator of `erf_inv_impl`+/// in the interval [0.75, 1] with x less than 3.+const ERF_INV_IMPL_CD: &[f64] = &[+ 1.0,+ 3.46625407242567245975,+ 5.38168345707006855425,+ 4.77846592945843778382,+ 2.59301921623620271374,+ 0.848854343457902036425,+ 0.152264338295331783612,+ 0.01105924229346489121,+];++/// Polynomial coefficients for a numerator of `erf_inv_impl`+/// in the interval [0.75, 1] with x between 3 and 6.+const ERF_INV_IMPL_DN: &[f64] = &[+ -0.0350353787183177984712,+ -0.00222426529213447927281,+ 0.0185573306514231072324,+ 0.00950804701325919603619,+ 0.00187123492819559223345,+ 0.000157544617424960554631,+ 0.460469890584317994083e-5,+ -0.230404776911882601748e-9,+ 0.266339227425782031962e-11,+];++/// Polynomial coefficients for a denominator of `erf_inv_impl`+/// in the interval [0.75, 1] with x between 3 and 6.+const ERF_INV_IMPL_DD: &[f64] = &[+ 1.0,+ 1.3653349817554063097,+ 0.762059164553623404043,+ 0.220091105764131249824,+ 0.0341589143670947727934,+ 0.00263861676657015992959,+ 0.764675292302794483503e-4,+];++/// Polynomial coefficients for a numerator of `erf_inv_impl`+/// in the interval [0.75, 1] with x between 6 and 18.+const ERF_INV_IMPL_EN: &[f64] = &[+ -0.0167431005076633737133,+ -0.00112951438745580278863,+ 0.00105628862152492910091,+ 0.000209386317487588078668,+ 0.149624783758342370182e-4,+ 0.449696789927706453732e-6,+ 0.462596163522878599135e-8,+ -0.281128735628831791805e-13,+ 0.99055709973310326855e-16,+];++/// Polynomial coefficients for a denominator of `erf_inv_impl`+/// in the interval [0.75, 1] with x between 6 and 18.+const ERF_INV_IMPL_ED: &[f64] = &[+ 1.0,+ 0.591429344886417493481,+ 0.138151865749083321638,+ 0.0160746087093676504695,+ 0.000964011807005165528527,+ 0.275335474764726041141e-4,+ 0.282243172016108031869e-6,+];++/// Polynomial coefficients for a numerator of `erf_inv_impl`+/// in the interval [0.75, 1] with x between 18 and 44.+const ERF_INV_IMPL_FN: &[f64] = &[+ -0.0024978212791898131227,+ -0.779190719229053954292e-5,+ 0.254723037413027451751e-4,+ 0.162397777342510920873e-5,+ 0.396341011304801168516e-7,+ 0.411632831190944208473e-9,+ 0.145596286718675035587e-11,+ -0.116765012397184275695e-17,+];++/// Polynomial coefficients for a denominator of `erf_inv_impl`+/// in the interval [0.75, 1] with x between 18 and 44.+const ERF_INV_IMPL_FD: &[f64] = &[+ 1.0,+ 0.207123112214422517181,+ 0.0169410838120975906478,+ 0.000690538265622684595676,+ 0.145007359818232637924e-4,+ 0.144437756628144157666e-6,+ 0.509761276599778486139e-9,+];++/// Polynomial coefficients for a numerator of `erf_inv_impl`+/// in the interval [0.75, 1] with x greater than 44.+const ERF_INV_IMPL_GN: &[f64] = &[+ -0.000539042911019078575891,+ -0.28398759004727721098e-6,+ 0.899465114892291446442e-6,+ 0.229345859265920864296e-7,+ 0.225561444863500149219e-9,+ 0.947846627503022684216e-12,+ 0.135880130108924861008e-14,+ -0.348890393399948882918e-21,+];++/// Polynomial coefficients for a denominator of `erf_inv_impl`+/// in the interval [0.75, 1] with x greater than 44.+const ERF_INV_IMPL_GD: &[f64] = &[+ 1.0,+ 0.0845746234001899436914,+ 0.00282092984726264681981,+ 0.468292921940894236786e-4,+ 0.399968812193862100054e-6,+ 0.161809290887904476097e-8,+ 0.231558608310259605225e-11,+];++// `erf_inv_impl` computes the inverse error function where+// `p`,`q`, and `s` are the first, second, and third intermediate+// parameters respectively+fn erf_inv_impl(p: f64, q: f64, s: f64) -> f64 {+ let result = if p <= 0.5 {+ let y = 0.0891314744949340820313;+ let g = p * (p + 10.0);+ let r = evaluate::polynomial(p, ERF_INV_IMPL_AN) / evaluate::polynomial(p, ERF_INV_IMPL_AD);+ g * y + g * r+ } else if q >= 0.25 {+ let y = 2.249481201171875;+ let g = (-2.0 * q.ln()).sqrt();+ let xs = q - 0.25;+ let r =+ evaluate::polynomial(xs, ERF_INV_IMPL_BN) / evaluate::polynomial(xs, ERF_INV_IMPL_BD);+ g / (y + r)+ } else {+ let x = (-q.ln()).sqrt();+ if x < 3.0 {+ let y = 0.807220458984375;+ let xs = x - 1.125;+ let r = evaluate::polynomial(xs, ERF_INV_IMPL_CN)+ / evaluate::polynomial(xs, ERF_INV_IMPL_CD);+ y * x + r * x+ } else if x < 6.0 {+ let y = 0.93995571136474609375;+ let xs = x - 3.0;+ let r = evaluate::polynomial(xs, ERF_INV_IMPL_DN)+ / evaluate::polynomial(xs, ERF_INV_IMPL_DD);+ y * x + r * x+ } else if x < 18.0 {+ let y = 0.98362827301025390625;+ let xs = x - 6.0;+ let r = evaluate::polynomial(xs, ERF_INV_IMPL_EN)+ / evaluate::polynomial(xs, ERF_INV_IMPL_ED);+ y * x + r * x+ } else if x < 44.0 {+ let y = 0.99714565277099609375;+ let xs = x - 18.0;+ let r = evaluate::polynomial(xs, ERF_INV_IMPL_FN)+ / evaluate::polynomial(xs, ERF_INV_IMPL_FD);+ y * x + r * x+ } else {+ let y = 0.99941349029541015625;+ let xs = x - 44.0;+ let r = evaluate::polynomial(xs, ERF_INV_IMPL_GN)+ / evaluate::polynomial(xs, ERF_INV_IMPL_GD);+ y * x + r * x+ }+ };+ s * result+}diff --git a/vendor/candle-core/src/cpu/features.rs b/vendor/candle-core/src/cpu/features.rsnew file mode 100644index 0000000..1384b13--- /dev/null+++ b/vendor/candle-core/src/cpu/features.rs@@ -0,0 +1,72 @@+use std::sync::OnceLock;++#[allow(dead_code)]+#[derive(Clone, Copy, Debug, Default)]+pub(crate) struct CpuFeatures {+ pub(crate) avx: bool,+ pub(crate) avx2: bool,+ pub(crate) fma: bool,+ pub(crate) f16c: bool,+ pub(crate) avx512f: bool,+ pub(crate) avx512bw: bool,+ pub(crate) avx512vl: bool,+ pub(crate) avx512bf16: bool,+ pub(crate) avx512vnni: bool,+ pub(crate) neon: bool,+ pub(crate) dotprod: bool,+ pub(crate) i8mm: bool,+ pub(crate) fp16: bool,+ pub(crate) bf16: bool,+}++pub(crate) fn get() -> CpuFeatures {+ static FEATURES: OnceLock<CpuFeatures> = OnceLock::new();+ *FEATURES.get_or_init(detect)+}++#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+fn detect() -> CpuFeatures {+ CpuFeatures {+ avx: std::is_x86_feature_detected!("avx"),+ avx2: std::is_x86_feature_detected!("avx2"),+ fma: std::is_x86_feature_detected!("fma"),+ f16c: std::is_x86_feature_detected!("f16c"),+ avx512f: std::is_x86_feature_detected!("avx512f"),+ avx512bw: std::is_x86_feature_detected!("avx512bw"),+ avx512vl: std::is_x86_feature_detected!("avx512vl"),+ avx512bf16: std::is_x86_feature_detected!("avx512bf16"),+ avx512vnni: std::is_x86_feature_detected!("avx512vnni"),+ ..CpuFeatures::default()+ }+}++#[cfg(target_arch = "aarch64")]+fn detect() -> CpuFeatures {+ CpuFeatures {+ neon: true,+ dotprod: std::arch::is_aarch64_feature_detected!("dotprod"),+ i8mm: std::arch::is_aarch64_feature_detected!("i8mm"),+ fp16: std::arch::is_aarch64_feature_detected!("fp16"),+ bf16: std::arch::is_aarch64_feature_detected!("bf16"),+ ..CpuFeatures::default()+ }+}++#[cfg(all(target_arch = "arm", not(target_arch = "aarch64")))]+fn detect() -> CpuFeatures {+ CpuFeatures {+ neon: cfg!(target_feature = "neon"),+ fp16: cfg!(target_feature = "fp16"),+ ..CpuFeatures::default()+ }+}++#[cfg(not(any(+ target_arch = "x86",+ target_arch = "x86_64",+ target_arch = "arm",+ target_arch = "aarch64"+)))]+fn detect() -> CpuFeatures {+ CpuFeatures::default()+}diff --git a/vendor/candle-core/src/cpu/kernels.rs b/vendor/candle-core/src/cpu/kernels.rsnew file mode 100644index 0000000..9928c8a--- /dev/null+++ b/vendor/candle-core/src/cpu/kernels.rs@@ -0,0 +1,307 @@+pub trait VecOps: num_traits::NumAssign + Copy {+ fn min(self, rhs: Self) -> Self;+ fn max(self, rhs: Self) -> Self;++ /// Element-wise addition of two slices into a third.+ #[inline(always)]+ fn vec_add(lhs: &[Self], rhs: &[Self], res: &mut [Self]) {+ lhs.iter()+ .zip(rhs)+ .zip(res)+ .for_each(|((&a, &b), y)| *y = a + b)+ }++ fn vec_mul(lhs: &[Self], rhs: &[Self], res: &mut [Self]) {+ lhs.iter()+ .zip(rhs)+ .zip(res)+ .for_each(|((&a, &b), y)| *y = a * b)+ }++ #[inline(always)]+ fn scalar_mul(scalar: Self, xs: &[Self], ys: &mut [Self]) {+ xs.iter().zip(ys).for_each(|(&x, y)| *y = x * scalar)+ }++ /// Add a broadcast scalar to every element of a slice: `ys[i] = xs[i] + scalar`.+ #[inline(always)]+ fn scalar_add(scalar: Self, xs: &[Self], ys: &mut [Self]) {+ xs.iter().zip(ys).for_each(|(&x, y)| *y = x + scalar)+ }++ /// Dot-product of two vectors.+ ///+ /// # Safety+ ///+ /// The length of `lhs` and `rhs` have to be at least `len`. `res` has to point to a valid+ /// element.+ #[inline(always)]+ unsafe fn vec_dot(lhs: *const Self, rhs: *const Self, res: *mut Self, len: usize) {+ *res = Self::zero();+ for i in 0..len {+ *res += *lhs.add(i) * *rhs.add(i)+ }+ }++ /// Sum of all elements in a vector.+ ///+ /// # Safety+ ///+ /// The length of `xs` must be at least `len`. `res` has to point to a valid+ /// element.+ #[inline(always)]+ unsafe fn vec_reduce_sum(xs: *const Self, res: *mut Self, len: usize) {+ *res = Self::zero();+ for i in 0..len {+ *res += *xs.add(i)+ }+ }++ /// Maximum element in a non-empty vector.+ ///+ /// # Safety+ ///+ /// The length of `xs` must be at least `len` and positive. `res` has to point to a valid+ /// element.+ #[inline(always)]+ unsafe fn vec_reduce_max(xs: *const Self, res: *mut Self, len: usize) {+ *res = *xs;+ for i in 1..len {+ *res = (*res).max(*xs.add(i))+ }+ }++ /// Minimum element in a non-empty vector.+ ///+ /// # Safety+ ///+ /// The length of `xs` must be at least `len` and positive. `res` has to point to a valid+ /// element.+ #[inline(always)]+ unsafe fn vec_reduce_min(xs: *const Self, res: *mut Self, len: usize) {+ *res = *xs;+ for i in 1..len {+ *res = (*res).min(*xs.add(i))+ }+ }+}++impl VecOps for f32 {+ #[inline(always)]+ fn min(self, other: Self) -> Self {+ Self::min(self, other)+ }++ #[inline(always)]+ fn max(self, other: Self) -> Self {+ Self::max(self, other)+ }++ fn vec_add(lhs: &[Self], rhs: &[Self], res: &mut [Self]) {+ #[cfg(feature = "mkl")]+ crate::mkl::vs_add(lhs, rhs, res);+ #[cfg(all(feature = "accelerate", not(feature = "mkl")))]+ crate::accelerate::vs_add(lhs, rhs, res);+ #[cfg(not(any(feature = "mkl", feature = "accelerate")))]+ lhs.iter()+ .zip(rhs)+ .zip(res)+ .for_each(|((&a, &b), y)| *y = a + b)+ }++ #[inline(always)]+ unsafe fn vec_dot(lhs: *const Self, rhs: *const Self, res: *mut Self, len: usize) {+ super::vec_dot_f32(lhs, rhs, res, len)+ }++ #[inline(always)]+ unsafe fn vec_reduce_sum(xs: *const Self, res: *mut Self, len: usize) {+ super::vec_sum(xs, res, len)+ }+}++impl VecOps for half::f16 {+ #[inline(always)]+ fn min(self, other: Self) -> Self {+ Self::min(self, other)+ }++ #[inline(always)]+ fn max(self, other: Self) -> Self {+ Self::max(self, other)+ }++ fn vec_add(lhs: &[Self], rhs: &[Self], res: &mut [Self]) {+ unsafe { super::vec_add_f16(lhs.as_ptr(), rhs.as_ptr(), res.as_mut_ptr(), lhs.len()) }+ }++ #[inline(always)]+ fn scalar_add(scalar: Self, xs: &[Self], ys: &mut [Self]) {+ unsafe { super::vec_scalar_add_f16(scalar, xs.as_ptr(), ys.as_mut_ptr(), xs.len()) }+ }++ #[inline(always)]+ unsafe fn vec_dot(lhs: *const Self, rhs: *const Self, res: *mut Self, len: usize) {+ let mut res_f32 = 0f32;+ super::vec_dot_f16(lhs, rhs, &mut res_f32, len);+ *res = half::f16::from_f32(res_f32);+ }+}++impl VecOps for f64 {+ #[inline(always)]+ fn min(self, other: Self) -> Self {+ Self::min(self, other)+ }++ #[inline(always)]+ fn max(self, other: Self) -> Self {+ Self::max(self, other)+ }++ #[inline(always)]+ fn vec_add(lhs: &[f64], rhs: &[f64], res: &mut [f64]) {+ #[cfg(feature = "mkl")]+ crate::mkl::vd_add(lhs, rhs, res);+ #[cfg(all(feature = "accelerate", not(feature = "mkl")))]+ crate::accelerate::vd_add(lhs, rhs, res);+ #[cfg(not(any(feature = "mkl", feature = "accelerate")))]+ lhs.iter()+ .zip(rhs)+ .zip(res)+ .for_each(|((&a, &b), y)| *y = a + b)+ }+}+impl VecOps for half::bf16 {+ #[inline(always)]+ fn min(self, other: Self) -> Self {+ Self::min(self, other)+ }++ #[inline(always)]+ fn max(self, other: Self) -> Self {+ Self::max(self, other)+ }++ #[inline(always)]+ fn vec_add(lhs: &[Self], rhs: &[Self], res: &mut [Self]) {+ unsafe { super::vec_add_bf16(lhs.as_ptr(), rhs.as_ptr(), res.as_mut_ptr(), lhs.len()) }+ }++ fn vec_mul(lhs: &[Self], rhs: &[Self], res: &mut [Self]) {+ unsafe { super::vec_mul_bf16(lhs.as_ptr(), rhs.as_ptr(), res.as_mut_ptr(), lhs.len()) }+ }++ #[inline(always)]+ fn scalar_add(scalar: Self, xs: &[Self], ys: &mut [Self]) {+ unsafe { super::vec_scalar_add_bf16(scalar, xs.as_ptr(), ys.as_mut_ptr(), xs.len()) }+ }++ #[inline(always)]+ unsafe fn vec_dot(lhs: *const Self, rhs: *const Self, res: *mut Self, len: usize) {+ let mut res_f32 = 0f32;+ super::vec_dot_bf16(lhs, rhs, &mut res_f32, len);+ *res = half::bf16::from_f32(res_f32);+ }+}+impl VecOps for u8 {+ #[inline(always)]+ fn min(self, other: Self) -> Self {+ <Self as Ord>::min(self, other)+ }++ #[inline(always)]+ fn max(self, other: Self) -> Self {+ <Self as Ord>::max(self, other)+ }+}+impl VecOps for u32 {+ #[inline(always)]+ fn min(self, other: Self) -> Self {+ <Self as Ord>::min(self, other)+ }++ #[inline(always)]+ fn max(self, other: Self) -> Self {+ <Self as Ord>::max(self, other)+ }+}+impl VecOps for i16 {+ #[inline(always)]+ fn min(self, other: Self) -> Self {+ <Self as Ord>::min(self, other)+ }++ #[inline(always)]+ fn max(self, other: Self) -> Self {+ <Self as Ord>::max(self, other)+ }+}+impl VecOps for i32 {+ #[inline(always)]+ fn min(self, other: Self) -> Self {+ <Self as Ord>::min(self, other)+ }++ #[inline(always)]+ fn max(self, other: Self) -> Self {+ <Self as Ord>::max(self, other)+ }+}+impl VecOps for i64 {+ #[inline(always)]+ fn min(self, other: Self) -> Self {+ <Self as Ord>::min(self, other)+ }++ #[inline(always)]+ fn max(self, other: Self) -> Self {+ <Self as Ord>::max(self, other)+ }+}++impl VecOps for float8::F8E4M3 {+ #[inline(always)]+ fn min(self, other: Self) -> Self {+ Self::min(self, other)+ }++ #[inline(always)]+ fn max(self, other: Self) -> Self {+ Self::max(self, other)+ }+}++#[inline(always)]+pub fn par_for_each(n_threads: usize, func: impl Fn(usize) + Send + Sync) {+ if n_threads == 1 {+ func(0)+ } else {+ rayon::scope(|s| {+ for thread_idx in 0..n_threads {+ let func = &func;+ s.spawn(move |_| func(thread_idx));+ }+ })+ }+}++#[inline(always)]+pub fn par_range(lo: usize, up: usize, n_threads: usize, func: impl Fn(usize) + Send + Sync) {+ if n_threads == 1 {+ for i in lo..up {+ func(i)+ }+ } else {+ rayon::scope(|s| {+ for thread_idx in 0..n_threads {+ let func = &func;+ s.spawn(move |_| {+ for i in (thread_idx..up).step_by(n_threads) {+ func(i)+ }+ });+ }+ })+ }+}diff --git a/vendor/candle-core/src/cpu/mod.rs b/vendor/candle-core/src/cpu/mod.rsnew file mode 100644index 0000000..afc684f--- /dev/null+++ b/vendor/candle-core/src/cpu/mod.rs@@ -0,0 +1,370 @@+//! Traits and methods for CPU-backed Tensors+#![allow(clippy::needless_range_loop)]+pub mod erf;+pub(crate) mod features;+pub mod kernels;++#[allow(unused)]+trait Cpu {+ type Unit;+ type Array;+ const STEP: usize;+ const EPR: usize;+ const ARR: usize = Self::STEP / Self::EPR;++ unsafe fn zero() -> Self::Unit;+ unsafe fn zero_array() -> Self::Array;+ unsafe fn load(mem_addr: *const f32) -> Self::Unit;+ unsafe fn vec_add(a: Self::Unit, b: Self::Unit) -> Self::Unit;+ unsafe fn vec_fma(a: Self::Unit, b: Self::Unit, c: Self::Unit) -> Self::Unit;+ unsafe fn vec_reduce(x: Self::Array, y: *mut f32);+ unsafe fn from_f32(v: f32) -> Self::Unit;+ unsafe fn vec_store(mem_addr: *mut f32, a: Self::Unit);+}++#[allow(unused)]+trait CpuF16 {+ type Unit;+ type Array;+ const STEP: usize;+ const EPR: usize;+ const ARR: usize = Self::STEP / Self::EPR;+ // How many outer loop steps to accumulate in Self::Unit before flushing to f32.+ // Set to a finite value to bound rounding error (16 for example).+ const FLUSH_INTERVAL: usize = usize::MAX;++ unsafe fn zero() -> Self::Unit;+ unsafe fn zero_array() -> Self::Array;+ unsafe fn load(mem_addr: *const f16) -> Self::Unit;+ unsafe fn vec_add(a: Self::Unit, b: Self::Unit) -> Self::Unit;+ unsafe fn vec_fma(a: Self::Unit, b: Self::Unit, c: Self::Unit) -> Self::Unit;+ unsafe fn vec_reduce(x: Self::Array, y: *mut f32);+ unsafe fn from_f32(v: f32) -> Self::Unit;+ unsafe fn vec_store(mem_addr: *mut f16, a: Self::Unit);+}++#[allow(unused)]+trait CpuBF16 {+ type Unit;+ type Array;+ const STEP: usize;+ const EPR: usize;+ const ARR: usize = Self::STEP / Self::EPR;++ unsafe fn zero() -> Self::Unit;+ unsafe fn zero_array() -> Self::Array;+ unsafe fn load(mem_addr: *const bf16) -> Self::Unit;+ unsafe fn vec_add(a: Self::Unit, b: Self::Unit) -> Self::Unit;+ unsafe fn vec_fma(a: Self::Unit, b: Self::Unit, c: Self::Unit) -> Self::Unit;+ unsafe fn vec_reduce(x: Self::Array, y: *mut f32);+ unsafe fn from_f32(v: f32) -> Self::Unit;+ unsafe fn vec_store(mem_addr: *mut bf16, a: Self::Unit);+}++use half::{bf16, f16};++#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+pub mod avx;++#[cfg(target_arch = "wasm32")]+#[cfg(target_feature = "simd128")]+pub mod simd128;++#[cfg(any(+ target_arch = "aarch64",+ all(target_arch = "arm", target_feature = "neon")+))]+pub mod neon;++#[inline(always)]+#[cfg(not(all(target_arch = "wasm32", target_feature = "simd128")))]+unsafe fn scalar_vec_dot_f32(a_row: *const f32, b_row: *const f32, c: *mut f32, k: usize) {+ for i in 0..k {+ *c += *a_row.add(i) * (*b_row.add(i));+ }+}++#[inline(always)]+#[cfg(not(all(target_arch = "wasm32", target_feature = "simd128")))]+unsafe fn scalar_vec_sum(row: *const f32, b: *mut f32, k: usize) {+ *b = 0f32;+ for i in 0..k {+ *b += *row.add(i)+ }+}++#[inline(always)]+unsafe fn scalar_vec_dot_f16(a_row: *const f16, b_row: *const f16, c: *mut f32, k: usize) {+ let mut sum = 0.0;+ for i in 0..k {+ sum += (*a_row.add(i)).to_f32() * (*b_row.add(i)).to_f32();+ }+ *c = sum;+}++#[inline(always)]+unsafe fn scalar_vec_dot_bf16(a_row: *const bf16, b_row: *const bf16, c: *mut f32, k: usize) {+ let mut sum = 0.0;+ for i in 0..k {+ sum += (*a_row.add(i)).to_f32() * (*b_row.add(i)).to_f32();+ }+ *c = sum;+}++#[inline(always)]+#[cfg(not(all(target_arch = "wasm32", target_feature = "simd128")))]+unsafe fn scalar_vec_add_f16(a_row: *const f16, b_row: *const f16, c: *mut f16, k: usize) {+ for i in 0..k {+ *c.add(i) = *a_row.add(i) + *b_row.add(i);+ }+}++#[inline(always)]+#[cfg(not(all(target_arch = "wasm32", target_feature = "simd128")))]+unsafe fn scalar_vec_add_bf16(a_row: *const bf16, b_row: *const bf16, c: *mut bf16, k: usize) {+ for i in 0..k {+ *c.add(i) = *a_row.add(i) + *b_row.add(i);+ }+}++#[inline(always)]+#[cfg(not(all(target_arch = "wasm32", target_feature = "simd128")))]+unsafe fn scalar_vec_scalar_add_f16(scalar: f16, xs: *const f16, ys: *mut f16, k: usize) {+ for i in 0..k {+ *ys.add(i) = *xs.add(i) + scalar;+ }+}++#[inline(always)]+#[cfg(not(all(target_arch = "wasm32", target_feature = "simd128")))]+unsafe fn scalar_vec_scalar_add_bf16(scalar: bf16, xs: *const bf16, ys: *mut bf16, k: usize) {+ for i in 0..k {+ *ys.add(i) = *xs.add(i) + scalar;+ }+}++#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+#[inline(always)]+fn use_avx2() -> bool {+ features::get().avx2+}++#[cfg(any(+ target_arch = "aarch64",+ all(target_arch = "arm", target_feature = "neon")+))]+#[inline(always)]+fn use_neon() -> bool {+ features::get().neon+}++#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+#[inline(always)]+fn use_avx2_f16() -> bool {+ let features = features::get();+ features.avx2 && (!cfg!(target_feature = "f16c") || features.f16c)+}++#[cfg(any(+ target_arch = "aarch64",+ all(target_arch = "arm", target_feature = "neon")+))]+#[inline(always)]+fn use_neon_f16() -> bool {+ let features = features::get();+ features.neon && (!cfg!(target_feature = "fp16") || features.fp16)+}++#[cfg(any(+ target_arch = "aarch64",+ all(target_arch = "arm", target_feature = "neon")+))]+#[inline(always)]+fn use_neon_bf16() -> bool {+ let features = features::get();+ features.neon && (!cfg!(target_feature = "bf16") || features.bf16)+}++#[inline(always)]+pub(crate) unsafe fn vec_dot_f32(a_row: *const f32, b_row: *const f32, c: *mut f32, k: usize) {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if use_avx2() {+ return avx::vec_dot_f32(a_row, b_row, c, k);+ }+ #[cfg(any(+ target_arch = "aarch64",+ all(target_arch = "arm", target_feature = "neon")+ ))]+ if use_neon() {+ return neon::vec_dot_f32(a_row, b_row, c, k);+ }+ #[cfg(all(target_arch = "wasm32", target_feature = "simd128"))]+ {+ simd128::vec_dot_f32(a_row, b_row, c, k)+ }+ #[cfg(not(all(target_arch = "wasm32", target_feature = "simd128")))]+ {+ scalar_vec_dot_f32(a_row, b_row, c, k)+ }+}++#[inline(always)]+pub(crate) unsafe fn vec_sum(row: *const f32, b: *mut f32, k: usize) {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if use_avx2() {+ return avx::vec_sum(row, b, k);+ }+ #[cfg(any(+ target_arch = "aarch64",+ all(target_arch = "arm", target_feature = "neon")+ ))]+ if use_neon() {+ return neon::vec_sum(row, b, k);+ }+ #[cfg(all(target_arch = "wasm32", target_feature = "simd128"))]+ {+ simd128::vec_sum(row, b, k)+ }+ #[cfg(not(all(target_arch = "wasm32", target_feature = "simd128")))]+ {+ scalar_vec_sum(row, b, k)+ }+}++#[inline(always)]+pub(crate) unsafe fn vec_dot_f16(a_row: *const f16, b_row: *const f16, c: *mut f32, k: usize) {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if use_avx2_f16() {+ return avx::vec_dot_f16(a_row, b_row, c, k);+ }+ #[cfg(any(+ target_arch = "aarch64",+ all(target_arch = "arm", target_feature = "neon")+ ))]+ if use_neon_f16() {+ return neon::vec_dot_f16(a_row, b_row, c, k);+ }+ scalar_vec_dot_f16(a_row, b_row, c, k)+}++#[inline(always)]+pub(crate) unsafe fn vec_dot_bf16(a_row: *const bf16, b_row: *const bf16, c: *mut f32, k: usize) {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if use_avx2() {+ return avx::vec_dot_bf16(a_row, b_row, c, k);+ }+ #[cfg(any(+ target_arch = "aarch64",+ all(target_arch = "arm", target_feature = "neon")+ ))]+ if use_neon_bf16() {+ return neon::vec_dot_bf16(a_row, b_row, c, k);+ }+ scalar_vec_dot_bf16(a_row, b_row, c, k)+}++#[inline(always)]+pub(crate) unsafe fn vec_add_f16(a_row: *const f16, b_row: *const f16, c: *mut f16, k: usize) {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if use_avx2_f16() {+ return avx::vec_add_f16(a_row, b_row, c, k);+ }+ #[cfg(any(+ target_arch = "aarch64",+ all(target_arch = "arm", target_feature = "neon")+ ))]+ if use_neon_f16() {+ return neon::vec_add_f16(a_row, b_row, c, k);+ }+ #[cfg(all(target_arch = "wasm32", target_feature = "simd128"))]+ {+ simd128::vec_add_f16(a_row, b_row, c, k)+ }+ #[cfg(not(all(target_arch = "wasm32", target_feature = "simd128")))]+ {+ scalar_vec_add_f16(a_row, b_row, c, k)+ }+}++#[inline(always)]+pub(crate) unsafe fn vec_add_bf16(a_row: *const bf16, b_row: *const bf16, c: *mut bf16, k: usize) {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if use_avx2() {+ return avx::vec_add_bf16(a_row, b_row, c, k);+ }+ #[cfg(any(+ target_arch = "aarch64",+ all(target_arch = "arm", target_feature = "neon")+ ))]+ if use_neon_bf16() {+ return neon::vec_add_bf16(a_row, b_row, c, k);+ }+ #[cfg(all(target_arch = "wasm32", target_feature = "simd128"))]+ {+ simd128::vec_add_bf16(a_row, b_row, c, k)+ }+ #[cfg(not(all(target_arch = "wasm32", target_feature = "simd128")))]+ {+ scalar_vec_add_bf16(a_row, b_row, c, k)+ }+}++#[inline(always)]+pub(crate) unsafe fn vec_scalar_add_f16(scalar: f16, xs: *const f16, ys: *mut f16, k: usize) {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if use_avx2_f16() {+ return avx::vec_scalar_add_f16(scalar, xs, ys, k);+ }+ #[cfg(any(+ target_arch = "aarch64",+ all(target_arch = "arm", target_feature = "neon")+ ))]+ if use_neon_f16() {+ return neon::vec_scalar_add_f16(scalar, xs, ys, k);+ }+ #[cfg(all(target_arch = "wasm32", target_feature = "simd128"))]+ {+ simd128::vec_scalar_add_f16(scalar, xs, ys, k)+ }+ #[cfg(not(all(target_arch = "wasm32", target_feature = "simd128")))]+ {+ scalar_vec_scalar_add_f16(scalar, xs, ys, k)+ }+}++#[inline(always)]+pub(crate) unsafe fn vec_mul_bf16(a_row: *const bf16, b_row: *const bf16, c: *mut bf16, k: usize) {+ #[cfg(any(+ target_arch = "aarch64",+ all(target_arch = "arm", target_feature = "neon")+ ))]+ if use_neon_bf16() {+ return neon::vec_mul_bf16(a_row, b_row, c, k);+ }+ for j in 0..k {+ *c.add(j) = *a_row.add(j) * *b_row.add(j);+ }+}++#[inline(always)]+pub(crate) unsafe fn vec_scalar_add_bf16(scalar: bf16, xs: *const bf16, ys: *mut bf16, k: usize) {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if use_avx2() {+ return avx::vec_scalar_add_bf16(scalar, xs, ys, k);+ }+ #[cfg(any(+ target_arch = "aarch64",+ all(target_arch = "arm", target_feature = "neon")+ ))]+ if use_neon_bf16() {+ return neon::vec_scalar_add_bf16(scalar, xs, ys, k);+ }+ #[cfg(all(target_arch = "wasm32", target_feature = "simd128"))]+ {+ simd128::vec_scalar_add_bf16(scalar, xs, ys, k)+ }+ #[cfg(not(all(target_arch = "wasm32", target_feature = "simd128")))]+ {+ scalar_vec_scalar_add_bf16(scalar, xs, ys, k)+ }+}diff --git a/vendor/candle-core/src/cpu/neon.rs b/vendor/candle-core/src/cpu/neon.rsnew file mode 100644index 0000000..5b5fa96--- /dev/null+++ b/vendor/candle-core/src/cpu/neon.rs@@ -0,0 +1,611 @@+use super::{Cpu, CpuBF16, CpuF16};+#[cfg(target_arch = "arm")]+use core::arch::arm::*;++#[cfg(target_arch = "aarch64")]+use core::arch::aarch64::*;+use half::{bf16, f16};++pub struct CurrentCpu {}++impl CurrentCpu {+ #[cfg(target_arch = "aarch64")]+ unsafe fn reduce_one(x: float32x4_t) -> f32 {+ vaddvq_f32(x)+ }++ #[cfg(target_arch = "arm")]+ unsafe fn reduce_one(x: float32x4_t) -> f32 {+ vgetq_lane_f32(x, 0) + vgetq_lane_f32(x, 1) + vgetq_lane_f32(x, 2) + vgetq_lane_f32(x, 3)+ }+}++impl Cpu for CurrentCpu {+ type Unit = float32x4_t;+ type Array = [float32x4_t; Self::ARR];++ const STEP: usize = 32;+ const EPR: usize = 4;++ unsafe fn zero() -> Self::Unit {+ vdupq_n_f32(0.0)+ }++ unsafe fn from_f32(x: f32) -> Self::Unit {+ vdupq_n_f32(x)+ }++ unsafe fn zero_array() -> Self::Array {+ [Self::zero(); Self::ARR]+ }++ unsafe fn load(mem_addr: *const f32) -> Self::Unit {+ vld1q_f32(mem_addr)+ }++ unsafe fn vec_add(a: Self::Unit, b: Self::Unit) -> Self::Unit {+ vaddq_f32(a, b)+ }++ unsafe fn vec_fma(a: Self::Unit, b: Self::Unit, c: Self::Unit) -> Self::Unit {+ vfmaq_f32(a, b, c)+ }++ unsafe fn vec_store(mem_addr: *mut f32, a: Self::Unit) {+ vst1q_f32(mem_addr, a);+ }++ unsafe fn vec_reduce(mut x: Self::Array, y: *mut f32) {+ for i in 0..Self::ARR / 2 {+ x[2 * i] = vaddq_f32(x[2 * i], x[2 * i + 1]);+ }+ for i in 0..Self::ARR / 4 {+ x[4 * i] = vaddq_f32(x[4 * i], x[4 * i + 2]);+ }+ for i in 0..Self::ARR / 8 {+ x[8 * i] = vaddq_f32(x[8 * i], x[8 * i + 4]);+ }+ *y = Self::reduce_one(x[0]);+ }+}++mod fp16 {+ use super::super::CpuF16;+ #[cfg(target_arch = "aarch64")]+ use core::arch::aarch64::*;+ #[cfg(target_arch = "arm")]+ use core::arch::arm::*;+ use half::f16;+ use std::arch::asm;++ // Fallback: widen f16 to f32 on load using FCVTL, accumulate in f32.+ #[cfg(not(target_feature = "fp16"))]+ mod inner {+ use super::*;++ pub struct CurrentCpuF16 {}++ impl CpuF16 for CurrentCpuF16 {+ type Unit = float32x4_t;+ type Array = [float32x4_t; Self::ARR];++ const STEP: usize = 32;+ const EPR: usize = 4;++ unsafe fn zero() -> Self::Unit {+ vdupq_n_f32(0.0)+ }++ unsafe fn zero_array() -> Self::Array {+ [Self::zero(); Self::ARR]+ }++ unsafe fn load(mem_addr: *const f16) -> Self::Unit {+ let bits = vld1_u16(mem_addr as *const u16);+ let result: Self::Unit;+ asm!(+ "fcvtl {dst:v}.4s, {src:v}.4h",+ src = in(vreg) bits,+ dst = out(vreg) result,+ options(pure, nomem, nostack),+ );+ result+ }++ unsafe fn vec_add(a: Self::Unit, b: Self::Unit) -> Self::Unit {+ vaddq_f32(a, b)+ }++ unsafe fn vec_fma(a: Self::Unit, b: Self::Unit, c: Self::Unit) -> Self::Unit {+ vfmaq_f32(a, b, c)+ }++ unsafe fn vec_reduce(mut x: Self::Array, y: *mut f32) {+ for i in 0..Self::ARR / 2 {+ x[2 * i] = vaddq_f32(x[2 * i], x[2 * i + 1]);+ }+ for i in 0..Self::ARR / 4 {+ x[4 * i] = vaddq_f32(x[4 * i], x[4 * i + 2]);+ }+ for i in 0..Self::ARR / 8 {+ x[8 * i] = vaddq_f32(x[8 * i], x[8 * i + 4]);+ }+ *y = vaddvq_f32(x[0]);+ }++ unsafe fn from_f32(v: f32) -> Self::Unit {+ vdupq_n_f32(v)+ }++ unsafe fn vec_store(mem_addr: *mut f16, a: Self::Unit) {+ let bits: uint16x4_t;+ asm!(+ "fcvtn {dst:v}.4h, {src:v}.4s",+ src = in(vreg) a,+ dst = out(vreg) bits,+ options(pure, nomem, nostack),+ );+ vst1_u16(mem_addr as *mut u16, bits);+ }+ }+ }++ // Optimized: accumulate in f16 using FMLA.8h, flush to f32 periodically.+ #[cfg(target_feature = "fp16")]+ mod inner {+ use super::*;++ pub struct CurrentCpuF16 {}++ impl CurrentCpuF16 {+ fn reduce_one(x: uint16x8_t) -> f32 {+ let result: f32;+ unsafe {+ asm!(+ // 1. Widen bottom 4 lanes of f16x8 to a full f32x4 vector+ "fcvtl {bot:v}.4s, {v:v}.4h",+ // 2. Widen top 4 lanes of f16x8 to a second f32x4 vector+ "fcvtl2 {top:v}.4s, {v:v}.8h",+ // 3. Perform a parallel element-wise vector addition+ "fadd {bot:v}.4s, {bot:v}.4s, {top:v}.4s",+ // 4. Pairwise add 4 lanes down to 2 lanes: [A+B, C+D, ?, ?]+ "faddp {bot:v}.4s, {bot:v}.4s, {bot:v}.4s",+ // 5. Pairwise add 2 lanes down to 1 scalar: [(A+B)+(C+D), ?, ?, ?]+ "faddp {bot:v}.4s, {bot:v}.4s, {bot:v}.4s",+ // 6. Extract the scalar from the lowest lane of the vector+ "mov {res:s}, {bot:v}.s[0]",+ v = in(vreg) x,+ top = lateout(vreg) _,+ bot = out(vreg) _,+ res = out(vreg) result,+ options(nostack, preserves_flags)+ );+ }+ result+ }+ }++ impl CpuF16 for CurrentCpuF16 {+ // Keep the f16 lane bits in the stable integer vector type. The+ // inline assembly assigns the floating-point interpretation.+ type Unit = uint16x8_t;+ type Array = [uint16x8_t; Self::ARR];++ const STEP: usize = 32;+ const EPR: usize = 8;+ // Flush f16 accumulators to f32 every `FLUSH_INTERVAL` steps to bound rounding error.+ const FLUSH_INTERVAL: usize = 16;++ unsafe fn zero() -> Self::Unit {+ let result: Self::Unit;+ asm!(+ "movi {v:v}.8h, #0",+ v = out(vreg) result,+ options(nostack, pure, nomem)+ );+ result+ }++ unsafe fn zero_array() -> Self::Array {+ [Self::zero(); Self::ARR]+ }++ unsafe fn load(mem_addr: *const f16) -> Self::Unit {+ std::ptr::read_unaligned(mem_addr.cast())+ }++ unsafe fn vec_add(a: Self::Unit, b: Self::Unit) -> Self::Unit {+ let result: Self::Unit;+ asm!(+ "fadd {v:v}.8h, {a:v}.8h, {b:v}.8h",+ v = out(vreg) result,+ a = in(vreg) a,+ b = in(vreg) b,+ options(pure, nomem, nostack)+ );+ result+ }++ unsafe fn vec_fma(a: Self::Unit, b: Self::Unit, c: Self::Unit) -> Self::Unit {+ let result: Self::Unit;+ asm!(+ "fmla {a:v}.8h, {b:v}.8h, {c:v}.8h",+ a = inout(vreg) a => result,+ b = in(vreg) b,+ c = in(vreg) c,+ options(pure, nomem, nostack)+ );+ result+ }++ unsafe fn vec_reduce(x: Self::Array, y: *mut f32) {+ let mut sum = 0.0f32;+ for item in x {+ sum += Self::reduce_one(item);+ }+ *y = sum;+ }++ unsafe fn from_f32(v: f32) -> Self::Unit {+ let result: Self::Unit;+ asm!(+ "fcvt {tmp:h}, {src:s}",+ "fmov {w:w}, {tmp:h}",+ "dup {dst:v}.8h, {w:w}",+ src = in(vreg) v,+ tmp = out(vreg) _,+ w = out(reg) _,+ dst = out(vreg) result,+ options(nostack, pure, nomem),+ );+ result+ }++ unsafe fn vec_store(mem_addr: *mut f16, a: Self::Unit) {+ asm!(+ "st1 {{ {vec:v}.8h }}, [{ptr}]",+ vec = in(vreg) a,+ ptr = in(reg) mem_addr,+ options(nostack, preserves_flags)+ );+ }+ }+ }++ pub use inner::CurrentCpuF16;+}++pub use fp16::CurrentCpuF16;++mod bfloat16 {+ use super::super::CpuBF16;+ use core::arch::aarch64::*;+ use half::bf16;++ // Fallback: accumulate in f32 using SHLL to widen bf16 to f32 on load.+ #[cfg(not(target_feature = "bf16"))]+ mod inner {+ use super::*;++ pub struct CurrentCpuBF16 {}++ impl CpuBF16 for CurrentCpuBF16 {+ type Unit = float32x4_t;+ type Array = [float32x4_t; Self::ARR];++ const STEP: usize = 32;+ const EPR: usize = 4;++ unsafe fn zero() -> Self::Unit {+ vdupq_n_f32(0.0)+ }++ unsafe fn zero_array() -> Self::Array {+ [Self::zero(); Self::ARR]+ }++ unsafe fn load(mem_addr: *const bf16) -> Self::Unit {+ // bf16 is the top 16 bits of f32; shift left by 16 to reconstruct f32 bits.+ let i = vld1_u16(mem_addr as *const u16);+ let s = vshll_n_u16::<16>(i);+ vreinterpretq_f32_u32(s)+ }++ unsafe fn vec_add(a: Self::Unit, b: Self::Unit) -> Self::Unit {+ vaddq_f32(a, b)+ }++ unsafe fn vec_fma(a: Self::Unit, b: Self::Unit, c: Self::Unit) -> Self::Unit {+ vfmaq_f32(a, b, c)+ }++ unsafe fn vec_reduce(mut x: Self::Array, y: *mut f32) {+ for i in 0..Self::ARR / 2 {+ x[2 * i] = vaddq_f32(x[2 * i], x[2 * i + 1]);+ }+ for i in 0..Self::ARR / 4 {+ x[4 * i] = vaddq_f32(x[4 * i], x[4 * i + 2]);+ }+ for i in 0..Self::ARR / 8 {+ x[8 * i] = vaddq_f32(x[8 * i], x[8 * i + 4]);+ }+ *y = vaddvq_f32(x[0]);+ }++ unsafe fn from_f32(v: f32) -> Self::Unit {+ vdupq_n_f32(v)+ }++ unsafe fn vec_store(mem_addr: *mut bf16, a: Self::Unit) {+ // Extract top 16 bits of each f32 lane, which is the bf16 representation.+ let s = vreinterpretq_u32_f32(a);+ let h = vshrn_n_u32::<16>(s);+ vst1_u16(mem_addr as *mut u16, h);+ }+ }+ }++ // Optimized: BFDOT accumulates bf16 pairs directly into f32 lanes.+ #[cfg(target_feature = "bf16")]+ mod inner {+ use super::*;+ use std::arch::asm;++ pub struct CurrentCpuBF16 {}++ impl CpuBF16 for CurrentCpuBF16 {+ type Unit = uint16x8_t;+ type Array = [uint16x8_t; Self::ARR];++ const STEP: usize = 32;+ const EPR: usize = 8;++ unsafe fn zero() -> Self::Unit {+ vreinterpretq_u16_f32(vdupq_n_f32(0.0))+ }++ unsafe fn zero_array() -> Self::Array {+ [Self::zero(); Self::ARR]+ }++ unsafe fn load(mem_addr: *const bf16) -> Self::Unit {+ vld1q_u16(mem_addr as *const u16)+ }++ unsafe fn vec_add(a: Self::Unit, b: Self::Unit) -> Self::Unit {+ let result: Self::Unit;+ asm!(+ "fadd {v:v}.4s, {a:v}.4s, {b:v}.4s",+ v = out(vreg) result,+ a = in(vreg) a,+ b = in(vreg) b,+ options(pure, nomem, nostack)+ );+ result+ }++ unsafe fn vec_fma(a: Self::Unit, b: Self::Unit, c: Self::Unit) -> Self::Unit {+ // IMPORTANT: `a` is not bf16x8 here. It is 128 bits used as f32x4 (.4s) to accumulate+ // results from bfdot. `b` and `c` are bf16x8 inputs (.8h).+ // bfdot: acc[i] += (f32)b[2i]*c[2i] + (f32)b[2i+1]*c[2i+1]+ let result: Self::Unit;+ asm!(+ "bfdot {acc:v}.4s, {b:v}.8h, {c:v}.8h",+ acc = inout(vreg) a => result,+ b = in(vreg) b,+ c = in(vreg) c,+ options(pure, nomem, nostack)+ );+ result+ }++ unsafe fn vec_reduce(x: Self::Array, y: *mut f32) {+ // IMPORTANT: `x` is not bf16x8 here. It is 128 bits used as f32x4.+ // This means that if you use `vec_reduce` where `x` is actually bf16x8 you will get+ // entirely wrong results.+ let mut xf: [float32x4_t; Self::ARR] = std::mem::transmute(x);+ for i in 0..Self::ARR / 2 {+ xf[2 * i] = vaddq_f32(xf[2 * i], xf[2 * i + 1]);+ }+ for i in 0..Self::ARR / 4 {+ xf[4 * i] = vaddq_f32(xf[4 * i], xf[4 * i + 2]);+ }+ *y = vaddvq_f32(xf[0]);+ }++ unsafe fn from_f32(v: f32) -> Self::Unit {+ vreinterpretq_u16_f32(vdupq_n_f32(v))+ }++ unsafe fn vec_store(mem_addr: *mut bf16, a: Self::Unit) {+ vst1q_u16(mem_addr as *mut u16, a);+ }+ }+ }++ pub use inner::CurrentCpuBF16;+}++pub use bfloat16::CurrentCpuBF16;++pub(crate) unsafe fn vec_dot_f32(a_row: *const f32, b_row: *const f32, c: *mut f32, k: usize) {+ let mut sum = CurrentCpu::zero_array();+ let mut i = 0;+ while i + CurrentCpu::STEP <= k {+ for j in 0..CurrentCpu::ARR {+ sum[j] = CurrentCpu::vec_fma(+ sum[j],+ CurrentCpu::load(a_row.add(i + j * CurrentCpu::EPR)),+ CurrentCpu::load(b_row.add(i + j * CurrentCpu::EPR)),+ );+ }+ i += CurrentCpu::STEP;+ }+ CurrentCpu::vec_reduce(sum, c);+ while i < k {+ *c += *a_row.add(i) * (*b_row.add(i));+ i += 1;+ }+}++pub(crate) unsafe fn vec_sum(row: *const f32, b: *mut f32, k: usize) {+ let np = k & !(CurrentCpu::STEP - 1);+ let mut sum = CurrentCpu::zero_array();+ let mut x = CurrentCpu::zero_array();+ for i in (0..np).step_by(CurrentCpu::STEP) {+ for j in 0..CurrentCpu::ARR {+ x[j] = CurrentCpu::load(row.add(i + j * CurrentCpu::EPR));+ sum[j] = CurrentCpu::vec_add(sum[j], x[j]);+ }+ }+ CurrentCpu::vec_reduce(sum, b);+ for i in np..k {+ *b += *row.add(i)+ }+}++pub(crate) unsafe fn vec_dot_f16(a_row: *const f16, b_row: *const f16, c: *mut f32, k: usize) {+ let mut sumf = 0.0f32;+ let mut sum = CurrentCpuF16::zero_array();+ let mut steps_since_flush = 0usize;+ let mut i = 0;+ while i + CurrentCpuF16::STEP <= k {+ for j in 0..CurrentCpuF16::ARR {+ sum[j] = CurrentCpuF16::vec_fma(+ sum[j],+ CurrentCpuF16::load(a_row.add(i + j * CurrentCpuF16::EPR)),+ CurrentCpuF16::load(b_row.add(i + j * CurrentCpuF16::EPR)),+ );+ }+ steps_since_flush += 1;+ if steps_since_flush == CurrentCpuF16::FLUSH_INTERVAL {+ let mut partial = 0.0f32;+ CurrentCpuF16::vec_reduce(sum, &mut partial);+ sumf += partial;+ sum = CurrentCpuF16::zero_array();+ steps_since_flush = 0;+ }+ i += CurrentCpuF16::STEP;+ }+ let mut partial = 0.0f32;+ CurrentCpuF16::vec_reduce(sum, &mut partial);+ sumf += partial;+ while i < k {+ sumf += (*a_row.add(i)).to_f32() * (*b_row.add(i)).to_f32();+ i += 1;+ }+ *c = sumf;+}++pub(crate) unsafe fn vec_dot_bf16(a_row: *const bf16, b_row: *const bf16, c: *mut f32, k: usize) {+ let mut sum = CurrentCpuBF16::zero_array();+ let mut i = 0;+ while i + CurrentCpuBF16::STEP <= k {+ for j in 0..CurrentCpuBF16::ARR {+ sum[j] = CurrentCpuBF16::vec_fma(+ sum[j],+ CurrentCpuBF16::load(a_row.add(i + j * CurrentCpuBF16::EPR)),+ CurrentCpuBF16::load(b_row.add(i + j * CurrentCpuBF16::EPR)),+ );+ }+ i += CurrentCpuBF16::STEP;+ }+ CurrentCpuBF16::vec_reduce(sum, c);+ while i < k {+ *c += (*a_row.add(i)).to_f32() * (*b_row.add(i)).to_f32();+ i += 1;+ }+}++pub(crate) unsafe fn vec_add_f16(a_row: *const f16, b_row: *const f16, c: *mut f16, k: usize) {+ let mut i = 0;+ while i + CurrentCpuF16::STEP <= k {+ for j in 0..CurrentCpuF16::ARR {+ CurrentCpuF16::vec_store(+ c.add(i + j * CurrentCpuF16::EPR),+ CurrentCpuF16::vec_add(+ CurrentCpuF16::load(a_row.add(i + j * CurrentCpuF16::EPR)),+ CurrentCpuF16::load(b_row.add(i + j * CurrentCpuF16::EPR)),+ ),+ );+ }+ i += CurrentCpuF16::STEP;+ }+ for j in i..k {+ *c.add(j) = *a_row.add(j) + *b_row.add(j);+ }+}++pub(crate) unsafe fn vec_add_bf16(a_row: *const bf16, b_row: *const bf16, c: *mut bf16, k: usize) {+ // Widen to f32, add, narrow. The bfdot-specialized CurrentCpuBF16 keeps raw bf16 bits in+ // its Unit and its vec_add interprets them as f32 lanes, so it must not be used here.+ let mut i = 0;+ while i + 8 <= k {+ let a = vld1q_u16(a_row.add(i) as *const u16);+ let b = vld1q_u16(b_row.add(i) as *const u16);+ let a_lo = vreinterpretq_f32_u32(vshll_n_u16::<16>(vget_low_u16(a)));+ let a_hi = vreinterpretq_f32_u32(vshll_high_n_u16::<16>(a));+ let b_lo = vreinterpretq_f32_u32(vshll_n_u16::<16>(vget_low_u16(b)));+ let b_hi = vreinterpretq_f32_u32(vshll_high_n_u16::<16>(b));+ let lo = vshrn_n_u32::<16>(vreinterpretq_u32_f32(vaddq_f32(a_lo, b_lo)));+ let hi = vshrn_n_u32::<16>(vreinterpretq_u32_f32(vaddq_f32(a_hi, b_hi)));+ vst1q_u16(c.add(i) as *mut u16, vcombine_u16(lo, hi));+ i += 8;+ }+ for j in i..k {+ *c.add(j) = *a_row.add(j) + *b_row.add(j);+ }+}++pub(crate) unsafe fn vec_scalar_add_f16(scalar: f16, xs: *const f16, ys: *mut f16, k: usize) {+ let sv = CurrentCpuF16::from_f32(scalar.to_f32());+ let mut i = 0;+ while i + CurrentCpuF16::STEP <= k {+ for j in 0..CurrentCpuF16::ARR {+ CurrentCpuF16::vec_store(+ ys.add(i + j * CurrentCpuF16::EPR),+ CurrentCpuF16::vec_add(CurrentCpuF16::load(xs.add(i + j * CurrentCpuF16::EPR)), sv),+ );+ }+ i += CurrentCpuF16::STEP;+ }+ for j in i..k {+ *ys.add(j) = *xs.add(j) + scalar;+ }+}++pub(crate) unsafe fn vec_mul_bf16(a_row: *const bf16, b_row: *const bf16, c: *mut bf16, k: usize) {+ let mut i = 0;+ while i + 8 <= k {+ let a = vld1q_u16(a_row.add(i) as *const u16);+ let b = vld1q_u16(b_row.add(i) as *const u16);+ let a_lo = vreinterpretq_f32_u32(vshll_n_u16::<16>(vget_low_u16(a)));+ let a_hi = vreinterpretq_f32_u32(vshll_high_n_u16::<16>(a));+ let b_lo = vreinterpretq_f32_u32(vshll_n_u16::<16>(vget_low_u16(b)));+ let b_hi = vreinterpretq_f32_u32(vshll_high_n_u16::<16>(b));+ let lo = vshrn_n_u32::<16>(vreinterpretq_u32_f32(vmulq_f32(a_lo, b_lo)));+ let hi = vshrn_n_u32::<16>(vreinterpretq_u32_f32(vmulq_f32(a_hi, b_hi)));+ vst1q_u16(c.add(i) as *mut u16, vcombine_u16(lo, hi));+ i += 8;+ }+ for j in i..k {+ *c.add(j) = *a_row.add(j) * *b_row.add(j);+ }+}++pub(crate) unsafe fn vec_scalar_add_bf16(scalar: bf16, xs: *const bf16, ys: *mut bf16, k: usize) {+ let sv = vdupq_n_f32(scalar.to_f32());+ let mut i = 0;+ while i + 8 <= k {+ let x = vld1q_u16(xs.add(i) as *const u16);+ let x_lo = vreinterpretq_f32_u32(vshll_n_u16::<16>(vget_low_u16(x)));+ let x_hi = vreinterpretq_f32_u32(vshll_high_n_u16::<16>(x));+ let lo = vshrn_n_u32::<16>(vreinterpretq_u32_f32(vaddq_f32(x_lo, sv)));+ let hi = vshrn_n_u32::<16>(vreinterpretq_u32_f32(vaddq_f32(x_hi, sv)));+ vst1q_u16(ys.add(i) as *mut u16, vcombine_u16(lo, hi));+ i += 8;+ }+ for j in i..k {+ *ys.add(j) = *xs.add(j) + scalar;+ }+}diff --git a/vendor/candle-core/src/cpu/simd128.rs b/vendor/candle-core/src/cpu/simd128.rsnew file mode 100644index 0000000..caacd82--- /dev/null+++ b/vendor/candle-core/src/cpu/simd128.rs@@ -0,0 +1,278 @@+use super::{Cpu, CpuBF16, CpuF16};+use core::arch::wasm32::*;+use half::{bf16, f16};++pub struct CurrentCpu {}++const STEP: usize = 16;+const EPR: usize = 4;+const ARR: usize = STEP / EPR;++impl Cpu for CurrentCpu {+ type Unit = v128;+ type Array = [v128; ARR];++ const STEP: usize = STEP;+ const EPR: usize = EPR;+ const ARR: usize = ARR;++ unsafe fn zero() -> Self::Unit {+ f32x4_splat(0.0)+ }++ unsafe fn zero_array() -> Self::Array {+ [Self::zero(); ARR]+ }++ unsafe fn from_f32(v: f32) -> Self::Unit {+ f32x4_splat(v)+ }++ unsafe fn load(mem_addr: *const f32) -> Self::Unit {+ v128_load(mem_addr as *mut v128)+ }++ unsafe fn vec_add(a: Self::Unit, b: Self::Unit) -> Self::Unit {+ f32x4_add(a, b)+ }++ unsafe fn vec_fma(a: Self::Unit, b: Self::Unit, c: Self::Unit) -> Self::Unit {+ f32x4_add(f32x4_mul(b, c), a)+ }++ unsafe fn vec_store(mem_addr: *mut f32, a: Self::Unit) {+ v128_store(mem_addr as *mut v128, a);+ }++ unsafe fn vec_reduce(mut x: Self::Array, y: *mut f32) {+ for i in 0..ARR / 2 {+ x[2 * i] = f32x4_add(x[2 * i], x[2 * i + 1]);+ }+ for i in 0..ARR / 4 {+ x[4 * i] = f32x4_add(x[4 * i], x[4 * i + 2]);+ }+ for i in 0..ARR / 8 {+ x[8 * i] = f32x4_add(x[8 * i], x[8 * i + 4]);+ }+ *y = f32x4_extract_lane::<0>(x[0])+ + f32x4_extract_lane::<1>(x[0])+ + f32x4_extract_lane::<2>(x[0])+ + f32x4_extract_lane::<3>(x[0]);+ }+}++// WebAssembly SIMD has no half-precision arithmetic, so widen f16 lanes to+// f32x4 as the AVX software fallback and the NEON fallback do.+pub struct CurrentCpuF16 {}++impl CpuF16 for CurrentCpuF16 {+ type Unit = v128;+ type Array = [v128; ARR];++ const STEP: usize = STEP;+ const EPR: usize = EPR;+ const ARR: usize = ARR;++ unsafe fn zero() -> Self::Unit {+ <CurrentCpu as Cpu>::zero()+ }++ unsafe fn zero_array() -> Self::Array {+ <CurrentCpu as Cpu>::zero_array()+ }++ unsafe fn from_f32(v: f32) -> Self::Unit {+ <CurrentCpu as Cpu>::from_f32(v)+ }++ unsafe fn load(mem_addr: *const f16) -> Self::Unit {+ let values = [+ (*mem_addr).to_f32(),+ (*mem_addr.add(1)).to_f32(),+ (*mem_addr.add(2)).to_f32(),+ (*mem_addr.add(3)).to_f32(),+ ];+ v128_load(values.as_ptr().cast())+ }++ unsafe fn vec_add(a: Self::Unit, b: Self::Unit) -> Self::Unit {+ <CurrentCpu as Cpu>::vec_add(a, b)+ }++ unsafe fn vec_fma(a: Self::Unit, b: Self::Unit, c: Self::Unit) -> Self::Unit {+ <CurrentCpu as Cpu>::vec_fma(a, b, c)+ }++ unsafe fn vec_store(mem_addr: *mut f16, a: Self::Unit) {+ let mut values = [0f32; EPR];+ v128_store(values.as_mut_ptr().cast(), a);+ for (i, value) in values.into_iter().enumerate() {+ *mem_addr.add(i) = f16::from_f32(value);+ }+ }++ unsafe fn vec_reduce(x: Self::Array, y: *mut f32) {+ <CurrentCpu as Cpu>::vec_reduce(x, y)+ }+}++// bf16 is the upper half of an f32. Widen its bits before doing arithmetic in+// f32x4, then use half's scalar conversion to preserve round-to-even on store.+pub struct CurrentCpuBF16 {}++impl CpuBF16 for CurrentCpuBF16 {+ type Unit = v128;+ type Array = [v128; ARR];++ const STEP: usize = STEP;+ const EPR: usize = EPR;+ const ARR: usize = ARR;++ unsafe fn zero() -> Self::Unit {+ <CurrentCpu as Cpu>::zero()+ }++ unsafe fn zero_array() -> Self::Array {+ <CurrentCpu as Cpu>::zero_array()+ }++ unsafe fn from_f32(v: f32) -> Self::Unit {+ <CurrentCpu as Cpu>::from_f32(v)+ }++ unsafe fn load(mem_addr: *const bf16) -> Self::Unit {+ let values = v128_load64_zero(mem_addr.cast());+ u32x4_shl(u32x4_extend_low_u16x8(values), 16)+ }++ unsafe fn vec_add(a: Self::Unit, b: Self::Unit) -> Self::Unit {+ <CurrentCpu as Cpu>::vec_add(a, b)+ }++ unsafe fn vec_fma(a: Self::Unit, b: Self::Unit, c: Self::Unit) -> Self::Unit {+ <CurrentCpu as Cpu>::vec_fma(a, b, c)+ }++ unsafe fn vec_store(mem_addr: *mut bf16, a: Self::Unit) {+ let mut values = [0f32; EPR];+ v128_store(values.as_mut_ptr().cast(), a);+ for (i, value) in values.into_iter().enumerate() {+ *mem_addr.add(i) = bf16::from_f32(value);+ }+ }++ unsafe fn vec_reduce(x: Self::Array, y: *mut f32) {+ <CurrentCpu as Cpu>::vec_reduce(x, y)+ }+}++pub(crate) unsafe fn vec_dot_f32(a_row: *const f32, b_row: *const f32, c: *mut f32, k: usize) {+ let mut sum = CurrentCpu::zero_array();+ let mut i = 0;+ while i + CurrentCpu::STEP <= k {+ for j in 0..CurrentCpu::ARR {+ sum[j] = CurrentCpu::vec_fma(+ sum[j],+ CurrentCpu::load(a_row.add(i + j * CurrentCpu::EPR)),+ CurrentCpu::load(b_row.add(i + j * CurrentCpu::EPR)),+ );+ }+ i += CurrentCpu::STEP;+ }+ CurrentCpu::vec_reduce(sum, c);+ while i < k {+ *c += *a_row.add(i) * *b_row.add(i);+ i += 1;+ }+}++pub(crate) unsafe fn vec_sum(row: *const f32, b: *mut f32, k: usize) {+ let np = k & !(CurrentCpu::STEP - 1);+ let mut sum = CurrentCpu::zero_array();+ let mut x = CurrentCpu::zero_array();+ for i in (0..np).step_by(CurrentCpu::STEP) {+ for j in 0..CurrentCpu::ARR {+ x[j] = CurrentCpu::load(row.add(i + j * CurrentCpu::EPR));+ sum[j] = CurrentCpu::vec_add(sum[j], x[j]);+ }+ }+ CurrentCpu::vec_reduce(sum, b);+ for i in np..k {+ *b += *row.add(i);+ }+}++pub(crate) unsafe fn vec_add_f16(a_row: *const f16, b_row: *const f16, c: *mut f16, k: usize) {+ let mut i = 0;+ while i + CurrentCpuF16::STEP <= k {+ for j in 0..CurrentCpuF16::ARR {+ CurrentCpuF16::vec_store(+ c.add(i + j * CurrentCpuF16::EPR),+ CurrentCpuF16::vec_add(+ CurrentCpuF16::load(a_row.add(i + j * CurrentCpuF16::EPR)),+ CurrentCpuF16::load(b_row.add(i + j * CurrentCpuF16::EPR)),+ ),+ );+ }+ i += CurrentCpuF16::STEP;+ }+ for j in i..k {+ *c.add(j) = *a_row.add(j) + *b_row.add(j);+ }+}++pub(crate) unsafe fn vec_add_bf16(a_row: *const bf16, b_row: *const bf16, c: *mut bf16, k: usize) {+ let mut i = 0;+ while i + CurrentCpuBF16::STEP <= k {+ for j in 0..CurrentCpuBF16::ARR {+ CurrentCpuBF16::vec_store(+ c.add(i + j * CurrentCpuBF16::EPR),+ CurrentCpuBF16::vec_add(+ CurrentCpuBF16::load(a_row.add(i + j * CurrentCpuBF16::EPR)),+ CurrentCpuBF16::load(b_row.add(i + j * CurrentCpuBF16::EPR)),+ ),+ );+ }+ i += CurrentCpuBF16::STEP;+ }+ for j in i..k {+ *c.add(j) = *a_row.add(j) + *b_row.add(j);+ }+}++pub(crate) unsafe fn vec_scalar_add_f16(scalar: f16, xs: *const f16, ys: *mut f16, k: usize) {+ let sv = CurrentCpuF16::from_f32(scalar.to_f32());+ let mut i = 0;+ while i + CurrentCpuF16::STEP <= k {+ for j in 0..CurrentCpuF16::ARR {+ CurrentCpuF16::vec_store(+ ys.add(i + j * CurrentCpuF16::EPR),+ CurrentCpuF16::vec_add(CurrentCpuF16::load(xs.add(i + j * CurrentCpuF16::EPR)), sv),+ );+ }+ i += CurrentCpuF16::STEP;+ }+ for j in i..k {+ *ys.add(j) = *xs.add(j) + scalar;+ }+}++pub(crate) unsafe fn vec_scalar_add_bf16(scalar: bf16, xs: *const bf16, ys: *mut bf16, k: usize) {+ let sv = CurrentCpuBF16::from_f32(scalar.to_f32());+ let mut i = 0;+ while i + CurrentCpuBF16::STEP <= k {+ for j in 0..CurrentCpuBF16::ARR {+ CurrentCpuBF16::vec_store(+ ys.add(i + j * CurrentCpuBF16::EPR),+ CurrentCpuBF16::vec_add(+ CurrentCpuBF16::load(xs.add(i + j * CurrentCpuBF16::EPR)),+ sv,+ ),+ );+ }+ i += CurrentCpuBF16::STEP;+ }+ for j in i..k {+ *ys.add(j) = *xs.add(j) + scalar;+ }+}diff --git a/vendor/candle-core/src/cpu_backend/conv2d.rs b/vendor/candle-core/src/cpu_backend/conv2d.rsnew file mode 100644index 0000000..b55e323--- /dev/null+++ b/vendor/candle-core/src/cpu_backend/conv2d.rs@@ -0,0 +1,419 @@+use std::borrow::Cow;++use rayon::iter::{IntoParallelIterator, ParallelIterator};++use crate::{+ conv::ParamsConv2D,+ cpu_backend::{copy_strided_src_, Im2Col, Map1, Map2, MatMul},+ shape::dims4,+ Layout, Result, WithDType,+};++pub(super) struct Conv2D<'a>(pub(super) &'a crate::conv::ParamsConv2D);++#[allow(dead_code)]+enum Conv2dImpl {+ TiledIm2Col,+ FullIm2Col,+ Direct,+}++const DEFAULT_CONV2D_IMPL: Conv2dImpl = Conv2dImpl::TiledIm2Col;++impl Map2 for Conv2D<'_> {+ const OP: &'static str = "conv2d";+ fn f<T: WithDType + num_traits::Num + Copy + 'static>(+ &self,+ inp: &[T],+ inp_l: &Layout,+ k: &[T],+ k_l: &Layout,+ ) -> Result<Vec<T>> {+ let p = self.0;++ // Specialization: pick the best algorithm based on parameters.+ // 1x1 convolutions with stride=1, padding=0, dilation=1+ if p.k_h == 1 && p.k_w == 1 && p.stride == 1 && p.padding == 0 && p.dilation == 1 {+ return conv2d_1x1(p, inp, inp_l, k, k_l);+ } else if p.k_h == 1 && p.k_w == 1 {+ // Other 1x1 convolutions for now are assumed faster with full im2col,+ // although with large enough input size, tiled will start beating it.+ return conv2d_im2col_gemm(p, inp, inp_l, k, k_l);+ }+ // TODO other cases++ // No fast path, fallback to default general impl.+ match DEFAULT_CONV2D_IMPL {+ Conv2dImpl::TiledIm2Col => conv2d_tiled(p, inp, inp_l, k, k_l),+ Conv2dImpl::Direct => conv2d_direct(p, inp, inp_l, k, k_l),+ Conv2dImpl::FullIm2Col => conv2d_im2col_gemm(p, inp, inp_l, k, k_l),+ }+ }+}++/// Fast kernel for 1x1 convolutions with stride=1, padding=0, dilation=1+/// These are just matrix multiplications: [c_out, c_in] @ [c_in, b*h*w] -> [c_out, b*h*w].+fn conv2d_1x1<T: WithDType + num_traits::Num + Copy + 'static>(+ p: &ParamsConv2D,+ inp: &[T],+ inp_l: &Layout,+ k: &[T],+ k_l: &Layout,+) -> Result<Vec<T>> {+ let inp = &inp[inp_l.start_offset()..];+ let inp_stride = inp_l.stride();+ let (inp_s0, inp_s1, inp_s2, inp_s3) =+ (inp_stride[0], inp_stride[1], inp_stride[2], inp_stride[3]);+ let k = &k[k_l.start_offset()..];+ let k_stride = k_l.stride();+ let (k_s0, k_s1) = (k_stride[0], k_stride[1]);+ let (out_h, out_w) = (p.out_h(), p.out_w());++ let spatial_size = out_h * out_w;+ let dst = vec![T::zero(); p.b_size * p.c_out * spatial_size];+ let k_reshaped: Cow<[T]> = if k_s0 == p.c_in && k_s1 == 1 {+ // Already contiguous, use slice directly+ Cow::Borrowed(&k[..p.c_out * p.c_in])+ } else {+ // Reshape kernel to [c_out, c_in]+ let mut k_reshaped = Vec::with_capacity(p.c_out * p.c_in);+ (0..p.c_out).for_each(|c_out_idx| {+ (0..p.c_in).for_each(|c_in_idx| {+ let k_idx = c_out_idx * k_s0 + c_in_idx * k_s1;+ k_reshaped.push(k[k_idx]);+ });+ });+ Cow::Owned(k_reshaped)+ };+ let k_layout = Layout::contiguous((p.c_out, p.c_in));++ // Process each batch+ (0..p.b_size).into_par_iter().try_for_each(|b_idx| {+ // Reshape input to [c_in, h*w] for this batch+ let mut inp_reshaped = Vec::with_capacity(p.c_in * spatial_size);+ for c_in_idx in 0..p.c_in {+ for h_idx in 0..p.i_h {+ for w_idx in 0..p.i_w {+ let inp_idx =+ b_idx * inp_s0 + c_in_idx * inp_s1 + h_idx * inp_s2 + w_idx * inp_s3;+ inp_reshaped.push(inp[inp_idx]);+ }+ }+ }+ let inp_layout = Layout::contiguous((p.c_in, spatial_size));++ // Perform matmul: [c_out, c_in] @ [c_in, spatial_size] -> [c_out, spatial_size]+ let matmul = MatMul((1, p.c_out, spatial_size, p.c_in));+ let result = matmul.f(&k_reshaped, &k_layout, &inp_reshaped, &inp_layout)?;++ // Copy result to output+ let out_offset = b_idx * p.c_out * spatial_size;+ for (i, r) in result.iter().enumerate() {+ unsafe {+ let ptr = dst.as_ptr().add(out_offset + i) as *mut T;+ *ptr = *r;+ }+ }+ Ok::<(), crate::Error>(())+ })?;++ Ok(dst)+}++/// General tiled convolution implementation using gemm.+///+/// Similar to full im2col, but instead of materializing the full matrix, we process input/output in tiles, in parallel.+fn conv2d_tiled<T: WithDType + num_traits::Num + Copy + 'static>(+ p: &ParamsConv2D,+ inp: &[T],+ inp_l: &Layout,+ k: &[T],+ k_l: &Layout,+) -> Result<Vec<T>> {+ let inp = &inp[inp_l.start_offset()..];+ let (inp_s0, inp_s1, inp_s2, inp_s3) = dims4(inp_l.stride())?;+ let k = &k[k_l.start_offset()..];+ let (k_s0, k_s1, k_s2, k_s3) = dims4(k_l.stride())?;+ let (out_h, out_w) = (p.out_h(), p.out_w());++ // Output shape: [b_size, c_out, out_h, out_w].+ let dst = vec![T::zero(); p.b_size * p.c_out * out_h * out_w];++ // Convert NCHW input to NHWC layout for tiled im2col.+ let cont_s0 = p.i_h * p.i_w * p.c_in;+ let cont_s1 = p.i_w * p.c_in;+ let cont_s2 = p.c_in;+ let mut inp_cont = vec![T::zero(); p.b_size * p.c_in * p.i_h * p.i_w];+ for b_idx in 0..p.b_size {+ for h_idx in 0..p.i_h {+ for w_idx in 0..p.i_w {+ for c_idx in 0..p.c_in {+ let src_idx = b_idx * inp_s0 + c_idx * inp_s1 + h_idx * inp_s2 + w_idx * inp_s3;+ let dst_idx = b_idx * cont_s0 + h_idx * cont_s1 + w_idx * cont_s2 + c_idx;+ inp_cont[dst_idx] = inp[src_idx]+ }+ }+ }+ }++ // shape of k: [c_out, c_in, k_h, k_w]+ // strides of k: [k_s0, k_s1, k_s2, k_s3]+ // For matmul, we need flattened k in shape [c_out, k_h * k_w * c_in]+ // with stride [k_h * k_w * c_in, 1]+ let k_size = p.c_in * p.k_h * p.k_w;+ let mut k_flat = Vec::with_capacity(p.c_out * k_size);+ for dst_c_idx in 0..p.c_out {+ for kh in 0..p.k_h {+ for kw in 0..p.k_w {+ for c_in_idx in 0..p.c_in {+ let k_idx = dst_c_idx * k_s0 + c_in_idx * k_s1 + kh * k_s2 + kw * k_s3;+ k_flat.push(k[k_idx]);+ }+ }+ }+ }+ // k_layout: [c_out, k_size] with stride [k_size, 1]+ let k_layout = Layout::contiguous((p.c_out, k_size));++ // TILE_SIZE is number of output pixels (out_h * out_w) per tile.+ // Higher tile size can be faster due to better usage of gemm,+ // but lower tile sizes enable bigger parallelism across tiles.+ // This parameter is impactful and may be dynamic or even runtime tunable in the future.+ const TILE_SIZE: usize = 512;++ let total_out_pixels = out_h * out_w;++ // Process batches and tiles in parallel using rayon.+ (0..p.b_size).into_par_iter().try_for_each(|b_idx| {+ let inp_offset = b_idx * cont_s0;+ let out_batch_offset = b_idx * (p.c_out * out_h * out_w);++ let num_tiles = total_out_pixels.div_ceil(TILE_SIZE);+ (0..num_tiles).into_par_iter().try_for_each(|tile_idx| {+ // Determine actual tile size (may be smaller at the end) {+ let tile_start = tile_idx * TILE_SIZE;+ let tile_end = (tile_start + TILE_SIZE).min(total_out_pixels);+ let tile_size = tile_end - tile_start;++ // Precompute output coordinates.+ // Used in both im2col extraction and writing output.+ let out_coords: Vec<_> = (tile_start..tile_end)+ .map(|idx| (idx / out_w, idx % out_w))+ .collect();++ // Build im2col tile: [k_size, tile_size]+ // This represents the input patches needed for this tile of outputs+ let mut col_tile = vec![T::zero(); k_size * tile_size];++ for (tile_idx, (out_y, out_x)) in out_coords.iter().enumerate() {+ // Extract the im2col patch for this output position+ for c_in in 0..p.c_in {+ let mut patch_offset = c_in;+ for kh in 0..p.k_h {+ let in_y =+ (out_y * p.stride + kh * p.dilation) as isize - p.padding as isize;+ if in_y < 0 || in_y >= p.i_h as isize {+ // Padding: already zero+ patch_offset += p.c_in * p.k_w;+ continue;+ }+ for kw in 0..p.k_w {+ let in_x =+ (out_x * p.stride + kw * p.dilation) as isize - p.padding as isize;++ if in_x >= 0 && in_x < p.i_w as isize {+ let in_y = in_y as usize;+ let in_x = in_x as usize;+ let inp_idx = inp_offset + in_y * cont_s1 + in_x * cont_s2 + c_in;+ let col_idx = patch_offset * tile_size + tile_idx;+ col_tile[col_idx] = inp_cont[inp_idx];+ }+ // Move to next position (skip c_in channels)+ patch_offset += p.c_in;+ }+ }+ }+ }++ // Now perform matmul: k_cache [c_out, k_size] @ col_tile [k_size, tile_size]+ let matmul = MatMul((1, p.c_out, tile_size, k_size));++ // Layouts for matmul+ // k_flat layout: [c_out, k_size] with stride [k_size, 1]+ // col_tile layout: [k_size, tile_size] with stride [tile_size, 1]+ let col_layout = Layout::contiguous((k_size, tile_size));++ // Perform matmul+ let result = matmul.f(&k_flat, &k_layout, &col_tile, &col_layout)?;++ // Copy results to output: result is [c_out, tile_size]+ for (tile_idx, (out_y, out_x)) in out_coords.iter().enumerate() {+ let dst_base = out_batch_offset + out_y * out_w + out_x;++ for c_out_idx in 0..p.c_out {+ let dst_idx = dst_base + c_out_idx * (out_h * out_w);+ let result_idx = c_out_idx * tile_size + tile_idx;+ // SAFETY: Each batch processes a distinct region of the output buffer.+ // Within each batch, tiles process non-overlapping output positions.+ // Therefore, no two threads will write to the same dst_idx.+ unsafe {+ let ptr = dst.as_ptr().add(dst_idx) as *mut T;+ *ptr = result[result_idx];+ }+ }+ }+ Ok::<(), crate::Error>(())+ })+ })?;++ Ok(dst)+}++/// General direct convolution impl. Decently fast for small inputs and kernels, but loses to full/tiled gemm.+fn conv2d_direct<T: WithDType + num_traits::Num + Copy + 'static>(+ p: &ParamsConv2D,+ inp: &[T],+ inp_l: &Layout,+ k: &[T],+ k_l: &Layout,+) -> Result<Vec<T>> {+ let inp = &inp[inp_l.start_offset()..];+ let (inp_s0, inp_s1, inp_s2, inp_s3) = crate::shape::dims4(inp_l.stride())?;+ let k = &k[k_l.start_offset()..];+ let (k_s0, k_s1, k_s2, k_s3) = crate::shape::dims4(k_l.stride())?;+ let (out_h, out_w) = (p.out_h(), p.out_w());++ // Output shape: [b_size, c_out, out_h, out_w].+ let dst = vec![T::zero(); p.b_size * p.c_out * out_h * out_w];++ // Convert NCHW input to NHWC layout for direct convolution.+ let cont_s0 = p.i_h * p.i_w * p.c_in;+ let cont_s1 = p.i_w * p.c_in;+ let cont_s2 = p.c_in;+ let mut inp_cont = vec![T::zero(); p.b_size * p.c_in * p.i_h * p.i_w];+ for b_idx in 0..p.b_size {+ for h_idx in 0..p.i_h {+ for w_idx in 0..p.i_w {+ for c_idx in 0..p.c_in {+ let src_idx = b_idx * inp_s0 + c_idx * inp_s1 + h_idx * inp_s2 + w_idx * inp_s3;+ let dst_idx = b_idx * cont_s0 + h_idx * cont_s1 + w_idx * cont_s2 + c_idx;+ inp_cont[dst_idx] = inp[src_idx]+ }+ }+ }+ }+ let inp_cont_len = inp_cont.len();++ let k_cache: Vec<Vec<T>> = (0..p.c_out)+ .map(|dst_c_idx| {+ (0..p.k_h * p.k_w)+ .flat_map(|kw_kh| {+ let offset_h = kw_kh / p.k_w;+ let offset_w = kw_kh % p.k_w;+ (0..p.c_in).map(move |c_in_idx| {+ k[dst_c_idx * k_s0 + c_in_idx * k_s1 + offset_h * k_s2 + offset_w * k_s3]+ })+ })+ .collect()+ })+ .collect();++ for b_idx in 0..p.b_size {+ for offset_h in 0..p.k_h {+ for offset_w in 0..p.k_w {+ let k_offset = offset_h * p.k_w + offset_w;++ (0..p.c_out).into_par_iter().for_each(|dst_c_idx| {+ let k_cont = &k_cache[dst_c_idx][k_offset * p.c_in..(k_offset + 1) * p.c_in];+ let base_dst_idx = dst_c_idx * out_w * out_h;+ let batch_dst_idx = base_dst_idx + b_idx * p.c_out * out_h * out_w;+ let batch_src_idx = b_idx * cont_s0;++ for dst_h in 0..out_h {+ let src_h = p.stride * dst_h + offset_h * p.dilation;+ if src_h < p.padding || src_h >= p.i_h + p.padding {+ continue;+ }+ let src_h = src_h - p.padding;+ let h_dst_idx = batch_dst_idx + dst_h * out_w;+ let h_src_idx = batch_src_idx + src_h * cont_s1;++ for dst_w in 0..out_w {+ let src_w = p.stride * dst_w + offset_w * p.dilation;+ if src_w < p.padding || src_w >= p.i_w + p.padding {+ continue;+ }+ let src_w = src_w - p.padding;+ let dst_idx = h_dst_idx + dst_w;+ let inp_idx_1 = h_src_idx + src_w * cont_s2;+ let inp_idx_2 = (inp_idx_1 + p.c_in).min(inp_cont_len);+ let inp_cont = &inp_cont[inp_idx_1..inp_idx_2];+ let mut d = T::zero();+ unsafe {+ T::vec_dot(inp_cont.as_ptr(), k_cont.as_ptr(), &mut d, p.c_in);+ let ptr = dst.as_ptr().add(dst_idx) as *mut T;+ *ptr += d;+ }+ }+ }+ });+ }+ }+ }++ Ok(dst)+}++#[allow(clippy::uninit_vec)]+fn alloc_uninit_vec<T: WithDType + Copy + 'static>(size: usize) -> Vec<T> {+ let mut v = Vec::with_capacity(size);+ unsafe { v.set_len(size) };+ v+}++/// Full im2col + gemm convolution implementation.+///+/// For large inputs im2col and copy_strided_src for output gets expensive.+fn conv2d_im2col_gemm<T: WithDType + num_traits::Num + Copy + 'static>(+ p: &ParamsConv2D,+ inp: &[T],+ inp_l: &Layout,+ kernel: &[T],+ kernel_l: &Layout,+) -> Result<Vec<T>> {+ let op = Im2Col {+ h_k: p.k_h,+ w_k: p.k_w,+ padding: p.padding,+ stride: p.stride,+ dilation: p.dilation,+ };+ let col = op.f(inp, inp_l)?;+ let b = p.b_size;+ let n = p.c_out;+ let (h_out, w_out) = (p.out_h(), p.out_w());+ let k = op.h_k * op.w_k * p.c_in;+ let m = h_out * w_out;+ let col_l = Layout::contiguous((b, m, k));+ let res: Vec<T> = if kernel_l.is_contiguous() {+ let kernel_l = Layout::contiguous_with_offset((1, n, k), kernel_l.start_offset())+ .transpose(1, 2)?+ .broadcast_as((b, k, n))?;+ MatMul((b, m, n, k)).f(&col, &col_l, kernel, &kernel_l)?+ } else {+ // copy_strided_src_ materializes the kernel into kernel_c starting at index 0, so the+ // matmul layout must use offset 0, not the original (strided) kernel's start offset.+ let mut kernel_c = alloc_uninit_vec(kernel_l.shape().elem_count());+ copy_strided_src_(kernel, &mut kernel_c, 0, kernel_l);+ let kernel_l = Layout::contiguous_with_offset((1, n, k), 0)+ .transpose(1, 2)?+ .broadcast_as((b, k, n))?;+ MatMul((b, m, n, k)).f(&col, &col_l, &kernel_c, &kernel_l)?+ };+ let res_l = Layout::contiguous((b, h_out, w_out, p.c_out))+ .transpose(1, 2)?+ .transpose(1, 3)?;+ let mut res_t = alloc_uninit_vec(res_l.shape().elem_count());+ copy_strided_src_(&res, &mut res_t, 0, &res_l);+ Ok(res_t)+}diff --git a/vendor/candle-core/src/cpu_backend/mod.rs b/vendor/candle-core/src/cpu_backend/mod.rsnew file mode 100644index 0000000..1e2bf64--- /dev/null+++ b/vendor/candle-core/src/cpu_backend/mod.rs@@ -0,0 +1,3415 @@+//! Implementation of Backend Fns for CPU+use crate::backend::{BackendDevice, BackendStorage};+use crate::op::{BinaryOpT, CmpOp, ReduceOp, UnaryOpT};+use crate::{DType, Error, IntDType, Layout, NdIter, Result, Shape, WithDType};+use float8::F8E4M3;+use half::{bf16, f16, slice::HalfFloatSliceExt, vec::HalfFloatVecExt};+use rayon::prelude::*;++mod utils;+pub use utils::{+ binary_map, binary_map_vec, binary_map_vec_par, unary_map, unary_map_vec, unary_map_vec_par,+ Map1, Map1Any, Map2, Map2InPlace, Map2U8,+};+mod conv2d;+use conv2d::Conv2D;++const USE_IM2COL_CONV1D: bool = true;+const USE_COL2IM_CONV1D_TR: bool = true;++// TODO: Maybe we should not implement [Clone] here and instead have an explicit allocator ++// intercept the oom errors to avoid panicking and provide a proper error.+#[derive(Debug, Clone)]+pub enum CpuStorage {+ U8(Vec<u8>),+ U32(Vec<u32>),+ I16(Vec<i16>),+ I32(Vec<i32>),+ I64(Vec<i64>),+ BF16(Vec<bf16>),+ F16(Vec<f16>),+ F32(Vec<f32>),+ F64(Vec<f64>),+ F8E4M3(Vec<F8E4M3>),+ // Dummy types that store raw bytes+ F6E2M3(Vec<u8>),+ F6E3M2(Vec<u8>),+ F4(Vec<u8>),+ F8E8M0(Vec<u8>),+}++#[derive(Debug, Clone)]+pub enum CpuStorageRef<'a> {+ U8(&'a [u8]),+ U32(&'a [u32]),+ I16(&'a [i16]),+ I32(&'a [i32]),+ I64(&'a [i64]),+ BF16(&'a [bf16]),+ F16(&'a [f16]),+ F32(&'a [f32]),+ F64(&'a [f64]),+ F8E4M3(&'a [F8E4M3]),+ // Dummy types that store raw bytes+ F6E2M3(&'a [u8]),+ F6E3M2(&'a [u8]),+ F4(&'a [u8]),+ F8E8M0(&'a [u8]),+}++#[derive(Debug, Clone)]+pub struct CpuDevice;++struct Cmp(CmpOp);+impl Map2U8 for Cmp {+ const OP: &'static str = "cmp";+ #[inline(always)]+ fn f<T: WithDType>(+ &self,+ lhs: &[T],+ lhs_l: &Layout,+ rhs: &[T],+ rhs_l: &Layout,+ ) -> Result<Vec<u8>> {+ let dst = match self.0 {+ CmpOp::Eq => binary_map(lhs_l, rhs_l, lhs, rhs, |x, y| u8::from(x == y)),+ CmpOp::Ne => binary_map(lhs_l, rhs_l, lhs, rhs, |x, y| u8::from(x != y)),+ CmpOp::Lt => binary_map(lhs_l, rhs_l, lhs, rhs, |x, y| u8::from(x < y)),+ CmpOp::Le => binary_map(lhs_l, rhs_l, lhs, rhs, |x, y| u8::from(x <= y)),+ CmpOp::Gt => binary_map(lhs_l, rhs_l, lhs, rhs, |x, y| u8::from(x > y)),+ CmpOp::Ge => binary_map(lhs_l, rhs_l, lhs, rhs, |x, y| u8::from(x >= y)),+ };+ Ok(dst)+ }+}++struct WCond<'a, T: IntDType>(&'a [T], &'a Layout);++#[allow(clippy::too_many_arguments)]+#[inline(always)]+fn where_impl<I, T, P, R, G>(+ nd_iter: NdIter<3>,+ ys_to_set: &mut [T],+ pred: &[I],+ t: &[T],+ f: &[T],+ pred_fn: P,+ t_fn: R,+ f_fn: G,+) where+ I: IntDType,+ T: Copy,+ P: Fn(usize, usize, usize) -> usize,+ R: Fn(usize, usize, usize) -> usize,+ G: Fn(usize, usize, usize) -> usize,+{+ let inner_size = nd_iter.inner_size;+ let [inner_ps, inner_ts, inner_fs] = nd_iter.inner_strides;+ let mut dst_off = 0usize;+ for [p_off, t_off, f_off] in nd_iter {+ for i in 0..inner_size {+ let ep = pred[pred_fn(p_off, i, inner_ps)];+ let et = t[t_fn(t_off, i, inner_ts)];+ let ef = f[f_fn(f_off, i, inner_fs)];++ ys_to_set[dst_off + i] = if ep.is_true() { et } else { ef };+ }+ dst_off += inner_size;+ }+}++impl<I: IntDType> Map2 for WCond<'_, I> {+ const OP: &'static str = "where";+ #[inline(always)]+ fn f<T: WithDType>(&self, t: &[T], t_l: &Layout, f: &[T], f_l: &Layout) -> Result<Vec<T>> {+ let vs = match (+ self.1.contiguous_offsets(),+ t_l.contiguous_offsets(),+ f_l.contiguous_offsets(),+ ) {+ (Some((o1, o2)), Some((o_t1, o_t2)), Some((o_f1, o_f2))) => {+ let pred = &self.0[o1..o2];+ let t = &t[o_t1..o_t2];+ let f = &f[o_f1..o_f2];+ pred.iter()+ .zip(t.iter().zip(f.iter()))+ .map(|(p, (&t, &f))| if p.is_true() { t } else { f })+ .collect::<Vec<_>>()+ }+ _ => {+ // Same allocation strategy as `binary_map_vec`+ let el_count = self.1.shape().elem_count();++ let mut ys: Vec<T> = Vec::with_capacity(el_count);+ let ys_to_set = unsafe {+ let s = ys.spare_capacity_mut();+ std::mem::transmute::<&mut [std::mem::MaybeUninit<T>], &mut [T]>(s)+ };++ let nd_iter = NdIter::new([self.1, t_l, f_l]);+ let [inner_ps, inner_ts, inner_fs] = nd_iter.inner_strides;++ let offset = |off, i, _| off + i;+ let boffset = |off, _, _| off;++ match (inner_ps, inner_ts, inner_fs) {+ (1, 1, 1) => {+ where_impl(nd_iter, ys_to_set, self.0, t, f, offset, offset, offset);+ }+ (1, 1, 0) => {+ where_impl(nd_iter, ys_to_set, self.0, t, f, offset, offset, boffset);+ }+ (1, 0, 1) => {+ where_impl(nd_iter, ys_to_set, self.0, t, f, offset, boffset, offset);+ }+ (1, 0, 0) => {+ where_impl(nd_iter, ys_to_set, self.0, t, f, offset, boffset, boffset);+ }+ (0, 1, 1) => {+ where_impl(nd_iter, ys_to_set, self.0, t, f, boffset, offset, offset);+ }+ (0, 1, 0) => {+ where_impl(nd_iter, ys_to_set, self.0, t, f, boffset, offset, boffset);+ }+ (0, 0, 1) => {+ where_impl(nd_iter, ys_to_set, self.0, t, f, boffset, boffset, offset);+ }+ (0, 0, 0) => {+ where_impl(nd_iter, ys_to_set, self.0, t, f, boffset, boffset, boffset);+ }+ _ => {+ let offset = |off, i, inner| off + i * inner;+ where_impl(nd_iter, ys_to_set, self.0, t, f, offset, offset, offset);+ }+ }++ // SAFETY: all el_count elements have been written in the dispatch loop above.+ unsafe { ys.set_len(el_count) };+ ys+ }+ };+ Ok(vs)+ }+}++struct ReduceIndex {+ reduce_dim_index: usize,+ use_min: bool,+ return_index: bool,+}++impl ReduceIndex {+ // The value gets replaced if f(s[current_acc], s[i]) returns true.+ #[inline(always)]+ fn fold_impl<T, U, F, G>(&self, src: &[T], src_l: &Layout, f: F, g: G) -> Result<Vec<U>>+ where+ T: Clone + Copy,+ U: Clone + Copy,+ F: Fn(T, T) -> bool,+ G: Fn(T, usize) -> U,+ {+ let reduce_dim_size = src_l.dims()[self.reduce_dim_index];+ let reduce_dim_stride = src_l.stride()[self.reduce_dim_index];+ let dst_len = src_l.shape().elem_count() / reduce_dim_size;+ let mut dst: Vec<U> = Vec::with_capacity(dst_len);+ let dst_to_set = dst.spare_capacity_mut();+ let dst_to_set =+ unsafe { std::mem::transmute::<&mut [std::mem::MaybeUninit<U>], &mut [U]>(dst_to_set) };+ match src_l.contiguous_offsets() {+ Some((o1, o2)) => {+ let src = &src[o1..o2];+ if reduce_dim_stride == 1 {+ for (start_src_i, dst_v) in dst_to_set.iter_mut().enumerate() {+ let start_src_i = start_src_i * reduce_dim_size;+ let src = &src[start_src_i..start_src_i + reduce_dim_size];+ let mut acc = 0;+ let mut val = src[0];+ for (src_i, &s) in src.iter().enumerate() {+ if f(val, s) {+ acc = src_i;+ val = s+ }+ }+ *dst_v = g(val, acc)+ }+ } else {+ for (start_src_i, dst_v) in dst_to_set.iter_mut().enumerate() {+ let (p, q) = (+ start_src_i / reduce_dim_stride,+ start_src_i % reduce_dim_stride,+ );+ // start_src_i = p * reduce_dim_stride + q+ let start_src_i = p * reduce_dim_stride * reduce_dim_size + q;+ let src = &src[start_src_i..];+ let mut acc = 0;+ let mut val = src[0];+ for src_i in 0..reduce_dim_size {+ let s = src[src_i * reduce_dim_stride];+ if f(val, s) {+ acc = src_i;+ val = s+ }+ }+ *dst_v = g(val, acc)+ }+ }+ }+ None => {+ let l = src_l.narrow(self.reduce_dim_index, 0, 1)?;+ for (unstr_index, src_index) in l.strided_index().enumerate() {+ let src = &src[src_index..];+ let mut acc = 0;+ let mut val = src[0];+ for src_i in 0..reduce_dim_size {+ let s = src[src_i * reduce_dim_stride];+ if f(val, s) {+ acc = src_i;+ val = s+ }+ }+ dst_to_set[unstr_index] = g(val, acc)+ }+ }+ }+ unsafe { dst.set_len(dst_len) };+ Ok(dst)+ }+}++impl Map1Any for ReduceIndex {+ #[inline(always)]+ fn f<T: WithDType, W: Fn(Vec<T>) -> CpuStorage>(+ &self,+ src: &[T],+ src_l: &Layout,+ wrap: W,+ ) -> Result<CpuStorage> {+ if src_l.shape().elem_count() == 0 {+ Err(Error::EmptyTensor { op: "reduce" }.bt())?+ }+ let dst = match (self.return_index, self.use_min) {+ (false, true) => wrap(self.fold_impl(src, src_l, |x, y| x > y, |v, _i| v)?),+ (false, false) => wrap(self.fold_impl(src, src_l, |x, y| x < y, |v, _i| v)?),+ (true, true) => {+ CpuStorage::U32(self.fold_impl(src, src_l, |x, y| x > y, |_v, i| i as u32)?)+ }+ (true, false) => {+ CpuStorage::U32(self.fold_impl(src, src_l, |x, y| x < y, |_v, i| i as u32)?)+ }+ };+ Ok(dst)+ }+}++struct ReduceSum<'a> {+ dst_shape: &'a Shape,+ reduce_dims: &'a [usize],+ reduce_dims_and_stride: Vec<(usize, usize)>,+}++impl ReduceSum<'_> {+ #[inline(always)]+ fn fold_impl<T>(&self, src: &[T], src_l: &Layout, start_elt: T) -> Result<Vec<T>>+ where+ T: WithDType,+ {+ let mut dst = vec![start_elt; self.dst_shape.elem_count()];+ match src_l.contiguous_offsets() {+ Some((o1, o2)) => {+ let src = &src[o1..o2];+ // Handle the case where we reduce over the last dimensions separately as it is+ // fairly common and easy to optimize. This rely on the layout being contiguous!+ // reduce_dims is sorted, check if it is ranging from a to n-1.+ let reduce_over_last_dims = self+ .reduce_dims+ .iter()+ .rev()+ .enumerate()+ .all(|(i, &v)| v == src_l.shape().rank() - 1 - i);+ if reduce_over_last_dims {+ let reduce_sz = self+ .reduce_dims_and_stride+ .iter()+ .map(|(u, _)| u)+ .product::<usize>();+ for (dst_i, dst_v) in dst.iter_mut().enumerate() {+ let src_i = dst_i * reduce_sz;+ unsafe {+ T::vec_reduce_sum(+ src[src_i..src_i + reduce_sz].as_ptr(),+ dst_v,+ reduce_sz,+ )+ };+ }+ return Ok(dst);+ };+ for (unstr_index, &src) in src.iter().enumerate() {+ let mut dst_index = unstr_index;+ // Set the reduce_dims indexes to 0.+ for &(dim, stride) in self.reduce_dims_and_stride.iter() {+ // The compiler is able to optimize the following in a single divmod op.+ let (pre, post) = (dst_index / stride, dst_index % stride);+ dst_index = (pre / dim) * stride + post;+ }+ dst[dst_index] += src;+ }+ }+ None => {+ for (unstr_index, src_index) in src_l.strided_index().enumerate() {+ let mut dst_index = unstr_index;+ // Set the reduce_dims indexes to 0.+ for &(dim, stride) in self.reduce_dims_and_stride.iter() {+ // The compiler is able to optimize the following in a single divmod op.+ let (pre, post) = (dst_index / stride, dst_index % stride);+ dst_index = (pre / dim) * stride + post;+ }+ dst[dst_index] += src[src_index];+ }+ }+ }+ Ok(dst)+ }+}++impl Map1 for ReduceSum<'_> {+ #[inline(always)]+ fn f<T: WithDType>(&self, src: &[T], src_l: &Layout) -> Result<Vec<T>> {+ self.fold_impl(src, src_l, T::zero())+ }+}++struct Affine(f64, f64);++impl Map1 for Affine {+ fn f<T: WithDType>(&self, vs: &[T], layout: &Layout) -> Result<Vec<T>> {+ let mul = T::from_f64(self.0);+ let add = T::from_f64(self.1);+ Ok(unary_map(vs, layout, |v| v * mul + add))+ }+}++struct AvgPool2D((usize, usize), (usize, usize));++impl Map1 for AvgPool2D {+ fn f<T: WithDType>(&self, src: &[T], layout: &Layout) -> Result<Vec<T>> {+ // https://pytorch.org/docs/stable/generated/torch.nn.AvgPool2d.html+ let (k_h, k_w) = self.0;+ let (s_h, s_w) = self.1;+ let (b_sz, c, h, w) = layout.shape().dims4()?;+ let stride = layout.stride();+ let (stride_h, stride_w) = (stride[2], stride[3]);+ let h_out = (h - k_h) / s_h + 1;+ let w_out = (w - k_w) / s_w + 1;+ let src_index = layout.start_offset();+ let mut dst = vec![T::zero(); b_sz * c * h_out * w_out];+ let scale = 1f64 / (k_h * k_w) as f64;+ let scale = T::from_f64(scale);+ for b_idx in 0..b_sz {+ let dst = &mut dst[b_idx * c * h_out * w_out..];+ let src_index = src_index + b_idx * stride[0];+ for c_idx in 0..c {+ let dst = &mut dst[c_idx * h_out * w_out..];+ let src_index = src_index + c_idx * stride[1];+ for h_idx in 0..h_out {+ for w_idx in 0..w_out {+ let mut sum = T::zero();+ for m in 0..k_h {+ for n in 0..k_w {+ let m = s_h * h_idx + m;+ let n = s_w * w_idx + n;+ sum += src[src_index + m * stride_h + n * stride_w]+ }+ }+ dst[h_idx * w_out + w_idx] = sum * scale;+ }+ }+ }+ }+ Ok(dst)+ }+}++struct MaxPool2D((usize, usize), (usize, usize));++impl Map1 for MaxPool2D {+ fn f<T: WithDType>(&self, src: &[T], layout: &Layout) -> Result<Vec<T>> {+ // https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html+ let (k_h, k_w) = self.0;+ let (s_h, s_w) = self.1;+ let (b_sz, c, h, w) = layout.shape().dims4()?;+ let stride = layout.stride();+ let (stride_h, stride_w) = (stride[2], stride[3]);+ let h_out = (h - k_h) / s_h + 1;+ let w_out = (w - k_w) / s_w + 1;+ let src_index = layout.start_offset();+ let mut dst = vec![T::zero(); b_sz * c * h_out * w_out];+ for b_idx in 0..b_sz {+ let dst = &mut dst[b_idx * c * h_out * w_out..];+ let src_index = src_index + b_idx * stride[0];+ for c_idx in 0..c {+ let dst = &mut dst[c_idx * h_out * w_out..];+ let src_index = src_index + c_idx * stride[1];+ for h_idx in 0..h_out {+ for w_idx in 0..w_out {+ let mut largest =+ src[src_index + s_h * h_idx * stride_h + s_w * w_idx * stride_w];+ for m in 0..k_h {+ for n in 0..k_w {+ let m = s_h * h_idx + m;+ let n = s_w * w_idx + n;+ if largest < src[src_index + m * stride_h + n * stride_w] {+ largest = src[src_index + m * stride_h + n * stride_w]+ }+ }+ }+ dst[h_idx * w_out + w_idx] = largest;+ }+ }+ }+ }+ Ok(dst)+ }+}++struct UpsampleNearest1D(usize);++impl Map1 for UpsampleNearest1D {+ fn f<T: WithDType>(&self, src: &[T], layout: &Layout) -> Result<Vec<T>> {+ // TODO: Specialized implementation for the case 2*sz?+ let dst_sz = self.0;+ let (b_sz, c, src_sz) = layout.shape().dims3()?;+ let stride = layout.stride();+ let stride_sz = stride[2];+ let src_index = layout.start_offset();+ let scale_sz = src_sz as f64 / dst_sz as f64;+ let mut dst = vec![T::zero(); b_sz * c * dst_sz];+ let src_idxs = (0..dst_sz)+ .map(|idx| usize::min(src_sz - 1, (idx as f64 * scale_sz) as usize))+ .collect::<Vec<_>>();+ for b_idx in 0..b_sz {+ let dst = &mut dst[b_idx * c * dst_sz..];+ let src_index = src_index + b_idx * stride[0];+ for c_idx in 0..c {+ let dst = &mut dst[c_idx * dst_sz..];+ let src_index = src_index + c_idx * stride[1];+ for (idx, src_idx) in src_idxs.iter().enumerate() {+ dst[idx] = src[src_index + src_idx * stride_sz]+ }+ }+ }+ Ok(dst)+ }+}++struct UpsampleNearest2D(usize, usize);++impl Map1 for UpsampleNearest2D {+ fn f<T: WithDType>(&self, src: &[T], layout: &Layout) -> Result<Vec<T>> {+ // TODO: Specialized implementation for the case 2*h, 2*w?+ let (dst_h, dst_w) = (self.0, self.1);+ let (b_sz, c, src_h, src_w) = layout.shape().dims4()?;+ let stride = layout.stride();+ let (stride_h, stride_w) = (stride[2], stride[3]);+ let src_index = layout.start_offset();+ let scale_h = src_h as f64 / dst_h as f64;+ let scale_w = src_w as f64 / dst_w as f64;+ let mut dst = vec![T::zero(); b_sz * c * dst_h * dst_w];+ let src_h_idxs = (0..dst_h)+ .map(|h_idx| usize::min(src_h - 1, (h_idx as f64 * scale_h) as usize))+ .collect::<Vec<_>>();+ let src_w_idxs = (0..dst_w)+ .map(|w_idx| usize::min(src_w - 1, (w_idx as f64 * scale_w) as usize))+ .collect::<Vec<_>>();+ for b_idx in 0..b_sz {+ let dst = &mut dst[b_idx * c * dst_h * dst_w..];+ let src_index = src_index + b_idx * stride[0];+ for c_idx in 0..c {+ let dst = &mut dst[c_idx * dst_h * dst_w..];+ let src_index = src_index + c_idx * stride[1];+ for (h_idx, src_h_idx) in src_h_idxs.iter().enumerate() {+ for (w_idx, src_w_idx) in src_w_idxs.iter().enumerate() {+ let src_index = src_index + src_h_idx * stride_h + src_w_idx * stride_w;+ dst[h_idx * dst_w + w_idx] = src[src_index]+ }+ }+ }+ }+ Ok(dst)+ }+}++struct UpsampleBilinear2D {+ target_h: usize,+ target_w: usize,+ align_corners: bool,+ scale_h_factor: Option<f64>,+ scale_w_factor: Option<f64>,+}++impl Map1 for UpsampleBilinear2D {+ fn f<T: WithDType>(&self, src: &[T], layout: &Layout) -> Result<Vec<T>> {+ let (batch, channels, height_in, width_in) = layout.shape().dims4()?;+ let height_out = self.target_h;+ let width_out = self.target_w;++ // Early return for identity case+ if height_in == height_out && width_in == width_out {+ return Ok(src.to_vec());+ }++ let stride = layout.stride();+ let src_offset = layout.start_offset();++ // Calculate scale factors following PyTorch's area_pixel_compute_scale logic+ let scale_h = if self.align_corners {+ if height_out > 1 {+ (height_in - 1) as f64 / (height_out - 1) as f64+ } else {+ 0.0+ }+ } else {+ // PyTorch's compute_scales_value logic:+ // If scale_factor was provided, use 1.0 / scale_factor+ // Otherwise, use input_size / output_size+ if let Some(scale_factor) = self.scale_h_factor {+ 1.0 / scale_factor+ } else {+ height_in as f64 / height_out as f64+ }+ };++ let scale_w = if self.align_corners {+ if width_out > 1 {+ (width_in - 1) as f64 / (width_out - 1) as f64+ } else {+ 0.0+ }+ } else if let Some(scale_factor) = self.scale_w_factor {+ 1.0 / scale_factor+ } else {+ width_in as f64 / width_out as f64+ };++ // Precompute indices and weights for height+ let mut h_indices = Vec::with_capacity(height_out);+ for h_out in 0..height_out {+ let src_h = if self.align_corners {+ scale_h * h_out as f64+ } else {+ scale_h * (h_out as f64 + 0.5) - 0.5+ };+ let src_h_clamped = src_h.max(0.0);+ let h0 = src_h_clamped.floor() as usize;+ let h1 = (h0 + 1).min(height_in - 1);+ let weight_h = (src_h_clamped - h0 as f64).clamp(0.0, 1.0);+ h_indices.push((h0, h1, weight_h));+ }++ // Precompute indices and weights for width+ let mut w_indices = Vec::with_capacity(width_out);+ for w_out in 0..width_out {+ let src_w = if self.align_corners {+ scale_w * w_out as f64+ } else {+ scale_w * (w_out as f64 + 0.5) - 0.5+ };+ let src_w_clamped = src_w.max(0.0);+ let w0 = src_w_clamped.floor() as usize;+ let w1 = (w0 + 1).min(width_in - 1);+ let weight_w = (src_w_clamped - w0 as f64).clamp(0.0, 1.0);+ w_indices.push((w0, w1, weight_w));+ }++ // Allocate output+ let mut dst = vec![T::zero(); batch * channels * height_out * width_out];++ // Perform bilinear interpolation+ for b in 0..batch {+ for c in 0..channels {+ let base_idx = src_offset + b * stride[0] + c * stride[1];+ let dst_base = (b * channels + c) * height_out * width_out;++ for (h_out, &(h0, h1, weight_h)) in h_indices.iter().enumerate() {+ for (w_out, &(w0, w1, weight_w)) in w_indices.iter().enumerate() {+ // Get four neighboring pixels+ let idx_00 = base_idx + h0 * stride[2] + w0 * stride[3];+ let idx_10 = base_idx + h0 * stride[2] + w1 * stride[3];+ let idx_01 = base_idx + h1 * stride[2] + w0 * stride[3];+ let idx_11 = base_idx + h1 * stride[2] + w1 * stride[3];++ let v00 = src[idx_00].to_f64();+ let v10 = src[idx_10].to_f64();+ let v01 = src[idx_01].to_f64();+ let v11 = src[idx_11].to_f64();++ // Bilinear interpolation+ let v_top = v00 * (1.0 - weight_w) + v10 * weight_w;+ let v_bottom = v01 * (1.0 - weight_w) + v11 * weight_w;+ let value = v_top * (1.0 - weight_h) + v_bottom * weight_h;++ dst[dst_base + h_out * width_out + w_out] = T::from_f64(value);+ }+ }+ }+ }++ Ok(dst)+ }+}++struct Gather<'a, I: IntDType> {+ ids: &'a [I],+ ids_l: &'a Layout,+ dim: usize,+}++impl<I: IntDType> Map1 for Gather<'_, I> {+ fn f<T: WithDType>(&self, src: &[T], src_l: &Layout) -> Result<Vec<T>> {+ let ids = match self.ids_l.contiguous_offsets() {+ Some((a, b)) => &self.ids[a..b],+ None => Err(Error::RequiresContiguous { op: "gather" }.bt())?,+ };+ let src = match src_l.contiguous_offsets() {+ Some((a, b)) => &src[a..b],+ None => Err(Error::RequiresContiguous { op: "gather" }.bt())?,+ };+ let dim = self.dim;+ let ids_dims = self.ids_l.dims();+ let src_dims = src_l.dims();+ let dst_len: usize = ids_dims.iter().product();+ let dst_left_len: usize = ids_dims[..dim].iter().product();+ let dst_dim_len = ids_dims[dim];+ let dst_right_len: usize = ids_dims[dim + 1..].iter().product();++ let src_dim_len = src_dims[dim];+ let src_right_len: usize = src_dims[dim + 1..].iter().product();++ let mut dst = vec![T::zero(); dst_len];+ for left_i in 0..dst_left_len {+ let start_src_idx = left_i * src_right_len * src_dim_len;+ let start_dst_idx = left_i * dst_right_len * dst_dim_len;+ for i in 0..dst_dim_len {+ let start_dst_idx = start_dst_idx + i * dst_right_len;+ for right_i in 0..dst_right_len {+ let dst_idx = start_dst_idx + right_i;+ let index = ids[dst_idx];+ if index == I::max_value() {+ dst[dst_idx] = T::zero();+ } else {+ let index = index.as_usize();+ if index >= src_dim_len {+ Err(Error::InvalidIndex {+ index,+ size: src_dim_len,+ op: "gather",+ }+ .bt())?+ }+ let src_idx = start_src_idx + index * src_right_len + right_i;+ dst[dst_idx] = src[src_idx]+ }+ }+ }+ }+ Ok(dst)+ }+}++struct IndexSelect<'a, T: IntDType> {+ ids: &'a [T],+ ids_l: &'a Layout,+ dim: usize,+}++impl<I: IntDType> Map1 for IndexSelect<'_, I> {+ fn f<T: WithDType>(&self, src: &[T], layout: &Layout) -> Result<Vec<T>> {+ let src = match layout.contiguous_offsets() {+ Some((a, b)) => &src[a..b],+ None => Err(Error::RequiresContiguous { op: "index-select" }.bt())?,+ };+ let dim = self.dim;+ let n_ids = match self.ids_l.dims() {+ [n_ids] => *n_ids,+ d => Err(Error::UnexpectedNumberOfDims {+ expected: 1,+ got: d.len(),+ shape: self.ids_l.shape().clone(),+ }+ .bt())?,+ };+ let stride_ids = self.ids_l.stride()[0];+ let mut dst_dims = layout.dims().to_vec();+ let src_dim = dst_dims[dim];+ dst_dims[dim] = n_ids;+ let dst_len: usize = dst_dims.iter().product();+ let left_len: usize = dst_dims[..dim].iter().product();+ let right_len: usize = dst_dims[dim + 1..].iter().product();+ let mut dst = vec![T::zero(); dst_len];+ for left_i in 0..left_len {+ let start_src_idx = left_i * right_len * src_dim;+ let start_dst_idx = left_i * right_len * n_ids;+ for i in 0..n_ids {+ let start_dst_idx = start_dst_idx + i * right_len;+ let index = self.ids[self.ids_l.start_offset() + stride_ids * i];+ if index == I::max_value() {+ dst[start_dst_idx..start_dst_idx + right_len].fill(T::zero());+ } else {+ let index = index.as_usize();+ if index >= src_dim {+ Err(Error::InvalidIndex {+ index,+ size: src_dim,+ op: "index-select",+ }+ .bt())?+ }+ let start_src_idx = start_src_idx + index * right_len;+ dst[start_dst_idx..start_dst_idx + right_len]+ .copy_from_slice(&src[start_src_idx..start_src_idx + right_len])+ }+ }+ }+ Ok(dst)+ }+}++trait ElemUpdate {+ fn f<T: WithDType>(dst: &mut T, src: T);+}++struct Set;+struct Add;++impl ElemUpdate for Set {+ fn f<T: WithDType>(dst: &mut T, src: T) {+ *dst = src+ }+}++impl ElemUpdate for Add {+ fn f<T: WithDType>(dst: &mut T, src: T) {+ *dst += src+ }+}++struct Scatter<'a, I: IntDType, M: ElemUpdate> {+ ids: &'a [I],+ ids_l: &'a Layout,+ dim: usize,+ _phantom: std::marker::PhantomData<M>,+}++impl<'a, I: IntDType, M: ElemUpdate> Scatter<'a, I, M> {+ fn new(ids: &'a [I], ids_l: &'a Layout, dim: usize) -> Self {+ Self {+ ids,+ ids_l,+ dim,+ _phantom: Default::default(),+ }+ }+}++impl<I: IntDType, M: ElemUpdate> Map2InPlace for Scatter<'_, I, M> {+ const OP: &'static str = "scatter";+ fn f<T: WithDType>(+ &self,+ dst: &mut [T],+ dst_l: &Layout,+ src: &[T],+ src_l: &Layout,+ ) -> Result<()> {+ let dst = match dst_l.contiguous_offsets() {+ None => Err(Error::RequiresContiguous { op: "scatter" }.bt())?,+ Some((o1, o2)) => &mut dst[o1..o2],+ };++ let src = match src_l.contiguous_offsets() {+ None => Err(Error::RequiresContiguous { op: "scatter" }.bt())?,+ Some((o1, o2)) => &src[o1..o2],+ };++ let dim = self.dim;+ let ids_dims = self.ids_l.dims();+ let dst_dims = dst_l.dims();+ let dst_dim_len = dst_dims[dim];+ let dst_right_len: usize = dst_dims[dim + 1..].iter().product();++ let ids_left_len: usize = ids_dims[..dim].iter().product();+ let ids_dim_len = ids_dims[dim];+ let ids_right_len: usize = ids_dims[dim + 1..].iter().product();++ let ids = match self.ids_l.contiguous_offsets() {+ Some((a, b)) => &self.ids[a..b],+ None => Err(Error::RequiresContiguous { op: "scatter" }.bt())?,+ };+ for left_i in 0..ids_left_len {+ let start_ids_idx = left_i * ids_right_len * ids_dim_len;+ let start_dst_idx = left_i * dst_right_len * dst_dim_len;+ for i in 0..ids_dim_len {+ let start_ids_idx = start_ids_idx + i * ids_right_len;+ for right_i in 0..dst_right_len {+ let ids_idx = start_ids_idx + right_i;+ let index = ids[ids_idx];+ if index == I::max_value() {+ continue;+ }+ let index = index.as_usize();+ if index >= dst_dim_len {+ Err(Error::InvalidIndex {+ index,+ size: dst_dim_len,+ op: "gather",+ }+ .bt())?+ }+ let dst_idx = start_dst_idx + index * dst_right_len + right_i;+ M::f(&mut dst[dst_idx], src[ids_idx])+ }+ }+ }++ Ok(())+ }+}++struct IndexAdd<'a, I: IntDType> {+ ids: &'a [I],+ dim: usize,+}++impl<I: IntDType> Map2 for IndexAdd<'_, I> {+ const OP: &'static str = "index-add";+ // https://pytorch.org/docs/stable/generated/torch.Tensor.index_add_.html#torch.Tensor.index_add_+ // v1, l1 -> self+ fn f<T: WithDType>(&self, v1: &[T], l1: &Layout, src: &[T], src_l: &Layout) -> Result<Vec<T>> {+ let dst_len = l1.shape().elem_count();+ let mut dst = vec![T::zero(); dst_len];+ copy_strided_src_(v1, &mut dst, 0, l1);+ let src = match src_l.contiguous_offsets() {+ None => Err(Error::RequiresContiguous { op: "index-add" }.bt())?,+ Some((o1, o2)) => &src[o1..o2],+ };+ let dim = self.dim;+ let max_idx = l1.dims()[dim];+ let pre_dim = src_l.dims()[..dim].iter().product::<usize>();+ let src_dim_sz = src_l.dims()[dim];+ let post_dim = src_l.dims()[dim + 1..].iter().product::<usize>();+ if dim == 0 {+ for (src_idx, dst_idx) in self.ids.iter().enumerate() {+ if *dst_idx == I::max_value() {+ continue;+ }+ let dst_idx = dst_idx.as_usize();+ if dst_idx >= max_idx {+ Err(Error::InvalidIndex {+ index: dst_idx,+ op: "index-add",+ size: max_idx,+ })?+ }+ let src_idx = src_idx * post_dim;+ let dst_idx = dst_idx * post_dim;+ let src = &src[src_idx..src_idx + post_dim];+ let dst = &mut dst[dst_idx..dst_idx + post_dim];+ for (d, &s) in dst.iter_mut().zip(src.iter()) {+ *d += s+ }+ }+ } else {+ for (src_idx, dst_idx) in self.ids.iter().enumerate() {+ if *dst_idx == I::max_value() {+ continue;+ }+ let dst_idx = dst_idx.as_usize();+ if dst_idx >= max_idx {+ Err(Error::InvalidIndex {+ index: dst_idx,+ op: "index-add",+ size: max_idx,+ })?+ }+ for pre_i in 0..pre_dim {+ let pre_src_i = (pre_i * src_dim_sz + src_idx) * post_dim;+ let pre_dst_i = (pre_i * max_idx + dst_idx) * post_dim;+ let src = &src[pre_src_i..pre_src_i + post_dim];+ let dst = &mut dst[pre_dst_i..pre_dst_i + post_dim];+ for (d, &s) in dst.iter_mut().zip(src.iter()) {+ *d += s+ }+ }+ }+ }+ Ok(dst)+ }+}++#[allow(clippy::too_many_arguments)]+fn copy2d_<T: Copy>(+ src: &[T],+ dst: &mut [T],+ d1: usize,+ d2: usize,+ src_stride1: usize,+ dst_stride1: usize,+ src_offset: usize,+ dst_offset: usize,+) {+ // Both are contiguous - one memcpy covers everything.+ if src_stride1 == d2 && dst_stride1 == d2 {+ let src_start = src_offset;+ let dst_start = dst_offset;+ dst[dst_start..dst_start + d1 * d2].copy_from_slice(&src[src_start..src_start + d1 * d2]);+ return;+ }+ let mut src_idx = src_offset;+ let mut dst_idx = dst_offset;+ for _ in 0..d1 {+ dst[dst_idx..dst_idx + d2].copy_from_slice(&src[src_idx..src_idx + d2]);+ src_idx += src_stride1;+ dst_idx += dst_stride1;+ }+}++fn copy_strided_src_<T: Copy>(src: &[T], dst: &mut [T], dst_offset: usize, src_l: &Layout) {+ match src_l.strided_blocks() {+ crate::StridedBlocks::SingleBlock { start_offset, len } => dst+ [dst_offset..dst_offset + len]+ .copy_from_slice(&src[start_offset..start_offset + len]),+ crate::StridedBlocks::UniformBlocks {+ start_offset,+ block_len,+ count,+ src_stride,+ } => copy2d_(+ src,+ dst,+ count,+ block_len,+ src_stride,+ block_len,+ start_offset,+ dst_offset,+ ),+ crate::StridedBlocks::MultipleBlocks {+ block_start_index,+ block_len: 1,+ } => {+ let n = block_start_index.len();+ let dst = &mut dst[dst_offset..dst_offset + n];+ for (dst_elem, src_index) in dst.iter_mut().zip(block_start_index) {+ *dst_elem = src[src_index]+ }+ }+ crate::StridedBlocks::MultipleBlocks {+ block_start_index,+ block_len,+ } => {+ let n_blocks = block_start_index.len();+ let dst = &mut dst[dst_offset..dst_offset + n_blocks * block_len];+ let mut dst_index = 0;+ for src_index in block_start_index {+ dst[dst_index..dst_index + block_len]+ .copy_from_slice(&src[src_index..src_index + block_len]);+ dst_index += block_len;+ }+ }+ }+}++struct Conv1D<'a>(&'a crate::conv::ParamsConv1D);++impl Map2 for Conv1D<'_> {+ const OP: &'static str = "conv1d";+ fn f<T: WithDType>(&self, inp: &[T], inp_l: &Layout, k: &[T], k_l: &Layout) -> Result<Vec<T>> {+ let p = self.0;+ let inp = &inp[inp_l.start_offset()..];+ let k = &k[k_l.start_offset()..];+ let (inp_s0, inp_s1, inp_s2) = crate::shape::dims3(inp_l.stride())?;+ let (k_s0, k_s1, k_s2) = crate::shape::dims3(k_l.stride())?;+ let l_out = p.l_out();+ let dst_elems = p.c_out * l_out * p.b_size;+ // The output shape is [b_size, c_out, l_out]+ let dst = vec![T::zero(); dst_elems];++ // TODO: Avoid making this copy if `inp` already has the appropriate layout.+ let mut inp_cont = vec![T::zero(); p.b_size * p.c_in * p.l_in];+ for b_idx in 0..p.b_size {+ for src_l in 0..p.l_in {+ for src_c_idx in 0..p.c_in {+ let inp_idx = b_idx * inp_s0 + src_c_idx * inp_s1 + src_l * inp_s2;+ inp_cont[b_idx * p.l_in * p.c_in + src_l * p.c_in + src_c_idx] = inp[inp_idx]+ }+ }+ }++ for offset in 0..p.k_size {+ (0..p.c_out).into_par_iter().for_each(|dst_c_idx| {+ let dst_idx = dst_c_idx * l_out;+ let k_cont = (0..p.c_in)+ .map(|c_in_idx| k[dst_c_idx * k_s0 + c_in_idx * k_s1 + offset * k_s2])+ .collect::<Vec<_>>();+ for b_idx in 0..p.b_size {+ let dst_idx = dst_idx + b_idx * p.c_out * l_out;+ for dst_l in 0..l_out {+ let dst_idx = dst_idx + dst_l;+ let src_l = p.stride * dst_l + offset * p.dilation;+ if src_l < p.padding || src_l >= p.padding + p.l_in {+ continue;+ }+ let src_l = src_l - p.padding;+ let inp_cont = &inp_cont[b_idx * p.l_in * p.c_in + src_l * p.c_in..];+ assert!(inp_cont.len() >= p.c_in);+ assert!(k_cont.len() >= p.c_in);+ let mut d = T::zero();+ unsafe { T::vec_dot(inp_cont.as_ptr(), k_cont.as_ptr(), &mut d, p.c_in) }+ let dst_p = dst.as_ptr();+ // Safety: dst_idx are uniques per dst_c_idx which is used to parallelise+ // the different tasks so no two threads can try to write at the same+ // location.+ unsafe {+ let ptr = dst_p.add(dst_idx) as *mut T;+ *ptr += d+ }+ }+ }+ })+ }+ Ok(dst)+ }+}++struct Im2Col1D {+ l_k: usize,+ stride: usize,+ dilation: usize,+ padding: usize,+}++impl Im2Col1D {+ fn l_out(&self, l: usize) -> usize {+ (l + 2 * self.padding - self.dilation * (self.l_k - 1) - 1) / self.stride + 1+ }+}++impl Map1 for Im2Col1D {+ fn f<T: WithDType>(&self, vs: &[T], layout: &Layout) -> Result<Vec<T>> {+ let &Self {+ l_k,+ stride,+ dilation,+ padding,+ } = self;+ let (b, c, l) = layout.shape().dims3()?;+ let l_out = self.l_out(l);+ let src = &vs[layout.start_offset()..];+ let mut dst = vec![T::zero(); b * l_out * c * l_k];+ let (src_s0, src_s1, src_s2) = {+ let s = layout.stride();+ (s[0], s[1], s[2])+ };+ // TODO: provide specialized kernels for the common use cases.+ // - l_k = 1+ // - padding = 0+ // - stride = 1+ // - dilation = 1+ for b_idx in 0..b {+ let src_idx = b_idx * src_s0;+ let dst_idx = b_idx * l_out * c * l_k;+ for l_idx in 0..l_out {+ let dst_idx = dst_idx + l_idx * c * l_k;+ for c_idx in 0..c {+ let dst_idx = dst_idx + c_idx * l_k;+ let src_idx = c_idx * src_s1 + src_idx;+ for l_k_idx in 0..l_k {+ let src_l = l_idx * stride + l_k_idx * dilation;+ if padding != 0 && (src_l < padding || src_l >= l + padding) {+ continue;+ }+ let src_l = src_l - padding;+ let src_idx = src_idx + src_l * src_s2;+ let dst_idx = dst_idx + l_k_idx;+ dst[dst_idx] = src[src_idx]+ }+ }+ }+ }+ Ok(dst)+ }+}++struct Im2Col {+ h_k: usize,+ w_k: usize,+ stride: usize,+ dilation: usize,+ padding: usize,+}++impl Im2Col {+ fn hw_out(&self, h: usize, w: usize) -> (usize, usize) {+ let h_out = (h + 2 * self.padding - self.dilation * (self.h_k - 1) - 1) / self.stride + 1;+ let w_out = (w + 2 * self.padding - self.dilation * (self.w_k - 1) - 1) / self.stride + 1;+ (h_out, w_out)+ }+}++impl Map1 for Im2Col {+ fn f<T: WithDType>(&self, vs: &[T], layout: &Layout) -> Result<Vec<T>> {+ let &Self {+ h_k,+ w_k,+ stride,+ dilation,+ padding,+ } = self;+ let (b, c, h, w) = layout.shape().dims4()?;+ let (h_out, w_out) = self.hw_out(h, w);+ let src = &vs[layout.start_offset()..];+ let mut dst = vec![T::zero(); b * h_out * w_out * c * h_k * w_k];+ let (src_s0, src_s1, src_s2, src_s3) = {+ let s = layout.stride();+ (s[0], s[1], s[2], s[3])+ };+ // TODO: provide specialized kernels for the common use cases.+ // - h_k = w_k = 1+ // - padding = 0+ // - stride = 1+ // - dilation = 1+ for b_idx in 0..b {+ let src_idx = b_idx * src_s0;+ let dst_idx = b_idx * h_out * w_out * c * h_k * w_k;+ for h_idx in 0..h_out {+ let dst_idx = dst_idx + h_idx * w_out * c * h_k * w_k;+ for w_idx in 0..w_out {+ let dst_idx = dst_idx + w_idx * c * h_k * w_k;+ for c_idx in 0..c {+ let dst_idx = dst_idx + c_idx * h_k * w_k;+ let src_idx = c_idx * src_s1 + src_idx;+ for h_k_idx in 0..h_k {+ let src_h = h_idx * stride + h_k_idx * dilation;+ if padding != 0 && (src_h < padding || src_h >= h + padding) {+ continue;+ }+ let src_h = src_h - padding;+ let src_idx = src_idx + src_h * src_s2;+ let dst_idx = dst_idx + h_k_idx * w_k;+ for w_k_idx in 0..w_k {+ let src_w = w_idx * stride + w_k_idx * dilation;+ if padding != 0 && (src_w < padding || src_w >= w + padding) {+ continue;+ }+ let src_w = src_w - padding;+ let src_idx = src_idx + src_w * src_s3;+ let dst_idx = dst_idx + w_k_idx;+ dst[dst_idx] = src[src_idx]+ }+ }+ }+ }+ }+ }+ Ok(dst)+ }+}++struct Col2Im1D {+ stride: usize,+}++impl Map1 for Col2Im1D {+ fn f<T: WithDType>(&self, col: &[T], l: &Layout) -> Result<Vec<T>> {+ let (b_size, l_in, c_out, k_size) = l.shape().dims4()?;+ let stride = self.stride;+ let l_out = (l_in - 1) * stride + k_size;+ let mut im = vec![T::zero(); b_size * c_out * l_out];+ let (dst_s0, dst_s1) = (c_out * l_out, l_out);+ let (src_s0, src_s1, src_s2) = (c_out * k_size * l_in, c_out * k_size, k_size);+ for l_in_i in 0..l_in {+ for k_i in 0..k_size {+ let l_out_i = l_in_i * stride + k_i;+ for b_i in 0..b_size {+ for c_i in 0..c_out {+ let dst_idx = b_i * dst_s0 + c_i * dst_s1 + l_out_i;+ let src_idx = b_i * src_s0 + l_in_i * src_s1 + c_i * src_s2 + k_i;+ im[dst_idx] += col[src_idx]+ }+ }+ }+ }+ Ok(im)+ }+}++struct ConvTranspose1D<'a>(&'a crate::conv::ParamsConvTranspose1D);++impl Map2 for ConvTranspose1D<'_> {+ const OP: &'static str = "conv_transpose1d";+ fn f<T: WithDType>(&self, inp: &[T], inp_l: &Layout, k: &[T], k_l: &Layout) -> Result<Vec<T>> {+ let p = self.0;+ let inp = &inp[inp_l.start_offset()..];+ let k = &k[k_l.start_offset()..];+ let (inp_s0, inp_s1, inp_s2) = crate::shape::dims3(inp_l.stride())?;+ let (k_s0, k_s1, k_s2) = crate::shape::dims3(k_l.stride())?;+ let l_out = p.l_out();++ // Output shape: [b_size, c_out, l_out].+ let dst_elems = p.c_out * l_out * p.b_size;+ let dst = vec![T::zero(); dst_elems];+ let dst_s0 = p.c_out * l_out;+ let dst_s1 = l_out;+ let dst_s2 = 1;++ // TODO: Avoid making this copy if `inp` already has the appropriate layout.+ let mut inp_cont = vec![T::zero(); p.b_size * p.c_in * p.l_in];+ let cont_s0 = p.l_in * p.c_in;+ let cont_s1 = p.c_in;+ for b_idx in 0..p.b_size {+ for l_idx in 0..p.l_in {+ for c_idx in 0..p.c_in {+ let src_idx = b_idx * inp_s0 + c_idx * inp_s1 + l_idx * inp_s2;+ let dst_idx = b_idx * cont_s0 + l_idx * cont_s1 + c_idx;+ inp_cont[dst_idx] = inp[src_idx]+ }+ }+ }++ for k_idx in 0..p.k_size {+ (0..p.c_out).into_par_iter().for_each(|dst_c_idx| {+ let k_cont = (0..p.c_in)+ .map(|c_in_idx| k[c_in_idx * k_s0 + dst_c_idx * k_s1 + k_idx * k_s2])+ .collect::<Vec<_>>();+ for b_idx in 0..p.b_size {+ for l_idx in 0..p.l_in {+ let out_idx = l_idx * p.stride + k_idx * p.dilation;+ if out_idx < p.padding {+ continue;+ }+ let out_idx = out_idx - p.padding;+ if out_idx < l_out {+ let inp_cont = &inp_cont[b_idx * cont_s0 + l_idx * cont_s1..];+ let dst_idx = b_idx * dst_s0 + out_idx * dst_s2 + dst_c_idx * dst_s1;+ let mut d = T::zero();+ unsafe {+ T::vec_dot(inp_cont.as_ptr(), k_cont.as_ptr(), &mut d, p.c_in)+ }+ let dst_p = dst.as_ptr();+ // Safety: dst_idx are uniques per dst_c_idx which is used to+ // parallelise the different tasks so no two threads can try to+ // write at the same location.+ unsafe {+ let ptr = dst_p.add(dst_idx) as *mut T;+ *ptr += d+ }+ }+ }+ }+ })+ }+ Ok(dst)+ }+}++struct ConvTranspose2D<'a>(&'a crate::conv::ParamsConvTranspose2D);++impl Map2 for ConvTranspose2D<'_> {+ const OP: &'static str = "conv_transpose2d";+ fn f<T: WithDType>(&self, inp: &[T], inp_l: &Layout, k: &[T], k_l: &Layout) -> Result<Vec<T>> {+ let p = self.0;+ let inp = &inp[inp_l.start_offset()..];+ let (inp_s0, inp_s1, inp_s2, inp_s3) = crate::shape::dims4(inp_l.stride())?;+ let k = &k[k_l.start_offset()..];+ let (k_s0, k_s1, k_s2, k_s3) = crate::shape::dims4(k_l.stride())?;+ let (out_h, out_w) = (p.out_h(), p.out_w());++ // Output shape: [b_size, c_out, out_h, out_w].+ let dst = vec![T::zero(); p.b_size * p.c_out * out_h * out_w];+ let dst_s0 = p.c_out * out_h * out_w;+ let dst_s1 = out_h * out_w;+ let dst_s2 = out_w;+ let dst_s3 = 1;++ // TODO: Avoid making this copy if `inp` already has the appropriate layout.+ let mut inp_cont = vec![T::zero(); p.b_size * p.c_in * p.i_h * p.i_w];+ let cont_s0 = p.i_h * p.i_w * p.c_in;+ let cont_s1 = p.i_w * p.c_in;+ let cont_s2 = p.c_in;+ for b_idx in 0..p.b_size {+ for h_idx in 0..p.i_h {+ for w_idx in 0..p.i_w {+ for c_idx in 0..p.c_in {+ let src_idx =+ b_idx * inp_s0 + c_idx * inp_s1 + h_idx * inp_s2 + w_idx * inp_s3;+ let dst_idx = b_idx * cont_s0 + h_idx * cont_s1 + w_idx * cont_s2 + c_idx;+ inp_cont[dst_idx] = inp[src_idx]+ }+ }+ }+ }++ for k_y in 0..p.k_h {+ for k_x in 0..p.k_w {+ (0..p.c_out).into_par_iter().for_each(|dst_c_idx| {+ let k_cont = (0..p.c_in)+ .map(|c_in_idx| {+ k[c_in_idx * k_s0 + dst_c_idx * k_s1 + k_y * k_s2 + k_x * k_s3]+ })+ .collect::<Vec<_>>();+ for b_idx in 0..p.b_size {+ for inp_y in 0..p.i_h {+ for inp_x in 0..p.i_w {+ let out_x = inp_x * p.stride + k_x * p.dilation;+ let out_y = inp_y * p.stride + k_y * p.dilation;+ if out_x < p.padding || out_y < p.padding {+ continue;+ }+ let out_x = out_x - p.padding;+ let out_y = out_y - p.padding;+ if out_x < out_w && out_y < out_h {+ let inp_cont = &inp_cont+ [b_idx * cont_s0 + inp_y * cont_s1 + inp_x * cont_s2..];+ let dst_idx = b_idx * dst_s0+ + out_y * dst_s2+ + out_x * dst_s3+ + dst_c_idx * dst_s1;+ let mut d = T::zero();+ unsafe {+ T::vec_dot(+ inp_cont.as_ptr(),+ k_cont.as_ptr(),+ &mut d,+ p.c_in,+ )+ }+ let dst_p = dst.as_ptr();+ // Safety: dst_idx are uniques per dst_c_idx which is used to+ // parallelise the different tasks so no two threads can try to+ // write at the same location.+ unsafe {+ let ptr = dst_p.add(dst_idx) as *mut T;+ *ptr += d+ }+ }+ }+ }+ }+ })+ }+ }+ Ok(dst)+ }+}++struct MatMul((usize, usize, usize, usize));++impl MatMul {+ fn striding_error(&self, lhs_l: &Layout, rhs_l: &Layout, msg: &'static str) -> Error {+ Error::MatMulUnexpectedStriding(Box::new(crate::error::MatMulUnexpectedStriding {+ lhs_l: lhs_l.clone(),+ rhs_l: rhs_l.clone(),+ bmnk: self.0,+ msg,+ }))+ .bt()+ }++ fn ab_skip(&self, lhs_l: &Layout, rhs_l: &Layout) -> Result<(usize, usize)> {+ let lhs_stride = lhs_l.stride();+ let rhs_stride = rhs_l.stride();+ let rank = lhs_stride.len();+ let (_b, m, n, k) = self.0;+ let a_skip: usize = match lhs_stride[..rank - 2] {+ [s1, stride] if s1 == stride * lhs_l.dims()[1] => stride,+ [_, stride] if lhs_l.dims()[0] == 1 => stride,+ [stride, _] if lhs_l.dims()[1] == 1 => stride,+ [stride] => stride,+ [] => m * k,+ _ => Err(self.striding_error(lhs_l, rhs_l, "non-contiguous lhs"))?,+ };+ let b_skip: usize = match rhs_stride[..rank - 2] {+ [s1, stride] if s1 == stride * rhs_l.dims()[1] => stride,+ [_, stride] if rhs_l.dims()[0] == 1 => stride,+ [stride, _] if rhs_l.dims()[1] == 1 => stride,+ [stride] => stride,+ [] => n * k,+ _ => Err(self.striding_error(lhs_l, rhs_l, "non-contiguous rhs"))?,+ };+ Ok((a_skip, b_skip))+ }+}++impl Map2 for MatMul {+ const OP: &'static str = "mat_mul";++ #[cfg(all(not(feature = "mkl"), not(feature = "accelerate")))]+ fn f<T: 'static + WithDType + num_traits::Num + Copy>(+ &self,+ lhs: &[T],+ lhs_l: &Layout,+ rhs: &[T],+ rhs_l: &Layout,+ ) -> Result<Vec<T>> {+ use gemm::{gemm, Parallelism};++ match T::DTYPE {+ DType::F16 | DType::F32 | DType::F64 => {}+ _ => Err(Error::UnsupportedDTypeForOp(T::DTYPE, "matmul").bt())?,+ }++ let (b, m, n, k) = self.0;+ let lhs = &lhs[lhs_l.start_offset()..];+ let rhs = &rhs[rhs_l.start_offset()..];++ let lhs_stride = lhs_l.stride();+ let rhs_stride = rhs_l.stride();+ let rank = lhs_stride.len();+ let lhs_cs = lhs_stride[rank - 1];+ let lhs_rs = lhs_stride[rank - 2];++ let rhs_cs = rhs_stride[rank - 1];+ let rhs_rs = rhs_stride[rank - 2];++ let (a_skip, b_skip) = self.ab_skip(lhs_l, rhs_l)?;+ let c_skip: usize = m * n;++ let dst_shape: Shape = (m, n).into();+ let dst_strides = dst_shape.stride_contiguous();+ let dst_rs = dst_strides[0];+ let dst_cs = dst_strides[1];++ let mut dst = vec![T::zero(); b * m * n];+ let num_threads = crate::utils::get_num_threads();+ let parallelism = if num_threads > 1 {+ Parallelism::Rayon(num_threads)+ } else {+ Parallelism::None+ };+ let (b, m, n, k) = if b_skip == 0 && a_skip == m * k {+ // A batch-invariant rhs lets the batches of lhs stack into the rows of a single+ // (b * m, k) matrix, which the destination already matches row for row.+ // a_skip and c_skip should be updated but step is always 0 so+ // it wouldn't matter.+ (1, b * m, n, k)+ } else {+ // There is deliberately no mirrored fold for a batch-invariant lhs. Stacking the+ // batches of rhs into the columns of a (k, b * n) matrix would need those columns+ // to be adjacent, but rhs is batch-major, and the destination rows would still be+ // n apart while each merged row is b * n wide. Both make the merged gemm write+ // over itself, so a stride-zero batch on lhs goes through the loop below.+ (b, m, n, k)+ };+ for step in 0..b {+ let lhs_p = &lhs[step * a_skip..];+ let rhs_p = &rhs[step * b_skip..];+ let dst_p = &mut dst[step * c_skip..];+ unsafe {+ gemm(+ /* m: usize = */ m,+ /* n: usize = */ n,+ /* k: usize = */ k,+ /* dst: *mut T = */ dst_p.as_mut_ptr(),+ /* dst_cs: isize = */ dst_cs as isize,+ /* dst_rs: isize = */ dst_rs as isize,+ /* read_dst: bool = */ false,+ /* lhs: *const T = */ lhs_p.as_ptr(),+ /* lhs_cs: isize = */ lhs_cs as isize,+ /* lhs_rs: isize = */ lhs_rs as isize,+ /* rhs: *const T = */ rhs_p.as_ptr(),+ /* rhs_cs: isize = */ rhs_cs as isize,+ /* rhs_rs: isize = */ rhs_rs as isize,+ /* alpha: T = */ T::zero(),+ /* beta: T = */ T::one(),+ /* conj_dst: bool = */ false,+ /* conj_lhs: bool = */ false,+ /* conj_rhs: bool = */ false,+ parallelism,+ )+ }+ }+ Ok(dst)+ }++ #[cfg(feature = "accelerate")]+ fn f<T: 'static + WithDType + num_traits::Num + Copy>(+ &self,+ lhs: &[T],+ lhs_l: &Layout,+ rhs: &[T],+ rhs_l: &Layout,+ ) -> Result<Vec<T>> {+ let (b, m, n, k) = self.0;+ let lhs = &lhs[lhs_l.start_offset()..];+ let rhs = &rhs[rhs_l.start_offset()..];++ let lhs_stride = lhs_l.stride();+ let rhs_stride = rhs_l.stride();++ let (a_skip, b_skip) = self.ab_skip(lhs_l, rhs_l)?;+ let c_skip: usize = m * n;++ let rhs_m1 = rhs_stride[rhs_stride.len() - 1];+ let rhs_m2 = rhs_stride[rhs_stride.len() - 2];+ let lhs_m1 = lhs_stride[lhs_stride.len() - 1];+ let lhs_m2 = lhs_stride[lhs_stride.len() - 2];++ let (lda, transa) = if (rhs_m1 == 1 || n == 1) && (rhs_m2 == n || k == 1) {+ (n as i32, b'N')+ } else if rhs_m1 == k && rhs_m2 == 1 {+ (k as i32, b'T')+ } else {+ Err(self.striding_error(lhs_l, rhs_l, "non-contiguous rhs"))?+ };+ // The b tensor has dims batching, m, k (lhs)+ let (ldb, transb) = if (lhs_m1 == 1 || k == 1) && (lhs_m2 == k || m == 1) {+ (k as i32, b'N')+ } else if lhs_m1 == m && lhs_m2 == 1 {+ (m as i32, b'T')+ } else {+ Err(self.striding_error(lhs_l, rhs_l, "non-contiguous lhs"))?+ };++ let mut dst = vec![T::zero(); b * m * n];+ match T::DTYPE {+ DType::F16 => {+ crate::bail!("the accelerate backend does not support f16 matmul")+ }+ DType::F32 => {+ for step in 0..b {+ let lhs_p = &lhs[step * a_skip..];+ let rhs_p = &rhs[step * b_skip..];+ let dst_p = &mut dst[step * c_skip..];+ unsafe {+ let a = rhs_p.as_ptr() as *const f32;+ let b = lhs_p.as_ptr() as *const f32;+ let c = dst_p.as_mut_ptr() as *mut f32;+ let a = std::slice::from_raw_parts(a, a_skip);+ let b = std::slice::from_raw_parts(b, b_skip);+ let c = std::slice::from_raw_parts_mut(c, c_skip);+ crate::accelerate::sgemm(+ transa, transb, /* m= */ n as i32, /* n= */ m as i32,+ /* k= */ k as i32, /* alpha= */ 1., /* a= */ a,+ /* lda= */ lda, /* b= */ b, /* ldb= */ ldb,+ /* beta= */ 0., /* c= */ c, /* ldc= */ n as i32,+ )+ }+ }+ }+ DType::F64 => {+ for step in 0..b {+ let lhs_p = &lhs[step * a_skip..];+ let rhs_p = &rhs[step * b_skip..];+ let dst_p = &mut dst[step * c_skip..];+ unsafe {+ let a = rhs_p.as_ptr() as *const f64;+ let b = lhs_p.as_ptr() as *const f64;+ let c = dst_p.as_mut_ptr() as *mut f64;+ let a = std::slice::from_raw_parts(a, a_skip);+ let b = std::slice::from_raw_parts(b, b_skip);+ let c = std::slice::from_raw_parts_mut(c, c_skip);+ crate::accelerate::dgemm(+ transa, transb, /* m= */ n as i32, /* n= */ m as i32,+ /* k= */ k as i32, /* alpha= */ 1., /* a= */ a,+ /* lda= */ lda, /* b= */ b, /* ldb= */ ldb,+ /* beta= */ 0., /* c= */ c, /* ldc= */ n as i32,+ )+ }+ }+ }+ dtype => Err(Error::UnsupportedDTypeForOp(dtype, "matmul").bt())?,+ }+ Ok(dst)+ }++ #[cfg(feature = "mkl")]+ fn f<T: 'static + WithDType + num_traits::Num + Copy>(+ &self,+ lhs: &[T],+ lhs_l: &Layout,+ rhs: &[T],+ rhs_l: &Layout,+ ) -> Result<Vec<T>> {+ let (b, m, n, k) = self.0;+ let lhs = &lhs[lhs_l.start_offset()..];+ let rhs = &rhs[rhs_l.start_offset()..];++ let lhs_stride = lhs_l.stride();+ let rhs_stride = rhs_l.stride();++ let (a_skip, b_skip) = self.ab_skip(lhs_l, rhs_l)?;+ let c_skip: usize = m * n;++ let rhs_m1 = rhs_stride[rhs_stride.len() - 1];+ let rhs_m2 = rhs_stride[rhs_stride.len() - 2];+ let lhs_m1 = lhs_stride[lhs_stride.len() - 1];+ let lhs_m2 = lhs_stride[lhs_stride.len() - 2];++ let (lda, transa) = if (rhs_m1 == 1 || n == 1) && (rhs_m2 == n || k == 1) {+ (n as i32, b'N')+ } else if rhs_m1 == k && rhs_m2 == 1 {+ (k as i32, b'T')+ } else {+ Err(self.striding_error(lhs_l, rhs_l, "non-contiguous rhs"))?+ };+ // The b tensor has dims batching, m, k (lhs)+ let (ldb, transb) = if (lhs_m1 == 1 || k == 1) && (lhs_m2 == k || m == 1) {+ (k as i32, b'N')+ } else if lhs_m1 == m && lhs_m2 == 1 {+ (m as i32, b'T')+ } else {+ Err(self.striding_error(lhs_l, rhs_l, "non-contiguous lhs"))?+ };++ let mut dst = vec![T::zero(); b * m * n];+ match T::DTYPE {+ DType::F16 => {+ for step in 0..b {+ let lhs_p = &lhs[step * a_skip..];+ let rhs_p = &rhs[step * b_skip..];+ let dst_p = &mut dst[step * c_skip..];+ unsafe {+ let a = rhs_p.as_ptr() as *const f16;+ let b = lhs_p.as_ptr() as *const f16;+ let c = dst_p.as_mut_ptr() as *mut f16;+ let a = std::slice::from_raw_parts(a, a_skip);+ let b = std::slice::from_raw_parts(b, b_skip);+ let c = std::slice::from_raw_parts_mut(c, c_skip);+ crate::mkl::hgemm(+ transa,+ transb,+ /* m= */ n as i32,+ /* n= */ m as i32,+ /* k= */ k as i32,+ /* alpha= */ f16::ONE,+ /* a= */ a,+ /* lda= */ lda,+ /* b= */ b,+ /* ldb= */ ldb,+ /* beta= */ f16::ZERO,+ /* c= */ c,+ /* ldc= */ n as i32,+ )+ }+ }+ }+ DType::F32 => {+ for step in 0..b {+ let lhs_p = &lhs[step * a_skip..];+ let rhs_p = &rhs[step * b_skip..];+ let dst_p = &mut dst[step * c_skip..];+ unsafe {+ let a = rhs_p.as_ptr() as *const f32;+ let b = lhs_p.as_ptr() as *const f32;+ let c = dst_p.as_mut_ptr() as *mut f32;+ let a = std::slice::from_raw_parts(a, a_skip);+ let b = std::slice::from_raw_parts(b, b_skip);+ let c = std::slice::from_raw_parts_mut(c, c_skip);+ crate::mkl::sgemm(+ transa, transb, /* m= */ n as i32, /* n= */ m as i32,+ /* k= */ k as i32, /* alpha= */ 1., /* a= */ a,+ /* lda= */ lda, /* b= */ b, /* ldb= */ ldb,+ /* beta= */ 0., /* c= */ c, /* ldc= */ n as i32,+ )+ }+ }+ }+ DType::F64 => {+ for step in 0..b {+ let lhs_p = &lhs[step * a_skip..];+ let rhs_p = &rhs[step * b_skip..];+ let dst_p = &mut dst[step * c_skip..];+ unsafe {+ let a = rhs_p.as_ptr() as *const f64;+ let b = lhs_p.as_ptr() as *const f64;+ let c = dst_p.as_mut_ptr() as *mut f64;+ let a = std::slice::from_raw_parts(a, a_skip);+ let b = std::slice::from_raw_parts(b, b_skip);+ let c = std::slice::from_raw_parts_mut(c, c_skip);+ crate::mkl::dgemm(+ transa, transb, /* m= */ n as i32, /* n= */ m as i32,+ /* k= */ k as i32, /* alpha= */ 1., /* a= */ a,+ /* lda= */ lda, /* b= */ b, /* ldb= */ ldb,+ /* beta= */ 0., /* c= */ c, /* ldc= */ n as i32,+ )+ }+ }+ }+ dtype => Err(Error::UnsupportedDTypeForOp(dtype, "matmul").bt())?,+ }+ Ok(dst)+ }+}++fn elu<T: num_traits::Float>(v: T, alpha: T) -> T {+ if v.is_sign_positive() {+ v+ } else {+ (v.exp() - T::one()) * alpha+ }+}++impl CpuStorage {+ pub fn as_slice<D: WithDType>(&self) -> Result<&[D]> {+ D::cpu_storage_as_slice(self)+ }++ pub fn concat(storages: &[CpuStorage]) -> Result<CpuStorage> {+ let storage0 = &storages[0];+ let s = match storage0 {+ Self::U8(_) => {+ let storages = storages+ .iter()+ .map(|s| match s {+ Self::U8(s) => Ok(s.as_slice()),+ _ => crate::bail!("dtype mismatch"),+ })+ .collect::<Result<Vec<_>>>()?+ .concat();+ Self::U8(storages)+ }+ Self::U32(_) => {+ let storages = storages+ .iter()+ .map(|s| match s {+ Self::U32(s) => Ok(s.as_slice()),+ _ => crate::bail!("dtype mismatch"),+ })+ .collect::<Result<Vec<_>>>()?+ .concat();+ Self::U32(storages)+ }+ Self::I16(_) => {+ let storages = storages+ .iter()+ .map(|s| match s {+ Self::I16(s) => Ok(s.as_slice()),+ _ => crate::bail!("dtype mismatch"),+ })+ .collect::<Result<Vec<_>>>()?+ .concat();+ Self::I16(storages)+ }+ Self::I32(_) => {+ let storages = storages+ .iter()+ .map(|s| match s {+ Self::I32(s) => Ok(s.as_slice()),+ _ => crate::bail!("dtype mismatch"),+ })+ .collect::<Result<Vec<_>>>()?+ .concat();+ Self::I32(storages)+ }+ Self::I64(_) => {+ let storages = storages+ .iter()+ .map(|s| match s {+ Self::I64(s) => Ok(s.as_slice()),+ _ => crate::bail!("dtype mismatch"),+ })+ .collect::<Result<Vec<_>>>()?+ .concat();+ Self::I64(storages)+ }+ Self::BF16(_) => {+ let storages = storages+ .iter()+ .map(|s| match s {+ Self::BF16(s) => Ok(s.as_slice()),+ _ => crate::bail!("dtype mismatch"),+ })+ .collect::<Result<Vec<_>>>()?+ .concat();+ Self::BF16(storages)+ }+ Self::F16(_) => {+ let storages = storages+ .iter()+ .map(|s| match s {+ Self::F16(s) => Ok(s.as_slice()),+ _ => crate::bail!("dtype mismatch"),+ })+ .collect::<Result<Vec<_>>>()?+ .concat();+ Self::F16(storages)+ }+ Self::F32(_) => {+ let storages = storages+ .iter()+ .map(|s| match s {+ Self::F32(s) => Ok(s.as_slice()),+ _ => crate::bail!("dtype mismatch"),+ })+ .collect::<Result<Vec<_>>>()?+ .concat();+ Self::F32(storages)+ }+ Self::F64(_) => {+ let storages = storages+ .iter()+ .map(|s| match s {+ Self::F64(s) => Ok(s.as_slice()),+ _ => crate::bail!("dtype mismatch"),+ })+ .collect::<Result<Vec<_>>>()?+ .concat();+ Self::F64(storages)+ }+ Self::F8E4M3(_) => {+ let storages = storages+ .iter()+ .map(|s| match s {+ Self::F8E4M3(s) => Ok(s.as_slice()),+ _ => crate::bail!("dtype mismatch"),+ })+ .collect::<Result<Vec<_>>>()?+ .concat();+ Self::F8E4M3(storages)+ }+ Self::F6E2M3(_) => {+ let storages = storages+ .iter()+ .map(|s| match s {+ Self::F6E2M3(s) => Ok(s.as_slice()),+ _ => crate::bail!("dtype mismatch"),+ })+ .collect::<Result<Vec<_>>>()?+ .concat();+ Self::F6E2M3(storages)+ }+ Self::F6E3M2(_) => {+ let storages = storages+ .iter()+ .map(|s| match s {+ Self::F6E3M2(s) => Ok(s.as_slice()),+ _ => crate::bail!("dtype mismatch"),+ })+ .collect::<Result<Vec<_>>>()?+ .concat();+ Self::F6E3M2(storages)+ }+ Self::F4(_) => {+ let storages = storages+ .iter()+ .map(|s| match s {+ Self::F4(s) => Ok(s.as_slice()),+ _ => crate::bail!("dtype mismatch"),+ })+ .collect::<Result<Vec<_>>>()?+ .concat();+ Self::F4(storages)+ }+ Self::F8E8M0(_) => {+ let storages = storages+ .iter()+ .map(|s| match s {+ Self::F8E8M0(s) => Ok(s.as_slice()),+ _ => crate::bail!("dtype mismatch"),+ })+ .collect::<Result<Vec<_>>>()?+ .concat();+ Self::F8E8M0(storages)+ }+ };+ Ok(s)+ }+}++impl BackendStorage for CpuStorage {+ type Device = CpuDevice;++ fn dtype(&self) -> DType {+ match self {+ Self::U8(_) => DType::U8,+ Self::U32(_) => DType::U32,+ Self::I16(_) => DType::I16,+ Self::I32(_) => DType::I32,+ Self::I64(_) => DType::I64,+ Self::BF16(_) => DType::BF16,+ Self::F16(_) => DType::F16,+ Self::F32(_) => DType::F32,+ Self::F64(_) => DType::F64,+ Self::F8E4M3(_) => DType::F8E4M3,+ Self::F6E2M3(_) => DType::F6E2M3,+ Self::F6E3M2(_) => DType::F6E3M2,+ Self::F4(_) => DType::F4,+ Self::F8E8M0(_) => DType::F8E8M0,+ }+ }++ fn to_dtype(&self, layout: &Layout, dtype: DType) -> Result<Self> {+ // TODO: find a way around the quadratic number of cases below.+ match (self, dtype) {+ (Self::U8(storage), DType::BF16) => {+ let data = unary_map(storage, layout, |v| bf16::from_f32(v as f32));+ Ok(Self::BF16(data))+ }+ (Self::U32(storage), DType::BF16) => {+ let data = unary_map(storage, layout, |v| bf16::from_f32(v as f32));+ Ok(Self::BF16(data))+ }+ (Self::I64(storage), DType::BF16) => {+ let data = unary_map(storage, layout, |v| bf16::from_f32(v as f32));+ Ok(Self::BF16(data))+ }+ (Self::BF16(storage), DType::BF16) => {+ let data = unary_map(storage, layout, |v| v);+ Ok(Self::BF16(data))+ }+ (Self::F16(storage), DType::BF16) => {+ let data = unary_map(storage, layout, |v| bf16::from_f32(v.to_f32()));+ Ok(Self::BF16(data))+ }+ (Self::F32(storage), DType::BF16) => {+ let data = unary_map(storage, layout, bf16::from_f32);+ Ok(Self::BF16(data))+ }+ (Self::F64(storage), DType::BF16) => {+ let data = unary_map(storage, layout, bf16::from_f64);+ Ok(Self::BF16(data))+ }+ (Self::U8(storage), DType::F16) => {+ let data = unary_map(storage, layout, |v| f16::from_f32(v as f32));+ Ok(Self::F16(data))+ }+ (Self::U32(storage), DType::F16) => {+ let data = unary_map(storage, layout, |v| f16::from_f32(v as f32));+ Ok(Self::F16(data))+ }+ (Self::I64(storage), DType::F16) => {+ let data = unary_map(storage, layout, |v| f16::from_f32(v as f32));+ Ok(Self::F16(data))+ }+ (Self::BF16(storage), DType::F16) => {+ let data = unary_map(storage, layout, |v| f16::from_f32(v.to_f32()));+ Ok(Self::F16(data))+ }+ (Self::F16(storage), DType::F16) => {+ let data = unary_map(storage, layout, |v| v);+ Ok(Self::F16(data))+ }+ (Self::F32(storage), DType::F16) => {+ let data = unary_map(storage, layout, f16::from_f32);+ Ok(Self::F16(data))+ }+ (Self::F64(storage), DType::F16) => {+ let data = unary_map(storage, layout, f16::from_f64);+ Ok(Self::F16(data))+ }+ (Self::U8(storage), DType::F32) => {+ let data = unary_map(storage, layout, |v| v as f32);+ Ok(Self::F32(data))+ }+ (Self::U32(storage), DType::F32) => {+ let data = unary_map(storage, layout, |v| v as f32);+ Ok(Self::F32(data))+ }+ (Self::I64(storage), DType::F32) => {+ let data = unary_map(storage, layout, |v| v as f32);+ Ok(Self::F32(data))+ }+ (Self::BF16(storage), DType::F32) => {+ let data = unary_map(storage, layout, |v| v.to_f32());+ Ok(Self::F32(data))+ }+ (Self::F16(storage), DType::F32) => {+ let data = unary_map(storage, layout, |v| v.to_f32());+ Ok(Self::F32(data))+ }+ (Self::F32(storage), DType::F32) => {+ let data = unary_map(storage, layout, |v| v);+ Ok(Self::F32(data))+ }+ (Self::F64(storage), DType::F32) => {+ let data = unary_map(storage, layout, |v| v as f32);+ Ok(Self::F32(data))+ }+ (Self::U8(storage), DType::U8) => {+ let data = unary_map(storage, layout, |v| v);+ Ok(Self::U8(data))+ }+ (Self::BF16(storage), DType::U8) => {+ let data = unary_map(storage, layout, |v| v.to_f32() as u8);+ Ok(Self::U8(data))+ }+ (Self::F16(storage), DType::U8) => {+ let data = unary_map(storage, layout, |v| v.to_f32() as u8);+ Ok(Self::U8(data))+ }+ (Self::F32(storage), DType::U8) => {+ let data = unary_map(storage, layout, |v| v as u8);+ Ok(Self::U8(data))+ }+ (Self::F64(storage), DType::U8) => {+ let data = unary_map(storage, layout, |v| v as u8);+ Ok(Self::U8(data))+ }+ (Self::U32(storage), DType::U8) => {+ let data = unary_map(storage, layout, |v| v as u8);+ Ok(Self::U8(data))+ }+ (Self::I64(storage), DType::U8) => {+ let data = unary_map(storage, layout, |v| v as u8);+ Ok(Self::U8(data))+ }+ (Self::U8(storage), DType::U32) => {+ let data = unary_map(storage, layout, |v| v as u32);+ Ok(Self::U32(data))+ }+ (Self::U32(storage), DType::U32) => {+ let data = unary_map(storage, layout, |v| v);+ Ok(Self::U32(data))+ }+ (Self::I64(storage), DType::U32) => {+ let data = unary_map(storage, layout, |v| v as u32);+ Ok(Self::U32(data))+ }+ (Self::BF16(storage), DType::U32) => {+ let data = unary_map(storage, layout, |v| v.to_f32() as u32);+ Ok(Self::U32(data))+ }+ (Self::F16(storage), DType::U32) => {+ let data = unary_map(storage, layout, |v| v.to_f32() as u32);+ Ok(Self::U32(data))+ }+ (Self::F32(storage), DType::U32) => {+ let data = unary_map(storage, layout, |v| v as u32);+ Ok(Self::U32(data))+ }+ (Self::F64(storage), DType::U32) => {+ let data = unary_map(storage, layout, |v| v as u32);+ Ok(Self::U32(data))+ }+ (Self::U8(storage), DType::I64) => {+ let data = unary_map(storage, layout, |v| v as i64);+ Ok(Self::I64(data))+ }+ (Self::U32(storage), DType::I64) => {+ let data = unary_map(storage, layout, |v| v as i64);+ Ok(Self::I64(data))+ }+ (Self::I64(storage), DType::I64) => {+ let data = unary_map(storage, layout, |v| v);+ Ok(Self::I64(data))+ }+ (Self::BF16(storage), DType::I64) => {+ let data = unary_map(storage, layout, |v| v.to_f32() as i64);+ Ok(Self::I64(data))+ }+ (Self::F16(storage), DType::I64) => {+ let data = unary_map(storage, layout, |v| v.to_f32() as i64);+ Ok(Self::I64(data))+ }+ (Self::F32(storage), DType::I64) => {+ let data = unary_map(storage, layout, |v| v as i64);+ Ok(Self::I64(data))+ }+ (Self::F64(storage), DType::I64) => {+ let data = unary_map(storage, layout, |v| v as i64);+ Ok(Self::I64(data))+ }+ (Self::U8(storage), DType::F64) => {+ let data = unary_map(storage, layout, |v| v as f64);+ Ok(Self::F64(data))+ }+ (Self::U32(storage), DType::F64) => {+ let data = unary_map(storage, layout, |v| v as f64);+ Ok(Self::F64(data))+ }+ (Self::I64(storage), DType::F64) => {+ let data = unary_map(storage, layout, |v| v as f64);+ Ok(Self::F64(data))+ }+ (Self::BF16(storage), DType::F64) => {+ let data = unary_map(storage, layout, |v| v.to_f64());+ Ok(Self::F64(data))+ }+ (Self::F16(storage), DType::F64) => {+ let data = unary_map(storage, layout, |v| v.to_f64());+ Ok(Self::F64(data))+ }+ (Self::F32(storage), DType::F64) => {+ let data = unary_map(storage, layout, |v| v as f64);+ Ok(Self::F64(data))+ }+ (Self::F64(storage), DType::F64) => {+ let data = unary_map(storage, layout, |v| v);+ Ok(Self::F64(data))+ }+ // Conversions to F8E4M3+ (Self::U8(storage), DType::F8E4M3) => {+ let data = unary_map(storage, layout, |v| F8E4M3::from_f32(v as f32));+ Ok(Self::F8E4M3(data))+ }+ (Self::U32(storage), DType::F8E4M3) => {+ let data = unary_map(storage, layout, |v| F8E4M3::from_f32(v as f32));+ Ok(Self::F8E4M3(data))+ }+ (Self::I64(storage), DType::F8E4M3) => {+ let data = unary_map(storage, layout, |v| F8E4M3::from_f32(v as f32));+ Ok(Self::F8E4M3(data))+ }+ (Self::BF16(storage), DType::F8E4M3) => {+ let data = unary_map(storage, layout, |v| F8E4M3::from_f32(v.to_f32()));+ Ok(Self::F8E4M3(data))+ }+ (Self::F16(storage), DType::F8E4M3) => {+ let data = unary_map(storage, layout, |v| F8E4M3::from_f32(v.to_f32()));+ Ok(Self::F8E4M3(data))+ }+ (Self::F32(storage), DType::F8E4M3) => {+ let data = unary_map(storage, layout, F8E4M3::from_f32);+ Ok(Self::F8E4M3(data))+ }+ (Self::F64(storage), DType::F8E4M3) => {+ let data = unary_map(storage, layout, F8E4M3::from_f64);+ Ok(Self::F8E4M3(data))+ }+ (Self::F8E4M3(storage), DType::F8E4M3) => {+ let data = unary_map(storage, layout, |v| v);+ Ok(Self::F8E4M3(data))+ }+ // Conversions from F8E4M3+ (Self::F8E4M3(storage), DType::U8) => {+ let data = unary_map(storage, layout, |v| v.to_f32() as u8);+ Ok(Self::U8(data))+ }+ (Self::F8E4M3(storage), DType::U32) => {+ let data = unary_map(storage, layout, |v| v.to_f32() as u32);+ Ok(Self::U32(data))+ }+ (Self::F8E4M3(storage), DType::I64) => {+ let data = unary_map(storage, layout, |v| v.to_f32() as i64);+ Ok(Self::I64(data))+ }+ (Self::F8E4M3(storage), DType::BF16) => {+ let data = unary_map(storage, layout, |v| bf16::from_f32(v.to_f32()));+ Ok(Self::BF16(data))+ }+ (Self::F8E4M3(storage), DType::F16) => {+ let data = unary_map(storage, layout, |v| f16::from_f32(v.to_f32()));+ Ok(Self::F16(data))+ }+ (Self::F8E4M3(storage), DType::F32) => {+ let data = unary_map(storage, layout, |v| v.to_f32());+ Ok(Self::F32(data))+ }+ (Self::F8E4M3(storage), DType::F64) => {+ let data = unary_map(storage, layout, |v| v.to_f64());+ Ok(Self::F64(data))+ }+ // Conversions to I16+ (Self::U8(storage), DType::I16) => {+ let data = unary_map(storage, layout, |v| v as i16);+ Ok(Self::I16(data))+ }+ (Self::U32(storage), DType::I16) => {+ let data = unary_map(storage, layout, |v| v as i16);+ Ok(Self::I16(data))+ }+ (Self::I16(storage), DType::I16) => {+ let data = unary_map(storage, layout, |v| v);+ Ok(Self::I16(data))+ }+ (Self::I32(storage), DType::I16) => {+ let data = unary_map(storage, layout, |v| v as i16);+ Ok(Self::I16(data))+ }+ (Self::I64(storage), DType::I16) => {+ let data = unary_map(storage, layout, |v| v as i16);+ Ok(Self::I16(data))+ }+ (Self::BF16(storage), DType::I16) => {+ let data = unary_map(storage, layout, |v| v.to_f32() as i16);+ Ok(Self::I16(data))+ }+ (Self::F16(storage), DType::I16) => {+ let data = unary_map(storage, layout, |v| v.to_f32() as i16);+ Ok(Self::I16(data))+ }+ (Self::F32(storage), DType::I16) => {+ let data = unary_map(storage, layout, |v| v as i16);+ Ok(Self::I16(data))+ }+ (Self::F64(storage), DType::I16) => {+ let data = unary_map(storage, layout, |v| v as i16);+ Ok(Self::I16(data))+ }+ (Self::F8E4M3(storage), DType::I16) => {+ let data = unary_map(storage, layout, |v| v.to_f32() as i16);+ Ok(Self::I16(data))+ }+ // Conversions to I32+ (Self::U8(storage), DType::I32) => {+ let data = unary_map(storage, layout, |v| v as i32);+ Ok(Self::I32(data))+ }+ (Self::U32(storage), DType::I32) => {+ let data = unary_map(storage, layout, |v| v as i32);+ Ok(Self::I32(data))+ }+ (Self::I16(storage), DType::I32) => {+ let data = unary_map(storage, layout, |v| v as i32);+ Ok(Self::I32(data))+ }+ (Self::I32(storage), DType::I32) => {+ let data = unary_map(storage, layout, |v| v);+ Ok(Self::I32(data))+ }+ (Self::I64(storage), DType::I32) => {+ let data = unary_map(storage, layout, |v| v as i32);+ Ok(Self::I32(data))+ }+ (Self::BF16(storage), DType::I32) => {+ let data = unary_map(storage, layout, |v| v.to_f32() as i32);+ Ok(Self::I32(data))+ }+ (Self::F16(storage), DType::I32) => {+ let data = unary_map(storage, layout, |v| v.to_f32() as i32);+ Ok(Self::I32(data))+ }+ (Self::F32(storage), DType::I32) => {+ let data = unary_map(storage, layout, |v| v as i32);+ Ok(Self::I32(data))+ }+ (Self::F64(storage), DType::I32) => {+ let data = unary_map(storage, layout, |v| v as i32);+ Ok(Self::I32(data))+ }+ (Self::F8E4M3(storage), DType::I32) => {+ let data = unary_map(storage, layout, |v| v.to_f32() as i32);+ Ok(Self::I32(data))+ }+ // Conversions from I16+ (Self::I16(storage), DType::U8) => {+ let data = unary_map(storage, layout, |v| v as u8);+ Ok(Self::U8(data))+ }+ (Self::I16(storage), DType::U32) => {+ let data = unary_map(storage, layout, |v| v as u32);+ Ok(Self::U32(data))+ }+ (Self::I16(storage), DType::I64) => {+ let data = unary_map(storage, layout, |v| v as i64);+ Ok(Self::I64(data))+ }+ (Self::I16(storage), DType::BF16) => {+ let data = unary_map(storage, layout, |v| bf16::from_f32(v as f32));+ Ok(Self::BF16(data))+ }+ (Self::I16(storage), DType::F16) => {+ let data = unary_map(storage, layout, |v| f16::from_f32(v as f32));+ Ok(Self::F16(data))+ }+ (Self::I16(storage), DType::F32) => {+ let data = unary_map(storage, layout, |v| v as f32);+ Ok(Self::F32(data))+ }+ (Self::I16(storage), DType::F64) => {+ let data = unary_map(storage, layout, |v| v as f64);+ Ok(Self::F64(data))+ }+ (Self::I16(storage), DType::F8E4M3) => {+ let data = unary_map(storage, layout, |v| F8E4M3::from_f32(v as f32));+ Ok(Self::F8E4M3(data))+ }+ // Conversions from I32+ (Self::I32(storage), DType::U8) => {+ let data = unary_map(storage, layout, |v| v as u8);+ Ok(Self::U8(data))+ }+ (Self::I32(storage), DType::U32) => {+ let data = unary_map(storage, layout, |v| v as u32);+ Ok(Self::U32(data))+ }+ (Self::I32(storage), DType::I64) => {+ let data = unary_map(storage, layout, |v| v as i64);+ Ok(Self::I64(data))+ }+ (Self::I32(storage), DType::BF16) => {+ let data = unary_map(storage, layout, |v| bf16::from_f32(v as f32));+ Ok(Self::BF16(data))+ }+ (Self::I32(storage), DType::F16) => {+ let data = unary_map(storage, layout, |v| f16::from_f32(v as f32));+ Ok(Self::F16(data))+ }+ (Self::I32(storage), DType::F32) => {+ let data = unary_map(storage, layout, |v| v as f32);+ Ok(Self::F32(data))+ }+ (Self::I32(storage), DType::F64) => {+ let data = unary_map(storage, layout, |v| v as f64);+ Ok(Self::F64(data))+ }+ (Self::I32(storage), DType::F8E4M3) => {+ let data = unary_map(storage, layout, |v| F8E4M3::from_f32(v as f32));+ Ok(Self::F8E4M3(data))+ }+ // Dummy types - return error for all conversions to/from dummy types+ (_, DType::F6E2M3) | (_, DType::F6E3M2) | (_, DType::F4) | (_, DType::F8E8M0) => {+ Err(Error::UnsupportedDTypeForOp(dtype, "to_dtype").bt())+ }+ (Self::F6E2M3(_), _)+ | (Self::F6E3M2(_), _)+ | (Self::F4(_), _)+ | (Self::F8E8M0(_), _) => {+ Err(Error::UnsupportedDTypeForOp(self.dtype(), "to_dtype").bt())+ }+ }+ }++ fn reduce_op(&self, op: ReduceOp, layout: &Layout, reduce_dims: &[usize]) -> Result<Self> {+ match op {+ ReduceOp::Sum => {+ let src_dims = layout.dims();+ let mut dst_dims = src_dims.to_vec();+ for &dim in reduce_dims.iter() {+ dst_dims[dim] = 1;+ }+ let dst_shape = Shape::from(dst_dims);+ let mut reduce_dims = reduce_dims.to_vec();+ // Sort the reduce_dims as they have to be processed from left to right when converting the+ // indexes.+ reduce_dims.sort();+ let reduce_dims_and_stride: Vec<_> = reduce_dims+ .iter()+ .map(|&d| (src_dims[d], src_dims[d + 1..].iter().product::<usize>()))+ .collect();+ ReduceSum {+ dst_shape: &dst_shape,+ reduce_dims: &reduce_dims,+ reduce_dims_and_stride,+ }+ .map(self, layout)+ }+ ReduceOp::Min | ReduceOp::ArgMin | ReduceOp::Max | ReduceOp::ArgMax => {+ let reduce_dim_index = match reduce_dims {+ [reduce_dim_index] => *reduce_dim_index,+ _ => {+ let op = match op {+ ReduceOp::Min => "min",+ ReduceOp::ArgMin => "argmin",+ ReduceOp::Max => "max",+ ReduceOp::ArgMax => "argmax",+ _ => unreachable!(),+ };+ let dims = reduce_dims.to_vec();+ Err(Error::OnlySingleDimension { op, dims })?+ }+ };+ let (use_min, return_index) = match op {+ ReduceOp::Min => (true, false),+ ReduceOp::ArgMin => (true, true),+ ReduceOp::Max => (false, false),+ ReduceOp::ArgMax => (false, true),+ _ => unreachable!(),+ };+ ReduceIndex {+ reduce_dim_index,+ use_min,+ return_index,+ }+ .map(self, layout)+ }+ }+ }++ fn cmp(&self, op: CmpOp, rhs: &Self, lhs_l: &Layout, rhs_l: &Layout) -> Result<Self> {+ Cmp(op).map(self, lhs_l, rhs, rhs_l)+ }++ fn affine(&self, layout: &Layout, mul: f64, add: f64) -> Result<Self> {+ Affine(mul, add).map(self, layout)+ }++ fn avg_pool2d(+ &self,+ layout: &Layout,+ kernel_size: (usize, usize),+ stride: (usize, usize),+ ) -> Result<Self> {+ AvgPool2D(kernel_size, stride).map(self, layout)+ }++ fn max_pool2d(+ &self,+ layout: &Layout,+ kernel_size: (usize, usize),+ stride: (usize, usize),+ ) -> Result<Self> {+ MaxPool2D(kernel_size, stride).map(self, layout)+ }++ fn upsample_nearest1d(&self, layout: &Layout, sz: usize) -> Result<Self> {+ UpsampleNearest1D(sz).map(self, layout)+ }++ fn upsample_nearest2d(&self, layout: &Layout, h: usize, w: usize) -> Result<Self> {+ UpsampleNearest2D(h, w).map(self, layout)+ }++ fn upsample_bilinear2d(+ &self,+ layout: &Layout,+ h: usize,+ w: usize,+ align_corners: bool,+ scale_h: Option<f64>,+ scale_w: Option<f64>,+ ) -> Result<Self> {+ UpsampleBilinear2D {+ target_h: h,+ target_w: w,+ align_corners,+ scale_h_factor: scale_h,+ scale_w_factor: scale_w,+ }+ .map(self, layout)+ }++ fn powf(&self, layout: &Layout, e: f64) -> Result<Self> {+ use num_traits::Float;+ // TODO: Have some generic map for functions that apply on num_traits::Float elements.+ match self {+ Self::BF16(storage) => {+ let data = unary_map(storage, layout, |v| v.powf(bf16::from_f64(e)));+ Ok(Self::BF16(data))+ }+ Self::F16(storage) => {+ let data = unary_map(storage, layout, |v| v.powf(f16::from_f64(e)));+ Ok(Self::F16(data))+ }+ Self::F32(storage) => {+ let data = unary_map(storage, layout, |v| v.powf(e as f32));+ Ok(Self::F32(data))+ }+ Self::F64(storage) => {+ let data = unary_map(storage, layout, |v| v.powf(e));+ Ok(Self::F64(data))+ }+ Self::F8E4M3(storage) => {+ let data = unary_map(storage, layout, |v| v.powf(F8E4M3::from_f64(e)));+ Ok(Self::F8E4M3(data))+ }+ Self::U8(_) => Err(Error::UnsupportedDTypeForOp(DType::U8, "powf").bt()),+ Self::U32(_) => Err(Error::UnsupportedDTypeForOp(DType::U32, "powf").bt()),+ Self::I16(_) => Err(Error::UnsupportedDTypeForOp(DType::I16, "powf").bt()),+ Self::I32(_) => Err(Error::UnsupportedDTypeForOp(DType::I32, "powf").bt()),+ Self::I64(_) => Err(Error::UnsupportedDTypeForOp(DType::I64, "powf").bt()),+ Self::F6E2M3(_) => Err(Error::UnsupportedDTypeForOp(DType::F6E2M3, "powf").bt()),+ Self::F6E3M2(_) => Err(Error::UnsupportedDTypeForOp(DType::F6E3M2, "powf").bt()),+ Self::F4(_) => Err(Error::UnsupportedDTypeForOp(DType::F4, "powf").bt()),+ Self::F8E8M0(_) => Err(Error::UnsupportedDTypeForOp(DType::F8E8M0, "powf").bt()),+ }+ }++ fn elu(&self, layout: &Layout, alpha: f64) -> Result<Self> {+ // TODO: Have some generic map for functions that apply on num_traits::Float elements.+ match self {+ Self::BF16(storage) => {+ let data = unary_map(storage, layout, |v| elu(v, bf16::from_f64(alpha)));+ Ok(Self::BF16(data))+ }+ Self::F16(storage) => {+ let data = unary_map(storage, layout, |v| elu(v, f16::from_f64(alpha)));+ Ok(Self::F16(data))+ }+ Self::F32(storage) => {+ let data = unary_map(storage, layout, |v| elu(v, f32::from_f64(alpha)));+ Ok(Self::F32(data))+ }+ Self::F64(storage) => {+ let data = unary_map(storage, layout, |v| elu(v, alpha));+ Ok(Self::F64(data))+ }+ Self::F8E4M3(storage) => {+ let data = unary_map(storage, layout, |v| elu(v, F8E4M3::from_f64(alpha)));+ Ok(Self::F8E4M3(data))+ }+ Self::U8(_) => Err(Error::UnsupportedDTypeForOp(DType::U8, "elu").bt()),+ Self::U32(_) => Err(Error::UnsupportedDTypeForOp(DType::U32, "elu").bt()),+ Self::I16(_) => Err(Error::UnsupportedDTypeForOp(DType::I16, "elu").bt()),+ Self::I32(_) => Err(Error::UnsupportedDTypeForOp(DType::I32, "elu").bt()),+ Self::I64(_) => Err(Error::UnsupportedDTypeForOp(DType::I64, "elu").bt()),+ Self::F6E2M3(_) => Err(Error::UnsupportedDTypeForOp(DType::F6E2M3, "elu").bt()),+ Self::F6E3M2(_) => Err(Error::UnsupportedDTypeForOp(DType::F6E3M2, "elu").bt()),+ Self::F4(_) => Err(Error::UnsupportedDTypeForOp(DType::F4, "elu").bt()),+ Self::F8E8M0(_) => Err(Error::UnsupportedDTypeForOp(DType::F8E8M0, "elu").bt()),+ }+ }++ fn unary_impl<B: UnaryOpT>(&self, layout: &Layout) -> Result<Self> {+ match self {+ Self::BF16(storage) => {+ let data = unary_map_vec_par(storage, layout, B::bf16, B::bf16_vec);+ Ok(Self::BF16(data))+ }+ Self::F16(storage) => {+ let data = unary_map_vec_par(storage, layout, B::f16, B::f16_vec);+ Ok(Self::F16(data))+ }+ Self::F32(storage) => {+ let data = unary_map_vec_par(storage, layout, B::f32, B::f32_vec);+ Ok(Self::F32(data))+ }+ Self::F64(storage) => {+ let data = unary_map_vec_par(storage, layout, B::f64, B::f64_vec);+ Ok(Self::F64(data))+ }+ Self::U8(storage) => {+ let data = unary_map(storage, layout, B::u8);+ Ok(Self::U8(data))+ }+ Self::U32(storage) => {+ let data = unary_map(storage, layout, B::u32);+ Ok(Self::U32(data))+ }+ Self::I16(storage) => {+ let data = unary_map(storage, layout, B::i16);+ Ok(Self::I16(data))+ }+ Self::I32(storage) => {+ let data = unary_map(storage, layout, B::i32);+ Ok(Self::I32(data))+ }+ Self::I64(storage) => {+ let data = unary_map(storage, layout, B::i64);+ Ok(Self::I64(data))+ }+ Self::F8E4M3(storage) => {+ let data = unary_map(storage, layout, B::f8e4m3);+ Ok(Self::F8E4M3(data))+ }+ Self::F6E2M3(_) => Err(Error::UnsupportedDTypeForOp(DType::F6E2M3, "unary").bt()),+ Self::F6E3M2(_) => Err(Error::UnsupportedDTypeForOp(DType::F6E3M2, "unary").bt()),+ Self::F4(_) => Err(Error::UnsupportedDTypeForOp(DType::F4, "unary").bt()),+ Self::F8E8M0(_) => Err(Error::UnsupportedDTypeForOp(DType::F8E8M0, "unary").bt()),+ }+ }++ fn binary_impl<B: BinaryOpT>(+ &self,+ rhs: &Self,+ lhs_l: &Layout,+ rhs_l: &Layout,+ ) -> Result<Self> {+ match (self, rhs) {+ (Self::BF16(lhs), Self::BF16(rhs)) => {+ let data = binary_map_vec_par(+ lhs_l,+ rhs_l,+ lhs,+ rhs,+ B::bf16,+ B::bf16_vec,+ B::bf16_scalar_vec,+ );+ Ok(Self::BF16(data))+ }+ (Self::F16(lhs), Self::F16(rhs)) => {+ let data = binary_map_vec_par(+ lhs_l,+ rhs_l,+ lhs,+ rhs,+ B::f16,+ B::f16_vec,+ B::f16_scalar_vec,+ );+ Ok(Self::F16(data))+ }+ (Self::F32(lhs), Self::F32(rhs)) => {+ let data = binary_map_vec_par(+ lhs_l,+ rhs_l,+ lhs,+ rhs,+ B::f32,+ B::f32_vec,+ B::f32_scalar_vec,+ );+ Ok(Self::F32(data))+ }+ (Self::F64(lhs), Self::F64(rhs)) => {+ let data = binary_map_vec_par(+ lhs_l,+ rhs_l,+ lhs,+ rhs,+ B::f64,+ B::f64_vec,+ B::f64_scalar_vec,+ );+ Ok(Self::F64(data))+ }+ (Self::U32(lhs), Self::U32(rhs)) => {+ let data = binary_map_vec_par(+ lhs_l,+ rhs_l,+ lhs,+ rhs,+ B::u32,+ B::u32_vec,+ B::u32_scalar_vec,+ );+ Ok(Self::U32(data))+ }+ (Self::I16(lhs), Self::I16(rhs)) => {+ let data = binary_map_vec_par(+ lhs_l,+ rhs_l,+ lhs,+ rhs,+ B::i16,+ B::i16_vec,+ B::i16_scalar_vec,+ );+ Ok(Self::I16(data))+ }+ (Self::I32(lhs), Self::I32(rhs)) => {+ let data = binary_map_vec_par(+ lhs_l,+ rhs_l,+ lhs,+ rhs,+ B::i32,+ B::i32_vec,+ B::i32_scalar_vec,+ );+ Ok(Self::I32(data))+ }+ (Self::I64(lhs), Self::I64(rhs)) => {+ let data = binary_map_vec_par(+ lhs_l,+ rhs_l,+ lhs,+ rhs,+ B::i64,+ B::i64_vec,+ B::i64_scalar_vec,+ );+ Ok(Self::I64(data))+ }+ (Self::U8(lhs), Self::U8(rhs)) => {+ let data =+ binary_map_vec(lhs_l, rhs_l, lhs, rhs, B::u8, B::u8_vec, B::u8_scalar_vec);+ Ok(Self::U8(data))+ }+ (Self::F8E4M3(lhs), Self::F8E4M3(rhs)) => {+ let data = binary_map(lhs_l, rhs_l, lhs, rhs, B::f8e4m3);+ Ok(Self::F8E4M3(data))+ }+ _ => {+ // This should be covered by the dtype check above.+ Err(Error::DTypeMismatchBinaryOp {+ lhs: self.dtype(),+ rhs: rhs.dtype(),+ op: B::NAME,+ }+ .bt())+ }+ }+ }++ fn copy2d(+ &self,+ dst: &mut Self,+ d1: usize,+ d2: usize,+ src_s: usize,+ dst_s: usize,+ src_o: usize,+ dst_o: usize,+ ) -> Result<()> {+ match (self, dst) {+ (Self::U8(src), Self::U8(dst)) => copy2d_(src, dst, d1, d2, src_s, dst_s, src_o, dst_o),+ (Self::U32(src), Self::U32(dst)) => {+ copy2d_(src, dst, d1, d2, src_s, dst_s, src_o, dst_o)+ }+ (Self::I16(src), Self::I16(dst)) => {+ copy2d_(src, dst, d1, d2, src_s, dst_s, src_o, dst_o)+ }+ (Self::I32(src), Self::I32(dst)) => {+ copy2d_(src, dst, d1, d2, src_s, dst_s, src_o, dst_o)+ }+ (Self::I64(src), Self::I64(dst)) => {+ copy2d_(src, dst, d1, d2, src_s, dst_s, src_o, dst_o)+ }+ (Self::BF16(src), Self::BF16(dst)) => {+ copy2d_(src, dst, d1, d2, src_s, dst_s, src_o, dst_o)+ }+ (Self::F16(src), Self::F16(dst)) => {+ copy2d_(src, dst, d1, d2, src_s, dst_s, src_o, dst_o)+ }+ (Self::F32(src), Self::F32(dst)) => {+ copy2d_(src, dst, d1, d2, src_s, dst_s, src_o, dst_o)+ }+ (Self::F64(src), Self::F64(dst)) => {+ copy2d_(src, dst, d1, d2, src_s, dst_s, src_o, dst_o)+ }+ (Self::F8E4M3(src), Self::F8E4M3(dst)) => {+ copy2d_(src, dst, d1, d2, src_s, dst_s, src_o, dst_o)+ }+ (Self::F6E2M3(src), Self::F6E2M3(dst)) => {+ copy2d_(src, dst, d1, d2, src_s, dst_s, src_o, dst_o)+ }+ (Self::F6E3M2(src), Self::F6E3M2(dst)) => {+ copy2d_(src, dst, d1, d2, src_s, dst_s, src_o, dst_o)+ }+ (Self::F4(src), Self::F4(dst)) => copy2d_(src, dst, d1, d2, src_s, dst_s, src_o, dst_o),+ (Self::F8E8M0(src), Self::F8E8M0(dst)) => {+ copy2d_(src, dst, d1, d2, src_s, dst_s, src_o, dst_o)+ }+ (_, dst) => {+ return Err(Error::DTypeMismatchBinaryOp {+ lhs: self.dtype(),+ rhs: dst.dtype(),+ op: "copy2d",+ }+ .bt());+ }+ }+ Ok(())+ }++ fn copy_strided_src(&self, dst: &mut Self, dst_offset: usize, src_l: &Layout) -> Result<()> {+ match (self, dst) {+ (Self::U8(src), Self::U8(dst)) => copy_strided_src_(src, dst, dst_offset, src_l),+ (Self::U32(src), Self::U32(dst)) => copy_strided_src_(src, dst, dst_offset, src_l),+ (Self::I16(src), Self::I16(dst)) => copy_strided_src_(src, dst, dst_offset, src_l),+ (Self::I32(src), Self::I32(dst)) => copy_strided_src_(src, dst, dst_offset, src_l),+ (Self::I64(src), Self::I64(dst)) => copy_strided_src_(src, dst, dst_offset, src_l),+ (Self::BF16(src), Self::BF16(dst)) => copy_strided_src_(src, dst, dst_offset, src_l),+ (Self::F16(src), Self::F16(dst)) => copy_strided_src_(src, dst, dst_offset, src_l),+ (Self::F32(src), Self::F32(dst)) => copy_strided_src_(src, dst, dst_offset, src_l),+ (Self::F64(src), Self::F64(dst)) => copy_strided_src_(src, dst, dst_offset, src_l),+ (Self::F8E4M3(src), Self::F8E4M3(dst)) => {+ copy_strided_src_(src, dst, dst_offset, src_l)+ }+ (Self::F6E2M3(src), Self::F6E2M3(dst)) => {+ copy_strided_src_(src, dst, dst_offset, src_l)+ }+ (Self::F6E3M2(src), Self::F6E3M2(dst)) => {+ copy_strided_src_(src, dst, dst_offset, src_l)+ }+ (Self::F4(src), Self::F4(dst)) => copy_strided_src_(src, dst, dst_offset, src_l),+ (Self::F8E8M0(src), Self::F8E8M0(dst)) => {+ copy_strided_src_(src, dst, dst_offset, src_l)+ }+ (_, dst) => {+ // This should be covered by the dtype check above.+ return Err(Error::DTypeMismatchBinaryOp {+ lhs: self.dtype(),+ rhs: dst.dtype(),+ op: "copy_strided",+ }+ .bt());+ }+ }+ Ok(())+ }++ fn where_cond(+ &self,+ layout: &Layout,+ t: &Self,+ t_l: &Layout,+ f: &Self,+ f_l: &Layout,+ ) -> Result<Self> {+ match self {+ Self::U8(pred) => WCond(pred, layout).map(t, t_l, f, f_l),+ Self::U32(pred) => WCond(pred, layout).map(t, t_l, f, f_l),+ Self::I16(pred) => WCond(pred, layout).map(t, t_l, f, f_l),+ Self::I32(pred) => WCond(pred, layout).map(t, t_l, f, f_l),+ Self::I64(pred) => WCond(pred, layout).map(t, t_l, f, f_l),+ _ => Err(Error::UnsupportedDTypeForOp(self.dtype(), "where-cond")),+ }+ }++ fn conv1d(+ &self,+ l: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &crate::conv::ParamsConv1D,+ ) -> Result<Self> {+ if !USE_IM2COL_CONV1D {+ return Conv1D(params).map(self, l, kernel, kernel_l);+ }+ let op = Im2Col1D {+ l_k: params.k_size,+ padding: params.padding,+ stride: params.stride,+ dilation: params.dilation,+ };+ let col = op.map(self, l)?;+ let b = params.b_size;+ let n = params.c_out;+ let l_out = params.l_out();+ let k = op.l_k * params.c_in;+ let m = l_out;+ let col_l = Layout::contiguous((b, m, k));+ let res = if kernel_l.is_contiguous() {+ let kernel_l = Layout::contiguous_with_offset((1, n, k), kernel_l.start_offset())+ .transpose(1, 2)?+ .broadcast_as((b, k, n))?;+ col.matmul(kernel, (b, m, n, k), &col_l, &kernel_l)?+ } else {+ // Make the kernel contiguous if not already the case.+ let mut kernel_c = unsafe {+ self.device()+ .alloc_uninit(kernel_l.shape(), kernel.dtype())?+ };+ kernel.copy_strided_src(&mut kernel_c, 0, kernel_l)?;+ let kernel_l = Layout::contiguous_with_offset((1, n, k), kernel_l.start_offset())+ .transpose(1, 2)?+ .broadcast_as((b, k, n))?;+ col.matmul(kernel, (b, m, n, k), &col_l, &kernel_l)?+ };+ let res_l = Layout::contiguous((b, l_out, params.c_out)).transpose(1, 2)?;+ let mut res_t = unsafe { self.device().alloc_uninit(res_l.shape(), res.dtype())? };+ res.copy_strided_src(&mut res_t, 0, &res_l)?;+ Ok(res_t)+ }++ fn conv_transpose1d(+ &self,+ l: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &crate::conv::ParamsConvTranspose1D,+ ) -> Result<Self> {+ let can_use_col2im = kernel_l.is_contiguous()+ && params.dilation == 1+ && params.padding == 0+ && params.output_padding == 0;+ if USE_COL2IM_CONV1D_TR && can_use_col2im {+ let (b_size, c_in, l_in) = l.shape().dims3()?;+ let (c_in2, c_out, k_size) = kernel_l.shape().dims3()?;+ if !kernel_l.is_contiguous() {+ crate::bail!(+ "convtr1d: the second argument (kernel) has to be contiguous {kernel_l:?}"+ )+ }+ if c_in != c_in2 {+ crate::bail!(+ "convtr1d: shape mismatch on c_in {:?} {:?}",+ l.shape(),+ kernel_l.shape()+ )+ }+ let col = {+ // This merges the last two dimensions of the kernel together.+ let kernel_l_mm = Layout::new(+ (b_size, c_in, k_size * c_out).into(),+ vec![0, k_size * c_out, 1],+ kernel_l.start_offset(),+ );+ self.matmul(+ kernel,+ (+ b_size,+ /* m */ l_in,+ /* n */ c_out * k_size,+ /* k */ c_in,+ ),+ &l.transpose(1, 2)?,+ &kernel_l_mm,+ )?+ };+ let col_l = Layout::contiguous((b_size, l_in, c_out, k_size));+ Col2Im1D {+ stride: params.stride,+ }+ .map(&col, &col_l)+ } else {+ ConvTranspose1D(params).map(self, l, kernel, kernel_l)+ }+ }++ fn conv2d(+ &self,+ l: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &crate::conv::ParamsConv2D,+ ) -> Result<Self> {+ Conv2D(params).map(self, l, kernel, kernel_l)+ }++ fn conv_transpose2d(+ &self,+ l: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &crate::conv::ParamsConvTranspose2D,+ ) -> Result<Self> {+ ConvTranspose2D(params).map(self, l, kernel, kernel_l)+ }++ fn index_select(&self, ids: &Self, l: &Layout, ids_l: &Layout, dim: usize) -> Result<Self> {+ match ids {+ Self::U8(ids) => IndexSelect { ids, ids_l, dim }.map(self, l),+ Self::U32(ids) => IndexSelect { ids, ids_l, dim }.map(self, l),+ Self::I64(ids) => IndexSelect { ids, ids_l, dim }.map(self, l),+ _ => Err(Error::UnsupportedDTypeForOp(self.dtype(), "index-select").bt()),+ }+ }++ fn gather(&self, l: &Layout, ids: &Self, ids_l: &Layout, dim: usize) -> Result<Self> {+ match ids {+ Self::U8(ids) => Gather { ids, ids_l, dim }.map(self, l),+ Self::U32(ids) => Gather { ids, ids_l, dim }.map(self, l),+ Self::I64(ids) => Gather { ids, ids_l, dim }.map(self, l),+ _ => Err(Error::UnsupportedDTypeForOp(self.dtype(), "gather").bt()),+ }+ }++ fn scatter_set(+ &mut self,+ l: &Layout,+ ids: &Self,+ ids_l: &Layout,+ src: &Self,+ src_l: &Layout,+ dim: usize,+ ) -> Result<()> {+ match ids {+ Self::U8(ids) => Scatter::<_, Set>::new(ids, ids_l, dim).map(self, l, src, src_l),+ Self::U32(ids) => Scatter::<_, Set>::new(ids, ids_l, dim).map(self, l, src, src_l),+ Self::I64(ids) => Scatter::<_, Set>::new(ids, ids_l, dim).map(self, l, src, src_l),+ _ => Err(Error::UnsupportedDTypeForOp(self.dtype(), "scatter").bt()),+ }+ }++ fn scatter_add_set(+ &mut self,+ l: &Layout,+ ids: &Self,+ ids_l: &Layout,+ src: &Self,+ src_l: &Layout,+ dim: usize,+ ) -> Result<()> {+ match ids {+ Self::U8(ids) => Scatter::<_, Add>::new(ids, ids_l, dim).map(self, l, src, src_l),+ Self::U32(ids) => Scatter::<_, Add>::new(ids, ids_l, dim).map(self, l, src, src_l),+ Self::I16(ids) => Scatter::<_, Add>::new(ids, ids_l, dim).map(self, l, src, src_l),+ Self::I32(ids) => Scatter::<_, Add>::new(ids, ids_l, dim).map(self, l, src, src_l),+ Self::I64(ids) => Scatter::<_, Add>::new(ids, ids_l, dim).map(self, l, src, src_l),+ _ => Err(Error::UnsupportedDTypeForOp(self.dtype(), "scatter-add").bt()),+ }+ }++ fn index_add(+ &self,+ l: &Layout,+ ids: &Self,+ ids_l: &Layout,+ src: &Self,+ src_l: &Layout,+ dim: usize,+ ) -> Result<Self> {+ match ids {+ Self::U8(ids) => {+ let ids = match ids_l.contiguous_offsets() {+ Some((a, b)) => &ids[a..b],+ None => Err(Error::RequiresContiguous { op: "index-add" }.bt())?,+ };+ IndexAdd { ids, dim }.map(self, l, src, src_l)+ }+ Self::U32(ids) => {+ let ids = match ids_l.contiguous_offsets() {+ Some((a, b)) => &ids[a..b],+ None => Err(Error::RequiresContiguous { op: "index-add" }.bt())?,+ };+ IndexAdd { ids, dim }.map(self, l, src, src_l)+ }+ Self::I16(ids) => {+ let ids = match ids_l.contiguous_offsets() {+ Some((a, b)) => &ids[a..b],+ None => Err(Error::RequiresContiguous { op: "index-add" }.bt())?,+ };+ IndexAdd { ids, dim }.map(self, l, src, src_l)+ }+ Self::I32(ids) => {+ let ids = match ids_l.contiguous_offsets() {+ Some((a, b)) => &ids[a..b],+ None => Err(Error::RequiresContiguous { op: "index-add" }.bt())?,+ };+ IndexAdd { ids, dim }.map(self, l, src, src_l)+ }+ Self::I64(ids) => {+ let ids = match ids_l.contiguous_offsets() {+ Some((a, b)) => &ids[a..b],+ None => Err(Error::RequiresContiguous { op: "index-add" }.bt())?,+ };+ IndexAdd { ids, dim }.map(self, l, src, src_l)+ }+ _ => Err(Error::UnsupportedDTypeForOp(self.dtype(), "index-add").bt()),+ }+ }++ fn matmul(+ &self,+ rhs: &Self,+ bmnk: (usize, usize, usize, usize),+ lhs_l: &Layout,+ rhs_l: &Layout,+ ) -> Result<Self> {+ // no bf16 gemm kernel yet: widen to f32, multiply, narrow back+ if let (Self::BF16(lhs_v), Self::BF16(rhs_v)) = (self, rhs) {+ let lhs32 = Self::F32(lhs_v.to_f32_vec());+ let rhs32 = Self::F32(rhs_v.to_f32_vec());+ let out = MatMul(bmnk).map(&lhs32, lhs_l, &rhs32, rhs_l)?;+ let Self::F32(out_v) = out else {+ crate::bail!("matmul dtype mismatch")+ };+ return Ok(Self::BF16(Vec::from_f32_slice(&out_v)));+ }+ MatMul(bmnk).map(self, lhs_l, rhs, rhs_l)+ }++ fn device(&self) -> &Self::Device {+ &CpuDevice+ }++ fn try_clone(&self, _: &Layout) -> Result<Self> {+ Ok(self.clone())+ }++ fn to_cpu_storage(&self) -> Result<CpuStorage> {+ Ok(self.clone())+ }++ fn const_set(&mut self, s: crate::scalar::Scalar, l: &Layout) -> Result<()> {+ use crate::scalar::Scalar;+ fn set<T: crate::WithDType>(src: &mut [T], l: &Layout, s: T) {+ match l.strided_blocks() {+ crate::StridedBlocks::SingleBlock { start_offset, len } => {+ src[start_offset..start_offset + len].fill(s)+ }+ crate::StridedBlocks::UniformBlocks {+ start_offset,+ block_len,+ count,+ src_stride,+ } => {+ for i in 0..count {+ let start = start_offset + i * src_stride;+ src[start..start + block_len].fill(s)+ }+ }+ crate::StridedBlocks::MultipleBlocks {+ block_start_index,+ block_len: 1,+ } => {+ for src_index in block_start_index {+ src[src_index] = s+ }+ }+ crate::StridedBlocks::MultipleBlocks {+ block_start_index,+ block_len,+ } => {+ for src_index in block_start_index {+ src[src_index..src_index + block_len].fill(s)+ }+ }+ }+ }+ match (self, s) {+ (Self::BF16(storage), Scalar::BF16(v)) => set(storage, l, v),+ (Self::F16(storage), Scalar::F16(v)) => set(storage, l, v),+ (Self::F32(storage), Scalar::F32(v)) => set(storage, l, v),+ (Self::F64(storage), Scalar::F64(v)) => set(storage, l, v),+ (Self::U8(storage), Scalar::U8(v)) => set(storage, l, v),+ (Self::U32(storage), Scalar::U32(v)) => set(storage, l, v),+ (Self::I16(storage), Scalar::I16(v)) => set(storage, l, v),+ (Self::I32(storage), Scalar::I32(v)) => set(storage, l, v),+ (Self::I64(storage), Scalar::I64(v)) => set(storage, l, v),+ (Self::F8E4M3(storage), Scalar::F8E4M3(v)) => set(storage, l, v),+ // Dummy types don't support scalar operations+ (Self::F6E2M3(_), _) => {+ crate::bail!("const_set not supported for dummy type F6E2M3")+ }+ (Self::F6E3M2(_), _) => {+ crate::bail!("const_set not supported for dummy type F6E3M2")+ }+ (Self::F4(_), _) => {+ crate::bail!("const_set not supported for dummy type F4")+ }+ (Self::F8E8M0(_), _) => {+ crate::bail!("const_set not supported for dummy type F8E8M0")+ }+ (st, s) => crate::bail!(+ "const_set dtype mismatch, expected {:?} but got {:?}",+ st.dtype(),+ s+ ),+ }+ Ok(())+ }+}++impl BackendDevice for CpuDevice {+ type Storage = CpuStorage;++ fn location(&self) -> crate::DeviceLocation {+ crate::DeviceLocation::Cpu+ }++ fn same_device(&self, _: &Self) -> bool {+ true+ }++ fn storage_from_slice<T: crate::WithDType>(&self, s: &[T]) -> Result<Self::Storage> {+ Ok(T::to_cpu_storage(s))+ }++ fn storage_from_cpu_storage(&self, s: &CpuStorage) -> Result<Self::Storage> {+ Ok(s.clone())+ }++ fn storage_from_cpu_storage_owned(&self, s: CpuStorage) -> Result<Self::Storage> {+ Ok(s)+ }++ fn new(_: usize) -> Result<Self> {+ Ok(Self)+ }++ fn set_seed(&self, _seed: u64) -> Result<()> {+ crate::bail!("cannot seed the CPU rng with set_seed")+ }++ fn get_current_seed(&self) -> Result<u64> {+ crate::bail!("cannot get the CPU rng seed with get_current_seed")+ }++ fn rand_uniform(&self, shape: &Shape, dtype: DType, min: f64, max: f64) -> Result<CpuStorage> {+ use rand::prelude::*;++ let elem_count = shape.elem_count();+ let mut rng = rand::rng();+ match dtype {+ DType::U8+ | DType::U32+ | DType::I16+ | DType::I32+ | DType::I64+ | DType::F6E2M3+ | DType::F6E3M2+ | DType::F4+ | DType::F8E8M0 => Err(Error::UnsupportedDTypeForOp(dtype, "rand_uniform").bt()),+ DType::BF16 => {+ let mut data = Vec::with_capacity(elem_count);+ let uniform = rand::distr::Uniform::new(bf16::from_f64(min), bf16::from_f64(max))+ .map_err(Error::wrap)?;+ for _i in 0..elem_count {+ data.push(rng.sample::<bf16, _>(uniform))+ }+ Ok(CpuStorage::BF16(data))+ }+ DType::F16 => {+ let mut data = Vec::with_capacity(elem_count);+ let uniform = rand::distr::Uniform::new(f16::from_f64(min), f16::from_f64(max))+ .map_err(Error::wrap)?;+ for _i in 0..elem_count {+ data.push(rng.sample::<f16, _>(uniform))+ }+ Ok(CpuStorage::F16(data))+ }+ DType::F8E4M3 => {+ let mut data = Vec::with_capacity(elem_count);+ let uniform =+ rand::distr::Uniform::new(F8E4M3::from_f64(min), F8E4M3::from_f64(max))+ .map_err(Error::wrap)?;+ for _i in 0..elem_count {+ data.push(rng.sample::<F8E4M3, _>(uniform))+ }+ Ok(CpuStorage::F8E4M3(data))+ }+ DType::F32 => {+ let mut data = Vec::with_capacity(elem_count);+ let uniform =+ rand::distr::Uniform::new(min as f32, max as f32).map_err(Error::wrap)?;+ for _i in 0..elem_count {+ data.push(rng.sample::<f32, _>(uniform))+ }+ Ok(CpuStorage::F32(data))+ }+ DType::F64 => {+ let mut data = Vec::with_capacity(elem_count);+ let uniform = rand::distr::Uniform::new(min, max).map_err(Error::wrap)?;+ for _i in 0..elem_count {+ data.push(rng.sample::<f64, _>(uniform))+ }+ Ok(CpuStorage::F64(data))+ }+ }+ }++ fn rand_normal(&self, shape: &Shape, dtype: DType, mean: f64, std: f64) -> Result<CpuStorage> {+ use rand::prelude::*;++ let elem_count = shape.elem_count();+ let mut rng = rand::rng();+ match dtype {+ DType::U8+ | DType::U32+ | DType::I16+ | DType::I32+ | DType::I64+ | DType::F6E2M3+ | DType::F6E3M2+ | DType::F4+ | DType::F8E8M0 => Err(Error::UnsupportedDTypeForOp(dtype, "rand_normal").bt()),+ DType::BF16 => {+ let mut data = Vec::with_capacity(elem_count);+ let normal = rand_distr::Normal::new(bf16::from_f64(mean), bf16::from_f64(std))+ .map_err(Error::wrap)?;+ for _i in 0..elem_count {+ data.push(normal.sample(&mut rng))+ }+ Ok(CpuStorage::BF16(data))+ }+ DType::F16 => {+ let mut data = Vec::with_capacity(elem_count);+ let normal = rand_distr::Normal::new(f16::from_f64(mean), f16::from_f64(std))+ .map_err(Error::wrap)?;+ for _i in 0..elem_count {+ data.push(normal.sample(&mut rng))+ }+ Ok(CpuStorage::F16(data))+ }+ DType::F8E4M3 => {+ let mut data = Vec::with_capacity(elem_count);+ let normal = rand_distr::Normal::new(F8E4M3::from_f64(mean), F8E4M3::from_f64(std))+ .map_err(Error::wrap)?;+ for _i in 0..elem_count {+ data.push(normal.sample(&mut rng))+ }+ Ok(CpuStorage::F8E4M3(data))+ }+ DType::F32 => {+ let mut data = Vec::with_capacity(elem_count);+ let normal =+ rand_distr::Normal::new(mean as f32, std as f32).map_err(Error::wrap)?;+ for _i in 0..elem_count {+ data.push(normal.sample(&mut rng))+ }+ Ok(CpuStorage::F32(data))+ }+ DType::F64 => {+ let mut data = Vec::with_capacity(elem_count);+ let normal = rand_distr::Normal::new(mean, std).map_err(Error::wrap)?;+ for _i in 0..elem_count {+ data.push(normal.sample(&mut rng))+ }+ Ok(CpuStorage::F64(data))+ }+ }+ }++ #[allow(clippy::uninit_vec)]+ unsafe fn alloc_uninit(&self, shape: &Shape, dtype: DType) -> Result<CpuStorage> {+ let elem_count = shape.elem_count();+ // The code below is highly unsafe but hopefully not directly unsound as we only consider+ // types that are Copy, not Drop, and for which all bit patterns are proper values.+ // It's still pretty risky, see the following for more details:+ // https://github.com/rust-lang/rust-clippy/issues/4483+ let storage = match dtype {+ DType::U8 => {+ let mut v = Vec::with_capacity(elem_count);+ v.set_len(elem_count);+ CpuStorage::U8(v)+ }+ DType::U32 => {+ let mut v = Vec::with_capacity(elem_count);+ v.set_len(elem_count);+ CpuStorage::U32(v)+ }+ DType::I16 => {+ let mut v = Vec::with_capacity(elem_count);+ v.set_len(elem_count);+ CpuStorage::I16(v)+ }+ DType::I32 => {+ let mut v = Vec::with_capacity(elem_count);+ v.set_len(elem_count);+ CpuStorage::I32(v)+ }+ DType::I64 => {+ let mut v = Vec::with_capacity(elem_count);+ v.set_len(elem_count);+ CpuStorage::I64(v)+ }+ DType::BF16 => {+ let mut v = Vec::with_capacity(elem_count);+ v.set_len(elem_count);+ CpuStorage::BF16(v)+ }+ DType::F16 => {+ let mut v = Vec::with_capacity(elem_count);+ v.set_len(elem_count);+ CpuStorage::F16(v)+ }+ DType::F32 => {+ let mut v = Vec::with_capacity(elem_count);+ v.set_len(elem_count);+ CpuStorage::F32(v)+ }+ DType::F64 => {+ let mut v = Vec::with_capacity(elem_count);+ v.set_len(elem_count);+ CpuStorage::F64(v)+ }+ DType::F8E4M3 => {+ let mut v = Vec::with_capacity(elem_count);+ v.set_len(elem_count);+ CpuStorage::F8E4M3(v)+ }+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ return Err(Error::UnsupportedDTypeForOp(dtype, "alloc_uninit").bt())+ }+ };+ Ok(storage)+ }++ fn zeros_impl(&self, shape: &Shape, dtype: DType) -> Result<CpuStorage> {+ let elem_count = shape.elem_count();+ let storage = match dtype {+ DType::U8 => CpuStorage::U8(vec![0u8; elem_count]),+ DType::U32 => CpuStorage::U32(vec![0u32; elem_count]),+ DType::I16 => CpuStorage::I16(vec![0i16; elem_count]),+ DType::I32 => CpuStorage::I32(vec![0i32; elem_count]),+ DType::I64 => CpuStorage::I64(vec![0i64; elem_count]),+ DType::BF16 => CpuStorage::BF16(vec![bf16::ZERO; elem_count]),+ DType::F16 => CpuStorage::F16(vec![f16::ZERO; elem_count]),+ DType::F32 => CpuStorage::F32(vec![0f32; elem_count]),+ DType::F64 => CpuStorage::F64(vec![0f64; elem_count]),+ DType::F8E4M3 => CpuStorage::F8E4M3(vec![F8E4M3::ZERO; elem_count]),+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ return Err(Error::UnsupportedDTypeForOp(dtype, "zeros").bt())+ }+ };+ Ok(storage)+ }++ fn synchronize(&self) -> Result<()> {+ Ok(())+ }+}++#[macro_export]+macro_rules! map_dtype {+ ($name:expr, $storage:ident, $fn:expr, ($($dtypes:ident),+)) => {+ match $storage {+ $(CpuStorage::$dtypes(__e) => CpuStorage::$dtypes($fn(__e)),)*+ s => Err(Error::UnsupportedDTypeForOp(s.dtype(), $name).bt())?,+ }+ };+}diff --git a/vendor/candle-core/src/cpu_backend/utils.rs b/vendor/candle-core/src/cpu_backend/utils.rsnew file mode 100644index 0000000..f2c4c1f--- /dev/null+++ b/vendor/candle-core/src/cpu_backend/utils.rs@@ -0,0 +1,574 @@+/// Helper functions to write CPU kernels.+use crate::backend::BackendStorage;+use crate::nditer::NdIter;+use crate::{Error, Layout, Result, WithDType};++type C = super::CpuStorage;+pub trait Map1 {+ fn f<T: WithDType>(&self, vs: &[T], layout: &Layout) -> Result<Vec<T>>;++ fn map(&self, vs: &C, layout: &Layout) -> Result<C> {+ match vs {+ C::U8(vs) => Ok(C::U8(self.f(vs, layout)?)),+ C::U32(vs) => Ok(C::U32(self.f(vs, layout)?)),+ C::I16(vs) => Ok(C::I16(self.f(vs, layout)?)),+ C::I32(vs) => Ok(C::I32(self.f(vs, layout)?)),+ C::I64(vs) => Ok(C::I64(self.f(vs, layout)?)),+ C::BF16(vs) => Ok(C::BF16(self.f(vs, layout)?)),+ C::F16(vs) => Ok(C::F16(self.f(vs, layout)?)),+ C::F32(vs) => Ok(C::F32(self.f(vs, layout)?)),+ C::F64(vs) => Ok(C::F64(self.f(vs, layout)?)),+ C::F8E4M3(vs) => Ok(C::F8E4M3(self.f(vs, layout)?)),+ // Dummy types don't support Map1 operations+ C::F6E2M3(_) => Err(Error::UnsupportedDTypeForOp(vs.dtype(), "map1").bt()),+ C::F6E3M2(_) => Err(Error::UnsupportedDTypeForOp(vs.dtype(), "map1").bt()),+ C::F4(_) => Err(Error::UnsupportedDTypeForOp(vs.dtype(), "map1").bt()),+ C::F8E8M0(_) => Err(Error::UnsupportedDTypeForOp(vs.dtype(), "map1").bt()),+ }+ }+}++pub trait Map1Any {+ fn f<T: WithDType, W: Fn(Vec<T>) -> C>(&self, vs: &[T], layout: &Layout, wrap: W) -> Result<C>;++ fn map(&self, vs: &C, layout: &Layout) -> Result<C> {+ match vs {+ C::U8(vs) => Ok(self.f(vs, layout, C::U8)?),+ C::U32(vs) => Ok(self.f(vs, layout, C::U32)?),+ C::I16(vs) => Ok(self.f(vs, layout, C::I16)?),+ C::I32(vs) => Ok(self.f(vs, layout, C::I32)?),+ C::I64(vs) => Ok(self.f(vs, layout, C::I64)?),+ C::BF16(vs) => Ok(self.f(vs, layout, C::BF16)?),+ C::F16(vs) => Ok(self.f(vs, layout, C::F16)?),+ C::F32(vs) => Ok(self.f(vs, layout, C::F32)?),+ C::F64(vs) => Ok(self.f(vs, layout, C::F64)?),+ C::F8E4M3(vs) => Ok(self.f(vs, layout, C::F8E4M3)?),+ // Dummy types don't support Map1Any operations+ C::F6E2M3(_) => Err(Error::UnsupportedDTypeForOp(vs.dtype(), "map1any").bt()),+ C::F6E3M2(_) => Err(Error::UnsupportedDTypeForOp(vs.dtype(), "map1any").bt()),+ C::F4(_) => Err(Error::UnsupportedDTypeForOp(vs.dtype(), "map1any").bt()),+ C::F8E8M0(_) => Err(Error::UnsupportedDTypeForOp(vs.dtype(), "map1any").bt()),+ }+ }+}++pub trait Map2 {+ const OP: &'static str;+ fn f<T: WithDType>(&self, v1: &[T], l1: &Layout, v2: &[T], l2: &Layout) -> Result<Vec<T>>;++ fn map(&self, v1: &C, l1: &Layout, v2: &C, l2: &Layout) -> Result<C> {+ match (v1, v2) {+ (C::U8(v1), C::U8(v2)) => Ok(C::U8(self.f(v1, l1, v2, l2)?)),+ (C::U32(v1), C::U32(v2)) => Ok(C::U32(self.f(v1, l1, v2, l2)?)),+ (C::I16(v1), C::I16(v2)) => Ok(C::I16(self.f(v1, l1, v2, l2)?)),+ (C::I32(v1), C::I32(v2)) => Ok(C::I32(self.f(v1, l1, v2, l2)?)),+ (C::I64(v1), C::I64(v2)) => Ok(C::I64(self.f(v1, l1, v2, l2)?)),+ (C::BF16(v1), C::BF16(v2)) => Ok(C::BF16(self.f(v1, l1, v2, l2)?)),+ (C::F16(v1), C::F16(v2)) => Ok(C::F16(self.f(v1, l1, v2, l2)?)),+ (C::F32(v1), C::F32(v2)) => Ok(C::F32(self.f(v1, l1, v2, l2)?)),+ (C::F64(v1), C::F64(v2)) => Ok(C::F64(self.f(v1, l1, v2, l2)?)),+ (C::F8E4M3(v1), C::F8E4M3(v2)) => Ok(C::F8E4M3(self.f(v1, l1, v2, l2)?)),+ _ => Err(Error::DTypeMismatchBinaryOp {+ lhs: v1.dtype(),+ rhs: v2.dtype(),+ op: Self::OP,+ }+ .bt()),+ }+ }+}++pub trait Map2InPlace {+ const OP: &'static str;+ fn f<T: WithDType>(&self, v1: &mut [T], l1: &Layout, v2: &[T], l2: &Layout) -> Result<()>;++ fn map(&self, v1: &mut C, l1: &Layout, v2: &C, l2: &Layout) -> Result<()> {+ match (v1, v2) {+ (C::U8(v1), C::U8(v2)) => self.f(v1, l1, v2, l2)?,+ (C::U32(v1), C::U32(v2)) => self.f(v1, l1, v2, l2)?,+ (C::I16(v1), C::I16(v2)) => self.f(v1, l1, v2, l2)?,+ (C::I32(v1), C::I32(v2)) => self.f(v1, l1, v2, l2)?,+ (C::I64(v1), C::I64(v2)) => self.f(v1, l1, v2, l2)?,+ (C::BF16(v1), C::BF16(v2)) => self.f(v1, l1, v2, l2)?,+ (C::F16(v1), C::F16(v2)) => self.f(v1, l1, v2, l2)?,+ (C::F32(v1), C::F32(v2)) => self.f(v1, l1, v2, l2)?,+ (C::F64(v1), C::F64(v2)) => self.f(v1, l1, v2, l2)?,+ (C::F8E4M3(v1), C::F8E4M3(v2)) => self.f(v1, l1, v2, l2)?,+ (v1, v2) => Err(Error::DTypeMismatchBinaryOp {+ lhs: v1.dtype(),+ rhs: v2.dtype(),+ op: Self::OP,+ }+ .bt())?,+ };+ Ok(())+ }+}++pub trait Map2U8 {+ const OP: &'static str;+ fn f<T: WithDType>(&self, v1: &[T], l1: &Layout, v2: &[T], l2: &Layout) -> Result<Vec<u8>>;++ fn map(&self, v1: &C, l1: &Layout, v2: &C, l2: &Layout) -> Result<C> {+ match (v1, v2) {+ (C::U8(v1), C::U8(v2)) => Ok(C::U8(self.f(v1, l1, v2, l2)?)),+ (C::U32(v1), C::U32(v2)) => Ok(C::U8(self.f(v1, l1, v2, l2)?)),+ (C::I16(v1), C::I16(v2)) => Ok(C::U8(self.f(v1, l1, v2, l2)?)),+ (C::I32(v1), C::I32(v2)) => Ok(C::U8(self.f(v1, l1, v2, l2)?)),+ (C::I64(v1), C::I64(v2)) => Ok(C::U8(self.f(v1, l1, v2, l2)?)),+ (C::BF16(v1), C::BF16(v2)) => Ok(C::U8(self.f(v1, l1, v2, l2)?)),+ (C::F16(v1), C::F16(v2)) => Ok(C::U8(self.f(v1, l1, v2, l2)?)),+ (C::F32(v1), C::F32(v2)) => Ok(C::U8(self.f(v1, l1, v2, l2)?)),+ (C::F64(v1), C::F64(v2)) => Ok(C::U8(self.f(v1, l1, v2, l2)?)),+ (C::F8E4M3(v1), C::F8E4M3(v2)) => Ok(C::U8(self.f(v1, l1, v2, l2)?)),+ _ => Err(Error::DTypeMismatchBinaryOp {+ lhs: v1.dtype(),+ rhs: v2.dtype(),+ op: Self::OP,+ }+ .bt()),+ }+ }+}++pub fn binary_map<T: Copy, U: Copy, F: FnMut(T, T) -> U>(+ lhs_l: &Layout,+ rhs_l: &Layout,+ lhs: &[T],+ rhs: &[T],+ mut f: F,+) -> Vec<U> {+ let el_count = lhs_l.shape().elem_count();+ let mut result: Vec<U> = Vec::with_capacity(el_count);++ let nd_iter = NdIter::new([lhs_l, rhs_l]);+ let inner_size = nd_iter.inner_size;+ let [inner_ls, inner_rs] = nd_iter.inner_strides;++ for [lhs_off, rhs_off] in nd_iter {+ for i in 0..inner_size {+ result.push(f(lhs[lhs_off + i * inner_ls], rhs[rhs_off + i * inner_rs]));+ }+ }++ result+}++// Similar to binary_map but with vectorized variants.+pub fn binary_map_vec<+ T: Copy,+ F: FnMut(T, T) -> T,+ FV: FnMut(&[T], &[T], &mut [T]),+ FSV: FnMut(T, &[T], &mut [T]),+>(+ lhs_l: &Layout,+ rhs_l: &Layout,+ lhs: &[T],+ rhs: &[T],+ mut f: F,+ mut f_vec: FV,+ mut f_scalar_vec: FSV,+) -> Vec<T> {+ let el_count = lhs_l.shape().elem_count();+ let mut ys: Vec<T> = Vec::with_capacity(el_count);+ let ys_to_set = unsafe {+ let s = ys.spare_capacity_mut();+ std::mem::transmute::<&mut [std::mem::MaybeUninit<T>], &mut [T]>(s)+ };++ let nd_iter = NdIter::new([lhs_l, rhs_l]);+ let inner_size = nd_iter.inner_size;+ let [inner_ls, inner_rs] = nd_iter.inner_strides;++ let mut dst_off = 0usize;++ for [lhs_off, rhs_off] in nd_iter {+ match (inner_ls, inner_rs) {+ (1, 1) => f_vec(+ &lhs[lhs_off..lhs_off + inner_size],+ &rhs[rhs_off..rhs_off + inner_size],+ &mut ys_to_set[dst_off..dst_off + inner_size],+ ),+ (1, 0) => {+ let r = rhs[rhs_off];+ f_scalar_vec(+ r,+ &lhs[lhs_off..lhs_off + inner_size],+ &mut ys_to_set[dst_off..dst_off + inner_size],+ );+ }+ (0, 1) => {+ let l = lhs[lhs_off];+ for i in 0..inner_size {+ ys_to_set[dst_off + i] = f(l, rhs[rhs_off + i]);+ }+ }+ _ => {+ for i in 0..inner_size {+ ys_to_set[dst_off + i] =+ f(lhs[lhs_off + i * inner_ls], rhs[rhs_off + i * inner_rs]);+ }+ }+ }+ dst_off += inner_size;+ }++ // SAFETY: all el_count elements have been written in the dispatch loop above.+ unsafe { ys.set_len(el_count) };+ ys+}++pub fn unary_map<T: Copy, U: Copy, F: FnMut(T) -> U>(+ vs: &[T],+ layout: &Layout,+ mut f: F,+) -> Vec<U> {+ match layout.strided_blocks() {+ crate::StridedBlocks::SingleBlock { start_offset, len } => vs+ [start_offset..start_offset + len]+ .iter()+ .map(|&v| f(v))+ .collect(),+ crate::StridedBlocks::UniformBlocks {+ start_offset,+ block_len,+ count,+ src_stride,+ } => {+ let mut result = Vec::with_capacity(count * block_len);+ if block_len == 1 {+ for i in 0..count {+ let v = unsafe { vs.get_unchecked(start_offset + i * src_stride) };+ result.push(f(*v))+ }+ } else {+ for i in 0..count {+ let src_start = start_offset + i * src_stride;+ for offset in 0..block_len {+ let v = unsafe { vs.get_unchecked(src_start + offset) };+ result.push(f(*v))+ }+ }+ }+ result+ }+ crate::StridedBlocks::MultipleBlocks {+ block_start_index,+ block_len,+ } => {+ let mut result = Vec::with_capacity(layout.shape().elem_count());+ // Specialize the case where block_len is one to avoid the second loop.+ if block_len == 1 {+ for index in block_start_index {+ let v = unsafe { vs.get_unchecked(index) };+ result.push(f(*v))+ }+ } else {+ for index in block_start_index {+ for offset in 0..block_len {+ let v = unsafe { vs.get_unchecked(index + offset) };+ result.push(f(*v))+ }+ }+ }+ result+ }+ }+}++pub fn unary_map_vec<T: Copy, U: Copy, F: FnMut(T) -> U, FV: FnMut(&[T], &mut [U])>(+ vs: &[T],+ layout: &Layout,+ mut f: F,+ mut f_vec: FV,+) -> Vec<U> {+ match layout.strided_blocks() {+ crate::StridedBlocks::SingleBlock { start_offset, len } => {+ let mut ys: Vec<U> = Vec::with_capacity(len);+ let ys_to_set = ys.spare_capacity_mut();+ let ys_to_set = unsafe {+ std::mem::transmute::<&mut [std::mem::MaybeUninit<U>], &mut [U]>(ys_to_set)+ };+ f_vec(&vs[start_offset..start_offset + len], ys_to_set);+ // SAFETY: values are all set by f_vec.+ unsafe { ys.set_len(len) };+ ys+ }+ crate::StridedBlocks::UniformBlocks {+ start_offset,+ block_len,+ count,+ src_stride,+ } => {+ let el_count = count * block_len;+ if block_len == 1 {+ let mut result = Vec::with_capacity(count);+ for i in 0..count {+ let v = unsafe { vs.get_unchecked(start_offset + i * src_stride) };+ result.push(f(*v))+ }+ result+ } else {+ let mut ys: Vec<U> = Vec::with_capacity(el_count);+ let ys_to_set = ys.spare_capacity_mut();+ let ys_to_set = unsafe {+ std::mem::transmute::<&mut [std::mem::MaybeUninit<U>], &mut [U]>(ys_to_set)+ };+ let mut dst_index = 0;+ for i in 0..count {+ let src_start = start_offset + i * src_stride;+ f_vec(+ &vs[src_start..src_start + block_len],+ &mut ys_to_set[dst_index..dst_index + block_len],+ );+ dst_index += block_len;+ }+ // SAFETY: values are all set by f_vec.+ unsafe { ys.set_len(el_count) };+ ys+ }+ }+ crate::StridedBlocks::MultipleBlocks {+ block_start_index,+ block_len,+ } => {+ let el_count = layout.shape().elem_count();+ // Specialize the case where block_len is one to avoid the second loop.+ if block_len == 1 {+ let mut result = Vec::with_capacity(el_count);+ for index in block_start_index {+ let v = unsafe { vs.get_unchecked(index) };+ result.push(f(*v))+ }+ result+ } else {+ let mut ys: Vec<U> = Vec::with_capacity(el_count);+ let ys_to_set = ys.spare_capacity_mut();+ let ys_to_set = unsafe {+ std::mem::transmute::<&mut [std::mem::MaybeUninit<U>], &mut [U]>(ys_to_set)+ };+ let mut dst_index = 0;+ for src_index in block_start_index {+ let vs = &vs[src_index..src_index + block_len];+ let ys = &mut ys_to_set[dst_index..dst_index + block_len];+ f_vec(vs, ys);+ dst_index += block_len;+ }+ // SAFETY: values are all set by f_vec.+ unsafe { ys.set_len(el_count) };+ ys+ }+ }+ }+}++// Below this many logical elements the barrier-pool split costs more than it saves.+const PAR_ELEMWISE_MIN: usize = 64 * 1024;+// Per-unit span: big enough to amortize dispatch, small enough to balance.+const PAR_ELEMWISE_CHUNK: usize = 16 * 1024;++pub fn unary_map_vec_par<+ T: Copy + Send + Sync,+ U: Copy + Send + Sync,+ F: Fn(T) -> U + Sync,+ FV: Fn(&[T], &mut [U]) + Sync,+>(+ vs: &[T],+ layout: &Layout,+ f: F,+ f_vec: FV,+) -> Vec<U> {+ if let crate::StridedBlocks::SingleBlock { start_offset, len } = layout.strided_blocks() {+ if len >= PAR_ELEMWISE_MIN {+ let src = &vs[start_offset..start_offset + len];+ let mut ys: Vec<U> = Vec::with_capacity(len);+ let ys_ptr = ys.as_mut_ptr() as usize;+ let n_units = len.div_ceil(PAR_ELEMWISE_CHUNK);+ crate::utils::barrier_pool().execute_chunked(n_units, |range| {+ let ys_ptr = ys_ptr as *mut U;+ for unit in range {+ let lo = unit * PAR_ELEMWISE_CHUNK;+ let hi = len.min(lo + PAR_ELEMWISE_CHUNK);+ let dst = unsafe { std::slice::from_raw_parts_mut(ys_ptr.add(lo), hi - lo) };+ f_vec(&src[lo..hi], dst);+ }+ });+ // SAFETY: every element is written by exactly one unit.+ unsafe { ys.set_len(len) };+ return ys;+ }+ }+ unary_map_vec(vs, layout, f, f_vec)+}++#[allow(clippy::too_many_arguments)]+pub fn binary_map_vec_par<+ T: Copy + Send + Sync,+ F: Fn(T, T) -> T + Sync,+ FV: Fn(&[T], &[T], &mut [T]) + Sync,+ FSV: Fn(T, &[T], &mut [T]) + Sync,+>(+ lhs_l: &Layout,+ rhs_l: &Layout,+ lhs: &[T],+ rhs: &[T],+ f: F,+ f_vec: FV,+ f_scalar_vec: FSV,+) -> Vec<T> {+ let el_count = lhs_l.shape().elem_count();+ if el_count < PAR_ELEMWISE_MIN {+ return binary_map_vec(lhs_l, rhs_l, lhs, rhs, f, f_vec, f_scalar_vec);+ }++ let nd_iter = NdIter::new([lhs_l, rhs_l]);+ let inner_size = nd_iter.inner_size;+ let [lhs_stride, rhs_stride] = nd_iter.inner_strides;+ debug_assert!(inner_size > 0);++ let mut ys: Vec<T> = Vec::with_capacity(el_count);+ let ys_ptr = ys.as_mut_ptr() as usize;+ let n_units = el_count.div_ceil(PAR_ELEMWISE_CHUNK);+ crate::utils::barrier_pool().execute_chunked(n_units, |range| {+ let ys_ptr = ys_ptr as *mut T;+ for unit in range {+ let mut dst_offset = unit * PAR_ELEMWISE_CHUNK;+ let hi = el_count.min(dst_offset + PAR_ELEMWISE_CHUNK);+ let mut inner_offset = dst_offset % inner_size;+ let mut nd_iter = NdIter::new([lhs_l, rhs_l]);+ let mut offsets = nd_iter+ .nth(dst_offset / inner_size)+ .expect("logical output offset must map to an NdIter block");++ while dst_offset < hi {+ let len = (inner_size - inner_offset).min(hi - dst_offset);+ let lhs_offset = offsets[0] + inner_offset * lhs_stride;+ let rhs_offset = offsets[1] + inner_offset * rhs_stride;+ // SAFETY: work units own disjoint output ranges, all written before set_len.+ let dst = unsafe { std::slice::from_raw_parts_mut(ys_ptr.add(dst_offset), len) };++ match (lhs_stride, rhs_stride) {+ (1, 1) => f_vec(+ &lhs[lhs_offset..lhs_offset + len],+ &rhs[rhs_offset..rhs_offset + len],+ dst,+ ),+ (1, 0) => {+ f_scalar_vec(rhs[rhs_offset], &lhs[lhs_offset..lhs_offset + len], dst)+ }+ _ => {+ for (i, dst) in dst.iter_mut().enumerate() {+ *dst = f(+ lhs[lhs_offset + i * lhs_stride],+ rhs[rhs_offset + i * rhs_stride],+ );+ }+ }+ }++ dst_offset += len;+ inner_offset = 0;+ if dst_offset < hi {+ offsets = nd_iter+ .next()+ .expect("logical output range must stay within NdIter blocks");+ }+ }+ }+ });+ // SAFETY: every element is written by exactly one work unit.+ unsafe { ys.set_len(el_count) };+ ys+}++#[cfg(test)]+mod tests {+ use super::*;+ use crate::Shape;+ use std::sync::atomic::{AtomicUsize, Ordering};++ #[test]+ fn binary_map_vec_par_chunks_non_contiguous_broadcasts() {+ const ROWS: usize = 2;+ const COLS: usize = 2 * PAR_ELEMWISE_CHUNK;+ const LHS_ROW_STRIDE: usize = COLS + 1;++ let mut lhs = vec![i64::MAX; ROWS * LHS_ROW_STRIDE];+ for row in 0..ROWS {+ for col in 0..COLS {+ lhs[row * LHS_ROW_STRIDE + col] = (row * COLS + col) as i64;+ }+ }+ let rhs = [11i64, 29];+ let shape = Shape::from((ROWS, COLS));+ let lhs_l = Layout::new(shape.clone(), vec![LHS_ROW_STRIDE, 1], 0);+ let rhs_l = Layout::new(shape, vec![1, 0], 0);++ let calls = AtomicUsize::new(0);+ let elements = AtomicUsize::new(0);+ let result = binary_map_vec_par(+ &lhs_l,+ &rhs_l,+ &lhs,+ &rhs,+ |lhs, rhs| lhs - rhs,+ |_, _, _| panic!("broadcast layout must not use the two-vector kernel"),+ |scalar, xs, ys| {+ assert!(xs.len() <= PAR_ELEMWISE_CHUNK);+ calls.fetch_add(1, Ordering::Relaxed);+ elements.fetch_add(xs.len(), Ordering::Relaxed);+ for (&x, y) in xs.iter().zip(ys) {+ *y = x - scalar;+ }+ },+ );++ let expected: Vec<_> = (0..ROWS)+ .flat_map(|row| (0..COLS).map(move |col| (row * COLS + col) as i64 - rhs[row]))+ .collect();+ assert_eq!(result, expected);+ assert_eq!(calls.load(Ordering::Relaxed), 4);+ assert_eq!(elements.load(Ordering::Relaxed), ROWS * COLS);+ }++ #[test]+ fn binary_map_vec_par_handles_two_strided_inputs() {+ const ROWS: usize = 2;+ const COLS: usize = 2 * PAR_ELEMWISE_CHUNK;+ const LHS_ROW_STRIDE: usize = COLS + 1;+ const RHS_ROW_STRIDE: usize = COLS + 2;++ let mut lhs = vec![0i64; ROWS * LHS_ROW_STRIDE];+ let mut rhs = vec![0i64; ROWS * RHS_ROW_STRIDE];+ for row in 0..ROWS {+ for col in 0..COLS {+ lhs[row * LHS_ROW_STRIDE + col] = (row * COLS + col) as i64;+ rhs[row * RHS_ROW_STRIDE + col] = (2 * row * COLS + col) as i64;+ }+ }+ let shape = Shape::from((COLS, ROWS));+ let lhs_l = Layout::new(shape.clone(), vec![1, LHS_ROW_STRIDE], 0);+ let rhs_l = Layout::new(shape, vec![1, RHS_ROW_STRIDE], 0);++ let result = binary_map_vec_par(+ &lhs_l,+ &rhs_l,+ &lhs,+ &rhs,+ |lhs, rhs| lhs - rhs,+ |_, _, _| panic!("strided layout must not use the two-vector kernel"),+ |_, _, _| panic!("strided layout must not use the scalar-vector kernel"),+ );++ let lhs = &lhs;+ let rhs = &rhs;+ let expected: Vec<_> = (0..COLS)+ .flat_map(|col| {+ (0..ROWS).map(move |row| {+ lhs[row * LHS_ROW_STRIDE + col] - rhs[row * RHS_ROW_STRIDE + col]+ })+ })+ .collect();+ assert_eq!(result, expected);+ }+}diff --git a/vendor/candle-core/src/cuda_backend/cudnn.rs b/vendor/candle-core/src/cuda_backend/cudnn.rsnew file mode 100644index 0000000..d7d8770--- /dev/null+++ b/vendor/candle-core/src/cuda_backend/cudnn.rs@@ -0,0 +1,225 @@+use crate::WithDType;+use cudarc;+use cudarc::cudnn::safe::{ConvForward, Cudnn};+use cudarc::driver::{CudaSlice, CudaView, DeviceRepr, ValidAsZeroBits};+use std::cell::RefCell;+use std::collections::HashMap;+use std::sync::Arc;++// The cudnn handles are stored per thread here rather than on the CudaDevice as they are neither+// send nor sync.+thread_local! {+ static CUDNN: RefCell<HashMap<crate::cuda_backend::DeviceId, Arc<Cudnn>>> = HashMap::new().into();+}++impl From<cudarc::cudnn::CudnnError> for crate::Error {+ fn from(err: cudarc::cudnn::CudnnError) -> Self {+ crate::Error::wrap(err)+ }+}++impl From<cudarc::driver::DriverError> for crate::Error {+ fn from(err: cudarc::driver::DriverError) -> Self {+ crate::Error::wrap(err)+ }+}++pub(crate) fn launch_conv2d<+ T: DeviceRepr + WithDType + ValidAsZeroBits + cudarc::cudnn::CudnnDataType,+ Y: cudarc::cudnn::CudnnDataType,+>(+ src: &CudaView<T>,+ src_l: &crate::Layout,+ filter: &CudaView<T>,+ dst: &mut CudaSlice<T>,+ params: &crate::conv::ParamsConv2D,+ dev: &crate::cuda_backend::CudaDevice,+) -> crate::Result<()> {+ use crate::conv::CudnnFwdAlgo as CandleAlgo;+ use cudarc::cudnn::sys::cudnnConvolutionFwdAlgo_t as A;++ let device_id = dev.id();+ let cudnn = CUDNN.with(|cudnn| {+ if let Some(cudnn) = cudnn.borrow().get(&device_id) {+ return Ok(cudnn.clone());+ }+ let c = Cudnn::new(dev.cuda_stream());+ if let Ok(c) = &c {+ cudnn.borrow_mut().insert(device_id, c.clone());+ }+ c+ })?;+ let conv = cudnn.create_conv2d::<Y>(+ /* pad */ [params.padding as i32, params.padding as i32],+ /* stride */ [params.stride as i32, params.stride as i32],+ /* dilation */ [params.dilation as i32, params.dilation as i32],+ cudarc::cudnn::sys::cudnnConvolutionMode_t::CUDNN_CROSS_CORRELATION,+ )?;+ let x_shape = [+ params.b_size as i32,+ params.c_in as i32,+ params.i_h as i32,+ params.i_w as i32,+ ];+ // Note that `src` already starts at the proper offset.+ let x = if src_l.is_contiguous() {+ cudnn.create_4d_tensor::<T>(+ cudarc::cudnn::sys::cudnnTensorFormat_t::CUDNN_TENSOR_NCHW,+ x_shape,+ )?+ } else {+ let s = src_l.stride();+ cudnn.create_4d_tensor_ex::<T>(+ x_shape,+ [s[0] as i32, s[1] as i32, s[2] as i32, s[3] as i32],+ )?+ };+ let w = cudnn.create_4d_filter::<T>(+ cudarc::cudnn::sys::cudnnTensorFormat_t::CUDNN_TENSOR_NCHW,+ [+ params.c_out as i32,+ params.c_in as i32,+ params.k_h as i32,+ params.k_w as i32,+ ],+ )?;+ let (w_out, h_out) = (params.out_w() as i32, params.out_h() as i32);+ let y = cudnn.create_4d_tensor::<T>(+ cudarc::cudnn::sys::cudnnTensorFormat_t::CUDNN_TENSOR_NCHW,+ [params.b_size as i32, params.c_out as i32, h_out, w_out],+ )?;+ let conv2d = ConvForward {+ conv: &conv,+ x: &x,+ w: &w,+ y: &y,+ };+ let alg = match params.cudnn_fwd_algo {+ None => conv2d.pick_algorithm()?,+ Some(CandleAlgo::ImplicitGemm) => A::CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_GEMM,+ Some(CandleAlgo::ImplicitPrecompGemm) => {+ A::CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_PRECOMP_GEMM+ }+ Some(CandleAlgo::Gemm) => A::CUDNN_CONVOLUTION_FWD_ALGO_GEMM,+ Some(CandleAlgo::Direct) => A::CUDNN_CONVOLUTION_FWD_ALGO_DIRECT,+ Some(CandleAlgo::Fft) => A::CUDNN_CONVOLUTION_FWD_ALGO_FFT,+ Some(CandleAlgo::FftTiling) => A::CUDNN_CONVOLUTION_FWD_ALGO_FFT_TILING,+ Some(CandleAlgo::Winograd) => A::CUDNN_CONVOLUTION_FWD_ALGO_WINOGRAD,+ Some(CandleAlgo::WinogradNonFused) => A::CUDNN_CONVOLUTION_FWD_ALGO_WINOGRAD_NONFUSED,+ Some(CandleAlgo::Count) => A::CUDNN_CONVOLUTION_FWD_ALGO_COUNT,+ };+ let workspace_size = conv2d.get_workspace_size(alg)?;+ let mut workspace = dev.cuda_stream().alloc_zeros::<u8>(workspace_size)?;+ unsafe {+ conv2d.launch::<CudaSlice<u8>, _, _, _>(+ alg,+ Some(&mut workspace),+ (T::one(), T::zero()),+ src,+ filter,+ dst,+ )?;+ }+ Ok(())+}++pub(crate) fn launch_conv1d<+ T: DeviceRepr + WithDType + ValidAsZeroBits + cudarc::cudnn::CudnnDataType,+ Y: cudarc::cudnn::CudnnDataType,+>(+ src: &CudaView<T>,+ src_l: &crate::Layout,+ filter: &CudaView<T>,+ dst: &mut CudaSlice<T>,+ params: &crate::conv::ParamsConv1D,+ dev: &crate::cuda_backend::CudaDevice,+) -> crate::Result<()> {+ use crate::conv::CudnnFwdAlgo as CandleAlgo;+ use cudarc::cudnn::sys::cudnnConvolutionFwdAlgo_t as A;++ let device_id = dev.id();+ let cudnn = CUDNN.with(|cudnn| {+ if let Some(cudnn) = cudnn.borrow().get(&device_id) {+ return Ok(cudnn.clone());+ }+ let c = Cudnn::new(dev.cuda_stream());+ if let Ok(c) = &c {+ cudnn.borrow_mut().insert(device_id, c.clone());+ }+ c+ })?;+ let conv = cudnn.create_conv2d::<Y>(+ /* pad */ [params.padding as i32, 0],+ /* stride */ [params.stride as i32, 1],+ /* dilation */ [params.dilation as i32, 1],+ cudarc::cudnn::sys::cudnnConvolutionMode_t::CUDNN_CROSS_CORRELATION,+ )?;+ // https://docs.nvidia.com/deeplearning/cudnn/backend/latest/api/cudnn-ops-library.html#cudnnsettensornddescriptor+ // > Tensors are restricted to having at least 4 dimensions, and at most CUDNN_DIM_MAX+ // > dimensions (defined in cudnn.h). When working with lower dimensional data, it is+ // > recommended that the user create a 4D tensor, and set the size along unused dimensions+ // > to 1.+ let x_shape = [+ params.b_size as i32,+ params.c_in as i32,+ params.l_in as i32,+ 1,+ ];+ // Note that `src` already starts at the proper offset.+ let x = if src_l.is_contiguous() {+ cudnn.create_4d_tensor::<T>(+ cudarc::cudnn::sys::cudnnTensorFormat_t::CUDNN_TENSOR_NCHW,+ x_shape,+ )?+ } else {+ let s = src_l.stride();+ cudnn.create_4d_tensor_ex::<T>(x_shape, [s[0] as i32, s[1] as i32, s[2] as i32, 1i32])?+ };+ let w = cudnn.create_4d_filter::<T>(+ cudarc::cudnn::sys::cudnnTensorFormat_t::CUDNN_TENSOR_NCHW,+ [+ params.c_out as i32,+ params.c_in as i32,+ params.k_size as i32,+ 1,+ ],+ )?;+ let l_out = params.l_out() as i32;+ let y = cudnn.create_4d_tensor::<T>(+ cudarc::cudnn::sys::cudnnTensorFormat_t::CUDNN_TENSOR_NCHW,+ [params.b_size as i32, params.c_out as i32, l_out, 1],+ )?;+ let conv1d = ConvForward {+ conv: &conv,+ x: &x,+ w: &w,+ y: &y,+ };+ let alg = match params.cudnn_fwd_algo {+ None => conv1d.pick_algorithm()?,+ Some(CandleAlgo::ImplicitGemm) => A::CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_GEMM,+ Some(CandleAlgo::ImplicitPrecompGemm) => {+ A::CUDNN_CONVOLUTION_FWD_ALGO_IMPLICIT_PRECOMP_GEMM+ }+ Some(CandleAlgo::Gemm) => A::CUDNN_CONVOLUTION_FWD_ALGO_GEMM,+ Some(CandleAlgo::Direct) => A::CUDNN_CONVOLUTION_FWD_ALGO_DIRECT,+ Some(CandleAlgo::Fft) => A::CUDNN_CONVOLUTION_FWD_ALGO_FFT,+ Some(CandleAlgo::FftTiling) => A::CUDNN_CONVOLUTION_FWD_ALGO_FFT_TILING,+ Some(CandleAlgo::Winograd) => A::CUDNN_CONVOLUTION_FWD_ALGO_WINOGRAD,+ Some(CandleAlgo::WinogradNonFused) => A::CUDNN_CONVOLUTION_FWD_ALGO_WINOGRAD_NONFUSED,+ Some(CandleAlgo::Count) => A::CUDNN_CONVOLUTION_FWD_ALGO_COUNT,+ };+ let workspace_size = conv1d.get_workspace_size(alg)?;+ let mut workspace = dev.cuda_stream().alloc_zeros::<u8>(workspace_size)?;+ unsafe {+ conv1d.launch::<CudaSlice<u8>, _, _, _>(+ alg,+ Some(&mut workspace),+ (T::one(), T::zero()),+ src,+ filter,+ dst,+ )?;+ }+ Ok(())+}diff --git a/vendor/candle-core/src/cuda_backend/cutile.rs b/vendor/candle-core/src/cuda_backend/cutile.rsnew file mode 100644index 0000000..cc4d977--- /dev/null+++ b/vendor/candle-core/src/cuda_backend/cutile.rs@@ -0,0 +1,268 @@+//! cuTile kernel authoring and CUDA stream interop.+//!+//! ```no_run+//! use candle_core::cutile;+//!+//! #[cutile::module]+//! mod kernels {+//! use candle_core::cutile;+//! use cutile::core::*;+//! use cutile::cutile_compiler;+//!+//! #[cutile::entry]+//! unsafe fn noop() {}+//! }+//! ```++use super::cudarc::driver::{CudaSlice, DevicePtr, DevicePtrMut, DeviceRepr, SyncOnDrop};+use super::{CudaDType, CudaDevice, CudaStorage, WrapErr};+use crate::{Error, Layout, Result};+use ::core::ffi::{c_int, c_void};+use std::collections::HashMap;+use std::path::{Path, PathBuf};+use std::process::Command;+use std::sync::{Arc, Mutex, OnceLock};++pub use ::cutile::*;++/// A cuTile launch context backed by Candle's current CUDA stream.+pub struct CutileContext {+ candle_stream: Arc<super::cudarc::driver::CudaStream>,+ cutile_stream: Arc<::cutile::cuda_core::Stream>,+}++impl CutileContext {+ /// Borrows Candle's CUDA context and stream without transferring ownership.+ pub fn new(device: &CudaDevice) -> Result<Self> {+ let candle_stream = device.cuda_stream();+ let candle_context = candle_stream.context();+ candle_context.bind_to_thread().w()?;+ check_device_support(candle_context.ordinal())?;+ let cutile_device = unsafe {+ ::cutile::cuda_core::Device::borrow_with_owner(+ candle_context.cu_ctx() as *mut c_void,+ candle_context.cu_device() as c_int,+ candle_context.ordinal(),+ candle_context.clone(),+ )+ };+ let cutile_stream = unsafe {+ ::cutile::cuda_core::Stream::borrow_with_owner(+ candle_stream.cu_stream() as *mut c_void,+ &cutile_device,+ candle_stream.clone(),+ )+ };+ Ok(Self {+ candle_stream,+ cutile_stream,+ })+ }++ /// The borrowed cuTile stream used by `compile_on` and `async_on`.+ pub fn stream(&self) -> &Arc<::cutile::cuda_core::Stream> {+ &self.cutile_stream+ }++ /// Borrows a typed Candle allocation for a cuTile kernel read.+ pub fn read<'a, T: DeviceRepr>(+ &'a self,+ slice: &'a CudaSlice<T>,+ offset: usize,+ ) -> Result<CutileRead<'a, T>> {+ if offset > slice.len() {+ return Err(Error::msg(format!(+ "cuTile read offset {offset} exceeds allocation length {}",+ slice.len()+ )));+ }+ let (pointer, guard) = slice.device_ptr(&self.candle_stream);+ let byte_offset = offset+ .checked_mul(std::mem::size_of::<T>())+ .ok_or_else(|| Error::msg("cuTile read offset overflow"))?;+ let pointer = pointer+ .checked_add(byte_offset as u64)+ .ok_or_else(|| Error::msg("cuTile read pointer overflow"))?;+ Ok(CutileRead {+ pointer: unsafe {+ ::cutile::cuda_async::device_buffer::DevicePointer::from_cu_deviceptr(pointer)+ },+ _guard: guard,+ })+ }++ /// Borrows a typed Candle allocation for a cuTile kernel write.+ pub fn write<'a, T: DeviceRepr>(+ &'a self,+ slice: &'a mut CudaSlice<T>,+ offset: usize,+ ) -> Result<CutileWrite<'a, T>> {+ if offset > slice.len() {+ return Err(Error::msg(format!(+ "cuTile write offset {offset} exceeds allocation length {}",+ slice.len()+ )));+ }+ let (pointer, guard) = slice.device_ptr_mut(&self.candle_stream);+ let byte_offset = offset+ .checked_mul(std::mem::size_of::<T>())+ .ok_or_else(|| Error::msg("cuTile write offset overflow"))?;+ let pointer = pointer+ .checked_add(byte_offset as u64)+ .ok_or_else(|| Error::msg("cuTile write pointer overflow"))?;+ Ok(CutileWrite {+ pointer: unsafe {+ ::cutile::cuda_async::device_buffer::DevicePointer::from_cu_deviceptr(pointer)+ },+ _guard: guard,+ })+ }++ /// Borrows a typed Candle storage and applies its layout offset.+ pub fn read_storage<'a, T: CudaDType + DeviceRepr + 'a>(+ &'a self,+ storage: &'a CudaStorage,+ layout: &Layout,+ ) -> Result<CutileRead<'a, T>> {+ self.read(storage.as_cuda_slice::<T>()?, layout.start_offset())+ }++ /// Mutably borrows a typed Candle storage and applies its layout offset.+ pub fn write_storage<'a, T: CudaDType + DeviceRepr + 'a>(+ &'a self,+ storage: &'a mut CudaStorage,+ layout: &Layout,+ ) -> Result<CutileWrite<'a, T>> {+ self.write(storage.as_cuda_slice_mut::<T>()?, layout.start_offset())+ }+}++/// Fails when the installed `tileiras` cannot target the device, if it can be probed.+fn check_device_support(ordinal: usize) -> Result<()> {+ use ::cutile::cutile_compiler::cuda_tile_runtime_utils::{get_gpu_name, tileiras_binary};+ let tileiras = tileiras_binary();+ let Some(supported) = tileiras_gpu_names(&tileiras) else {+ return Ok(());+ };+ ensure_gpu_supported(ordinal, &get_gpu_name(ordinal), &tileiras, &supported)+}++fn ensure_gpu_supported(+ ordinal: usize,+ gpu_name: &str,+ tileiras: &Path,+ supported: &[String],+) -> Result<()> {+ if supported.iter().any(|name| name == gpu_name) {+ return Ok(());+ }+ Err(Error::msg(format!(+ "cuTile cannot target CUDA device {ordinal}: it is {gpu_name}, but {} supports only {}. \+ Use a supported GPU or a CUDA toolkit whose tileiras supports {gpu_name}.",+ tileiras.display(),+ supported.join(", "),+ )))+}++type GpuNames = Option<Arc<[String]>>;++/// Architectures accepted by `tileiras --gpu-name`, probed once per binary.+fn tileiras_gpu_names(tileiras: &Path) -> GpuNames {+ static CACHE: OnceLock<Mutex<HashMap<PathBuf, GpuNames>>> = OnceLock::new();+ let mut cache = CACHE+ .get_or_init(Default::default)+ .lock()+ .unwrap_or_else(|poisoned| poisoned.into_inner());+ if let Some(cached) = cache.get(tileiras) {+ return cached.clone();+ }+ let names = probe_tileiras_gpu_names(tileiras);+ cache.insert(tileiras.to_path_buf(), names.clone());+ names+}++fn probe_tileiras_gpu_names(tileiras: &Path) -> GpuNames {+ let output = Command::new(tileiras).arg("--help").output().ok()?;+ let mut help = String::from_utf8_lossy(&output.stdout).into_owned();+ help.push_str(&String::from_utf8_lossy(&output.stderr));+ let names = parse_tileiras_gpu_names(&help);+ (!names.is_empty()).then(|| names.into())+}++/// Extracts the `=sm_XY` choices listed under `--gpu-name` in `tileiras --help`.+fn parse_tileiras_gpu_names(help: &str) -> Vec<String> {+ let mut names = Vec::new();+ let mut in_gpu_name = false;+ for line in help.lines() {+ let line = line.trim_start();+ if line.starts_with("--gpu-name") {+ in_gpu_name = true;+ } else if in_gpu_name {+ match line+ .strip_prefix('=')+ .and_then(|rest| rest.split_whitespace().next())+ {+ Some(name) => names.push(name.to_string()),+ None => in_gpu_name = false,+ }+ }+ }+ names+}++/// A typed cuTile pointer that records a Candle read after it is dropped.+#[must_use = "keep the pointer guard alive until the cuTile launch is enqueued"]+pub struct CutileRead<'a, T> {+ pointer: ::cutile::cuda_async::device_buffer::DevicePointer<T>,+ _guard: SyncOnDrop<'a>,+}++impl<T> CutileRead<'_, T> {+ /// Returns the pointer accepted by raw-pointer cuTile entry functions.+ pub fn device_pointer(&self) -> ::cutile::cuda_async::device_buffer::DevicePointer<T> {+ unsafe {+ ::cutile::cuda_async::device_buffer::DevicePointer::from_cu_deviceptr(+ self.pointer.cu_deviceptr(),+ )+ }+ }+}++/// A typed cuTile pointer that records a Candle write after it is dropped.+#[must_use = "keep the pointer guard alive until the cuTile launch is enqueued"]+pub struct CutileWrite<'a, T> {+ pointer: ::cutile::cuda_async::device_buffer::DevicePointer<T>,+ _guard: SyncOnDrop<'a>,+}++impl<T> CutileWrite<'_, T> {+ /// Returns the pointer accepted by raw-pointer cuTile entry functions.+ pub fn device_pointer(&self) -> ::cutile::cuda_async::device_buffer::DevicePointer<T> {+ unsafe {+ ::cutile::cuda_async::device_buffer::DevicePointer::from_cu_deviceptr(+ self.pointer.cu_deviceptr(),+ )+ }+ }+}++/// Converts a cuTile error or panic into a Candle error.+pub fn kernel<T, E: std::fmt::Display>(+ operation: &str,+ f: impl FnOnce() -> std::result::Result<T, E>,+) -> Result<T> {+ match std::panic::catch_unwind(std::panic::AssertUnwindSafe(f)) {+ Ok(Ok(value)) => Ok(value),+ Ok(Err(error)) => Err(Error::msg(format!("cuTile {operation}: {error}"))),+ Err(payload) => {+ let message = payload+ .downcast_ref::<String>()+ .map(String::as_str)+ .or_else(|| payload.downcast_ref::<&'static str>().copied())+ .unwrap_or("non-string panic");+ Err(Error::msg(format!(+ "cuTile {operation} panicked: {message}"+ )))+ }+ }+}diff --git a/vendor/candle-core/src/cuda_backend/device.rs b/vendor/candle-core/src/cuda_backend/device.rsnew file mode 100644index 0000000..ab809a0--- /dev/null+++ b/vendor/candle-core/src/cuda_backend/device.rs@@ -0,0 +1,815 @@+use crate::backend::{BackendDevice, BackendStorage};+use crate::{CpuStorage, CpuStorageRef, DType, Layout, Result, Shape};+pub use candle_kernels as kernels;+pub use cudarc;+use cudarc::driver::CudaFunction;+use float8::F8E4M3;+use half::{bf16, f16};+use std::any::{Any, TypeId};+use std::cell::{Cell, RefCell};+use std::collections::HashMap;+use std::sync::{Arc, Mutex, RwLock};++use super::{CudaError, CudaStorage, CudaStorageSlice, WrapErr};++/// Unique identifier for cuda devices.+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]+pub struct DeviceId(usize);++impl DeviceId {+ fn new() -> Self {+ // https://users.rust-lang.org/t/idiomatic-rust-way-to-generate-unique-id/33805+ use std::sync::atomic;+ static COUNTER: atomic::AtomicUsize = atomic::AtomicUsize::new(1);+ Self(COUNTER.fetch_add(1, atomic::Ordering::Relaxed))+ }+}++struct CudaRng(cudarc::curand::CudaRng);+unsafe impl Send for CudaRng {}++const CUDA_GRAPH_HTOD_CACHE_MAX_BYTES: usize = 4096;++type CudaGraphHtodCacheKey = (DeviceId, TypeId, Vec<u8>);+type CudaGraphHtodCache = HashMap<CudaGraphHtodCacheKey, Box<dyn Any>>;++thread_local! {+ static CUDA_GRAPH_HTOD_CACHE: RefCell<CudaGraphHtodCache> =+ RefCell::new(HashMap::new());+ static CUDA_GRAPH_HTOD_CACHE_DEPTH: Cell<usize> = const { Cell::new(0) };+}++#[must_use]+pub struct CudaGraphHtodCacheGuard;++impl Drop for CudaGraphHtodCacheGuard {+ fn drop(&mut self) {+ CUDA_GRAPH_HTOD_CACHE_DEPTH.with(|depth| {+ debug_assert!(depth.get() > 0);+ depth.set(depth.get().saturating_sub(1));+ });+ }+}++pub struct ModuleStore {+ mdls: [Option<Arc<cudarc::driver::CudaModule>>; kernels::ALL_IDS.len()],+}++#[derive(Clone)]+pub struct CudaDevice {+ id: DeviceId,+ context: Arc<cudarc::driver::CudaContext>,+ modules: Arc<std::sync::RwLock<ModuleStore>>,+ custom_modules: Arc<std::sync::RwLock<HashMap<String, Arc<cudarc::driver::CudaModule>>>>,+ stream: Arc<cudarc::driver::CudaStream>,+ pub(crate) blas: Arc<cudarc::cublas::CudaBlas>,+ curand: Arc<Mutex<CudaRng>>,+ seed_value: Arc<RwLock<u64>>,+}++impl std::fmt::Debug for CudaDevice {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ write!(f, "CudaDevice({:?})", self.id)+ }+}++impl CudaDevice {+ #[allow(clippy::missing_safety_doc)]+ pub unsafe fn alloc<T: cudarc::driver::DeviceRepr>(+ &self,+ len: usize,+ ) -> Result<cudarc::driver::CudaSlice<T>> {+ self.stream.alloc::<T>(len).w()+ }++ pub fn alloc_zeros<T: cudarc::driver::DeviceRepr + cudarc::driver::ValidAsZeroBits>(+ &self,+ len: usize,+ ) -> Result<cudarc::driver::CudaSlice<T>> {+ self.stream.alloc_zeros::<T>(len).w()+ }++ pub fn enable_cuda_graph_htod_cache(&self) -> CudaGraphHtodCacheGuard {+ CUDA_GRAPH_HTOD_CACHE_DEPTH.with(|depth| depth.set(depth.get() + 1));+ CudaGraphHtodCacheGuard+ }++ pub fn memcpy_htod<+ T: cudarc::driver::DeviceRepr + 'static,+ Src: cudarc::driver::HostSlice<T> + ?Sized,+ Dst: cudarc::driver::DevicePtrMut<T>,+ >(+ &self,+ src: &Src,+ dst: &mut Dst,+ ) -> Result<()> {+ if cuda_graph_htod_cache_enabled() && !src.is_empty() {+ return self.memcpy_htod_capture_cached(src, dst);+ }+ self.stream.memcpy_htod(src, dst).w()+ }++ pub fn clone_dtoh<T: cudarc::driver::DeviceRepr, Src: cudarc::driver::DevicePtr<T>>(+ &self,+ src: &Src,+ ) -> Result<Vec<T>> {+ self.check_capture_copy("clone_dtoh")?;+ self.stream.clone_dtoh(src).w()+ }++ pub fn memcpy_dtod<+ T,+ Src: cudarc::driver::DevicePtr<T>,+ Dst: cudarc::driver::DevicePtrMut<T>,+ >(+ &self,+ src: &Src,+ dst: &mut Dst,+ ) -> Result<()> {+ self.stream.memcpy_dtod(src, dst).w()+ }++ pub fn memcpy_dtoh<+ T: cudarc::driver::DeviceRepr,+ Src: cudarc::driver::DevicePtr<T>,+ Dst: cudarc::driver::HostSlice<T>,+ >(+ &self,+ src: &Src,+ dst: &mut Dst,+ ) -> Result<()> {+ self.check_capture_copy("memcpy_dtoh")?;+ self.stream.memcpy_dtoh(src, dst).w()+ }++ pub fn clone_htod<+ T: cudarc::driver::DeviceRepr + 'static,+ Src: cudarc::driver::HostSlice<T> + ?Sized,+ >(+ &self,+ src: &Src,+ ) -> Result<cudarc::driver::CudaSlice<T>> {+ if cuda_graph_htod_cache_enabled() && !src.is_empty() {+ return self.clone_htod_capture_cached(src);+ }+ self.stream.clone_htod(src).w()+ }++ fn check_capture_copy(&self, op: &str) -> Result<()> {+ if cuda_graph_htod_cache_enabled() {+ let is_capturing = self.cuda_graph_capture_active();+ if is_capturing {+ crate::bail!(+ "{op} during CUDA graph capture: host/device copies are not \+ permitted while capturing (use param cache for kernel parameters)"+ );+ }+ }+ Ok(())+ }++ fn memcpy_htod_capture_cached<+ T: cudarc::driver::DeviceRepr + 'static,+ Src: cudarc::driver::HostSlice<T> + ?Sized,+ Dst: cudarc::driver::DevicePtrMut<T>,+ >(+ &self,+ src: &Src,+ dst: &mut Dst,+ ) -> Result<()> {+ assert!(dst.len() >= src.len());+ if let Some(cached) = self.cached_htod_slice(src)? {+ return self.stream.memcpy_dtod(&cached, dst).w();+ }+ self.stream.memcpy_htod(src, dst).w()+ }++ fn clone_htod_capture_cached<+ T: cudarc::driver::DeviceRepr + 'static,+ Src: cudarc::driver::HostSlice<T> + ?Sized,+ >(+ &self,+ src: &Src,+ ) -> Result<cudarc::driver::CudaSlice<T>> {+ if let Some(cached) = self.cached_htod_slice(src)? {+ return Ok(cached);+ }+ self.stream.clone_htod(src).w()+ }++ fn cached_htod_slice<+ T: cudarc::driver::DeviceRepr + 'static,+ Src: cudarc::driver::HostSlice<T> + ?Sized,+ >(+ &self,+ src: &Src,+ ) -> Result<Option<cudarc::driver::CudaSlice<T>>> {+ let (src_slice, _sync) = unsafe { src.stream_synced_slice(&self.stream) };+ let byte_len = std::mem::size_of_val(src_slice);+ if byte_len > CUDA_GRAPH_HTOD_CACHE_MAX_BYTES {+ if self.cuda_graph_capture_active() {+ crate::bail!("CUDA graph capture cannot upload uncached host data");+ }+ return Ok(None);+ }++ let bytes =+ unsafe { std::slice::from_raw_parts(src_slice.as_ptr().cast::<u8>(), byte_len) }+ .to_vec();+ let key = (self.id, TypeId::of::<T>(), bytes);+ if let Some(cached) = CUDA_GRAPH_HTOD_CACHE.with(|cache| {+ cache.borrow().get(&key).and_then(|cached| {+ cached+ .downcast_ref::<cudarc::driver::CudaSlice<T>>()+ .cloned()+ })+ }) {+ return Ok(Some(cached));+ }+ if self.cuda_graph_capture_active() {+ crate::bail!("CUDA graph capture missing cached host data");+ }++ let cached = self.stream.clone_htod(src).w()?;+ CUDA_GRAPH_HTOD_CACHE.with(|cache| {+ cache.borrow_mut().insert(key, Box::new(cached.clone()));+ });+ Ok(Some(cached))+ }++ fn cuda_graph_capture_active(&self) -> bool {+ matches!(+ self.stream.capture_status(),+ Ok(status) if status != cudarc::driver::sys::CUstreamCaptureStatus::CU_STREAM_CAPTURE_STATUS_NONE+ )+ }+}++pub(crate) fn cuda_graph_htod_cache_enabled() -> bool {+ CUDA_GRAPH_HTOD_CACHE_DEPTH.with(|depth| depth.get() > 0)+}++pub struct CudaFunc {+ func: CudaFunction,+ stream: Arc<cudarc::driver::CudaStream>,+}++impl std::ops::Deref for CudaFunc {+ type Target = CudaFunction;++ fn deref(&self) -> &Self::Target {+ &self.func+ }+}++impl CudaFunc {+ pub fn into_cuda_function(self) -> CudaFunction {+ self.func+ }+}++#[macro_export]+macro_rules! builder_arg {+ ($b:ident, $($arg:expr),*) => {+ $(+ let __arg = $arg;+ $b.arg(&__arg);+ )*+ };+}++impl CudaFunc {+ pub fn builder(&self) -> cudarc::driver::LaunchArgs<'_> {+ self.stream.launch_builder(&self.func)+ }+}++impl CudaDevice {+ pub fn cuda_stream(&self) -> Arc<cudarc::driver::CudaStream> {+ self.stream.clone()+ }++ /// When turned on, all cuda tensors **created after calling this function** will+ /// not track uses via cuda events.+ ///+ /// # Safety+ ///+ /// It is up to the user to ensure proper synchronization between multiple streams:+ /// - Ensure that no tensor is freed before a use on another stream is finished.+ /// - Ensure that a tensor is not used on another stream before allocation on the+ /// allocating stream finishes.+ /// - Ensure that a tensor is not written two concurrently by multiple streams.+ pub unsafe fn disable_event_tracking(&self) {+ self.context.disable_event_tracking()+ }++ pub fn is_event_tracking(&self) -> bool {+ self.context.is_event_tracking()+ }++ pub fn id(&self) -> DeviceId {+ self.id+ }++ pub fn get_or_load_custom_func(+ &self,+ fn_name: &str,+ module_name: &str,+ ptx: &str,+ ) -> Result<CudaFunc> {+ let ms = self.custom_modules.read().unwrap();+ if let Some(mdl) = ms.get(module_name).as_ref() {+ let func = mdl.load_function(fn_name).w()?;+ return Ok(CudaFunc {+ func,+ stream: self.stream.clone(),+ });+ }+ drop(ms);+ let mut ms = self.custom_modules.write().unwrap();+ let cuda_module = self.context.load_module(ptx.into()).w()?;+ ms.insert(module_name.to_string(), cuda_module.clone());+ let func = cuda_module.load_function(fn_name).w()?;+ Ok(CudaFunc {+ func,+ stream: self.stream.clone(),+ })+ }++ pub fn get_or_load_func(&self, fn_name: &str, mdl: &kernels::Module) -> Result<CudaFunc> {+ let ms = self.modules.read().unwrap();+ if let Some(mdl) = ms.mdls[mdl.index()].as_ref() {+ let func = mdl.load_function(fn_name).w()?;+ return Ok(CudaFunc {+ func,+ stream: self.stream.clone(),+ });+ }+ drop(ms);+ let mut ms = self.modules.write().unwrap();+ let cuda_module = self.context.load_module(mdl.ptx().into()).w()?;+ ms.mdls[mdl.index()] = Some(cuda_module.clone());+ let func = cuda_module.load_function(fn_name).w()?;+ Ok(CudaFunc {+ func,+ stream: self.stream.clone(),+ })+ }++ pub fn cublas_handle(&self) -> Arc<cudarc::cublas::CudaBlas> {+ self.blas.clone()+ }+}++impl CudaDevice {+ pub fn new_with_stream(ordinal: usize) -> Result<Self> {+ let context = cudarc::driver::CudaContext::new(ordinal).w()?;+ let stream = context.new_stream().w()?;+ Self::from_context_and_stream(context, stream)+ }++ fn from_context_and_stream(+ context: Arc<cudarc::driver::CudaContext>,+ stream: Arc<cudarc::driver::CudaStream>,+ ) -> Result<Self> {+ let blas = cudarc::cublas::CudaBlas::new(stream.clone()).w()?;+ let curand = cudarc::curand::CudaRng::new(299792458, stream.clone()).w()?;+ let module_store = ModuleStore {+ mdls: [const { None }; kernels::ALL_IDS.len()],+ };+ Ok(Self {+ id: DeviceId::new(),+ context,+ stream,+ blas: Arc::new(blas),+ curand: Arc::new(Mutex::new(CudaRng(curand))),+ modules: Arc::new(std::sync::RwLock::new(module_store)),+ custom_modules: Arc::new(std::sync::RwLock::new(HashMap::new())),+ seed_value: Arc::new(RwLock::new(299792458)),+ })+ }+}++impl BackendDevice for CudaDevice {+ type Storage = CudaStorage;++ fn new(ordinal: usize) -> Result<Self> {+ let context = cudarc::driver::CudaContext::new(ordinal).w()?;+ let stream = context.per_thread_stream();+ Self::from_context_and_stream(context, stream)+ }++ fn set_seed(&self, seed: u64) -> Result<()> {+ // We do not call set_seed but instead create a new curand object. This ensures that the+ // state will be identical and the same random numbers will be generated.+ let mut curand = self.curand.lock().unwrap();+ curand.0 = cudarc::curand::CudaRng::new(seed, self.stream.clone()).w()?;+ *self.seed_value.write().unwrap() = seed;+ Ok(())+ }++ fn get_current_seed(&self) -> Result<u64> {+ Ok(*self.seed_value.read().unwrap())+ }++ fn location(&self) -> crate::DeviceLocation {+ crate::DeviceLocation::Cuda {+ gpu_id: self.context.ordinal(),+ }+ }++ fn same_device(&self, rhs: &Self) -> bool {+ self.id == rhs.id+ }++ fn zeros_impl(&self, shape: &Shape, dtype: DType) -> Result<CudaStorage> {+ let elem_count = shape.elem_count();+ let slice = match dtype {+ DType::U8 => {+ let data = self.alloc_zeros::<u8>(elem_count)?;+ CudaStorageSlice::U8(data)+ }+ DType::U32 => {+ let data = self.alloc_zeros::<u32>(elem_count)?;+ CudaStorageSlice::U32(data)+ }+ DType::I16 => {+ let data = self.alloc_zeros::<i16>(elem_count)?;+ CudaStorageSlice::I16(data)+ }+ DType::I32 => {+ let data = self.alloc_zeros::<i32>(elem_count)?;+ CudaStorageSlice::I32(data)+ }+ DType::I64 => {+ let data = self.alloc_zeros::<i64>(elem_count)?;+ CudaStorageSlice::I64(data)+ }+ DType::BF16 => {+ let data = self.alloc_zeros::<bf16>(elem_count)?;+ CudaStorageSlice::BF16(data)+ }+ DType::F16 => {+ let data = self.alloc_zeros::<f16>(elem_count)?;+ CudaStorageSlice::F16(data)+ }+ DType::F32 => {+ let data = self.alloc_zeros::<f32>(elem_count)?;+ CudaStorageSlice::F32(data)+ }+ DType::F64 => {+ let data = self.alloc_zeros::<f64>(elem_count)?;+ CudaStorageSlice::F64(data)+ }+ DType::F8E4M3 => {+ let data = self.alloc_zeros::<F8E4M3>(elem_count)?;+ CudaStorageSlice::F8E4M3(data)+ }+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ return Err(+ CudaError::InternalError("Dummy types not supported in CUDA backend").into(),+ )+ }+ };+ Ok(CudaStorage {+ slice,+ device: self.clone(),+ })+ }++ fn rand_uniform(&self, shape: &Shape, dtype: DType, lo: f64, up: f64) -> Result<CudaStorage> {+ let elem_count = shape.elem_count();+ let curand = self.curand.lock().unwrap();+ let slice = match dtype {+ // TODO: Add support for F16 and BF16 though this is likely to require some upstream+ // cudarc changes.+ DType::U8+ | DType::U32+ | DType::I16+ | DType::I32+ | DType::I64+ | DType::F16+ | DType::BF16 => Err(CudaError::UnsupportedDtype {+ dtype,+ op: "rand_uniform",+ })+ .w()?,+ DType::F32 => {+ let mut data = unsafe { self.alloc::<f32>(elem_count)? };+ curand.0.fill_with_uniform(&mut data).w()?;+ CudaStorageSlice::F32(data)+ }+ DType::F64 => {+ let mut data = unsafe { self.alloc::<f64>(elem_count)? };+ curand.0.fill_with_uniform(&mut data).w()?;+ CudaStorageSlice::F64(data)+ }+ DType::F8E4M3 | DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ Err(CudaError::UnsupportedDtype {+ dtype,+ op: "rand_uniform",+ })+ .w()?+ }+ };+ let slice = if lo == 0. && up == 1.0 {+ slice+ } else {+ use super::utils::Map1;+ let layout = Layout::contiguous(shape);+ super::Affine(up - lo, lo).map(&slice, self, &layout)?+ };+ Ok(CudaStorage {+ slice,+ device: self.clone(),+ })+ }++ fn rand_normal(&self, shape: &Shape, dtype: DType, mean: f64, std: f64) -> Result<CudaStorage> {+ // TODO: Add support for F16 and BF16 though this is likely to require some upstream+ // cudarc changes.+ let elem_count = shape.elem_count();+ let curand = self.curand.lock().unwrap();+ // curand can only generate an odd number of values.+ // https://github.com/huggingface/candle/issues/734+ let elem_count_round = if elem_count % 2 == 1 {+ elem_count + 1+ } else {+ elem_count+ };+ let slice = match dtype {+ DType::U8+ | DType::U32+ | DType::I16+ | DType::I32+ | DType::I64+ | DType::F16+ | DType::BF16 => Err(CudaError::UnsupportedDtype {+ dtype,+ op: "rand_normal",+ })+ .w()?,+ DType::F32 => {+ let mut data = unsafe { self.alloc::<f32>(elem_count_round)? };+ curand+ .0+ .fill_with_normal(&mut data, mean as f32, std as f32)+ .w()?;+ CudaStorageSlice::F32(data)+ }+ DType::F64 => {+ let mut data = unsafe { self.alloc::<f64>(elem_count_round)? };+ curand.0.fill_with_normal(&mut data, mean, std).w()?;+ CudaStorageSlice::F64(data)+ }+ DType::F8E4M3 | DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ Err(CudaError::UnsupportedDtype {+ dtype,+ op: "rand_normal",+ })+ .w()?+ }+ };+ Ok(CudaStorage {+ slice,+ device: self.clone(),+ })+ }++ unsafe fn alloc_uninit(&self, shape: &Shape, dtype: DType) -> Result<Self::Storage> {+ let elem_count = shape.elem_count();+ let slice = match dtype {+ DType::U8 => {+ let data = self.alloc::<u8>(elem_count)?;+ CudaStorageSlice::U8(data)+ }+ DType::U32 => {+ let data = self.alloc::<u32>(elem_count)?;+ CudaStorageSlice::U32(data)+ }+ DType::I16 => {+ let data = self.alloc::<i16>(elem_count)?;+ CudaStorageSlice::I16(data)+ }+ DType::I32 => {+ let data = self.alloc::<i32>(elem_count)?;+ CudaStorageSlice::I32(data)+ }+ DType::I64 => {+ let data = self.alloc::<i64>(elem_count)?;+ CudaStorageSlice::I64(data)+ }+ DType::BF16 => {+ let data = self.alloc::<bf16>(elem_count)?;+ CudaStorageSlice::BF16(data)+ }+ DType::F16 => {+ let data = self.alloc::<f16>(elem_count)?;+ CudaStorageSlice::F16(data)+ }+ DType::F32 => {+ let data = self.alloc::<f32>(elem_count)?;+ CudaStorageSlice::F32(data)+ }+ DType::F64 => {+ let data = self.alloc::<f64>(elem_count)?;+ CudaStorageSlice::F64(data)+ }+ DType::F8E4M3 => {+ let data = self.alloc::<F8E4M3>(elem_count)?;+ CudaStorageSlice::F8E4M3(data)+ }+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ return Err(+ CudaError::InternalError("Dummy types not supported in CUDA backend").into(),+ )+ }+ };+ Ok(CudaStorage {+ slice,+ device: self.clone(),+ })+ }++ fn storage_from_slice<T: crate::WithDType>(&self, s: &[T]) -> Result<Self::Storage> {+ let slice = match T::cpu_storage_ref(s) {+ CpuStorageRef::U8(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::U8(data)+ }+ CpuStorageRef::U32(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::U32(data)+ }+ CpuStorageRef::I16(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::I16(data)+ }+ CpuStorageRef::I32(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::I32(data)+ }+ CpuStorageRef::I64(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::I64(data)+ }+ CpuStorageRef::BF16(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::BF16(data)+ }+ CpuStorageRef::F16(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::F16(data)+ }+ CpuStorageRef::F32(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::F32(data)+ }+ CpuStorageRef::F64(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::F64(data)+ }+ CpuStorageRef::F8E4M3(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::F8E4M3(data)+ }+ CpuStorageRef::F4(_)+ | CpuStorageRef::F6E2M3(_)+ | CpuStorageRef::F6E3M2(_)+ | CpuStorageRef::F8E8M0(_) => {+ return Err(CudaError::UnsupportedDtype {+ dtype: T::DTYPE,+ op: "storage_from_slice",+ }+ .into());+ }+ };+ Ok(CudaStorage {+ slice,+ device: self.clone(),+ })+ }++ fn storage_from_cpu_storage(&self, storage: &CpuStorage) -> Result<CudaStorage> {+ let slice = match storage {+ CpuStorage::U8(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::U8(data)+ }+ CpuStorage::U32(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::U32(data)+ }+ CpuStorage::I16(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::I16(data)+ }+ CpuStorage::I32(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::I32(data)+ }+ CpuStorage::I64(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::I64(data)+ }+ CpuStorage::BF16(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::BF16(data)+ }+ CpuStorage::F16(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::F16(data)+ }+ CpuStorage::F32(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::F32(data)+ }+ CpuStorage::F64(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::F64(data)+ }+ CpuStorage::F8E4M3(storage) => {+ let data = self.clone_htod(storage)?;+ CudaStorageSlice::F8E4M3(data)+ }+ CpuStorage::F4(_)+ | CpuStorage::F6E2M3(_)+ | CpuStorage::F6E3M2(_)+ | CpuStorage::F8E8M0(_) => {+ return Err(CudaError::UnsupportedDtype {+ dtype: storage.dtype(),+ op: "storage_from_cpu_storage",+ }+ .into());+ }+ };+ Ok(CudaStorage {+ slice,+ device: self.clone(),+ })+ }++ fn storage_from_cpu_storage_owned(&self, storage: CpuStorage) -> Result<CudaStorage> {+ let slice = match storage {+ CpuStorage::U8(storage) => {+ let data = self.clone_htod(&storage)?;+ CudaStorageSlice::U8(data)+ }+ CpuStorage::U32(storage) => {+ let data = self.clone_htod(&storage)?;+ CudaStorageSlice::U32(data)+ }+ CpuStorage::I16(storage) => {+ let data = self.clone_htod(&storage)?;+ CudaStorageSlice::I16(data)+ }+ CpuStorage::I32(storage) => {+ let data = self.clone_htod(&storage)?;+ CudaStorageSlice::I32(data)+ }+ CpuStorage::I64(storage) => {+ let data = self.clone_htod(&storage)?;+ CudaStorageSlice::I64(data)+ }+ CpuStorage::BF16(storage) => {+ let data = self.clone_htod(&storage)?;+ CudaStorageSlice::BF16(data)+ }+ CpuStorage::F16(storage) => {+ let data = self.clone_htod(&storage)?;+ CudaStorageSlice::F16(data)+ }+ CpuStorage::F32(storage) => {+ let data = self.clone_htod(&storage)?;+ CudaStorageSlice::F32(data)+ }+ CpuStorage::F64(storage) => {+ let data = self.clone_htod(&storage)?;+ CudaStorageSlice::F64(data)+ }+ CpuStorage::F8E4M3(storage) => {+ let data = self.clone_htod(&storage)?;+ CudaStorageSlice::F8E4M3(data)+ }+ CpuStorage::F4(_)+ | CpuStorage::F6E2M3(_)+ | CpuStorage::F6E3M2(_)+ | CpuStorage::F8E8M0(_) => {+ return Err(CudaError::UnsupportedDtype {+ dtype: storage.dtype(),+ op: "storage_from_cpu_storage_owned",+ }+ .into());+ }+ };+ Ok(CudaStorage {+ slice,+ device: self.clone(),+ })+ }++ fn synchronize(&self) -> Result<()> {+ self.stream.synchronize().map_err(crate::Error::wrap)?;+ Ok(())+ }+}diff --git a/vendor/candle-core/src/cuda_backend/error.rs b/vendor/candle-core/src/cuda_backend/error.rsnew file mode 100644index 0000000..bd6f8ac--- /dev/null+++ b/vendor/candle-core/src/cuda_backend/error.rs@@ -0,0 +1,62 @@+use crate::{DType, Layout};++/// cudarc related errors+#[derive(thiserror::Error, Debug)]+pub enum CudaError {+ #[error(transparent)]+ Cuda(#[from] cudarc::driver::DriverError),++ #[error(transparent)]+ Compiler(#[from] cudarc::nvrtc::CompileError),++ #[error(transparent)]+ Cublas(#[from] cudarc::cublas::result::CublasError),++ #[error(transparent)]+ Curand(#[from] cudarc::curand::result::CurandError),++ #[error("missing kernel '{module_name}'")]+ MissingKernel { module_name: String },++ #[error("unsupported dtype {dtype:?} for {op}")]+ UnsupportedDtype { dtype: DType, op: &'static str },++ #[error("internal error '{0}'")]+ InternalError(&'static str),++ #[error("matmul is only supported for contiguous tensors lstride: {lhs_stride:?} rstride: {rhs_stride:?} mnk: {mnk:?}")]+ MatMulNonContiguous {+ lhs_stride: Layout,+ rhs_stride: Layout,+ mnk: (usize, usize, usize),+ },++ #[error("{msg}, expected: {expected:?}, got: {got:?}")]+ UnexpectedDType {+ msg: &'static str,+ expected: DType,+ got: DType,+ },++ #[error("{cuda} when loading {module_name}")]+ Load {+ cuda: cudarc::driver::DriverError,+ module_name: String,+ },+}++impl From<CudaError> for crate::Error {+ fn from(val: CudaError) -> Self {+ crate::Error::Cuda(Box::new(val)).bt()+ }+}++pub trait WrapErr<O> {+ fn w(self) -> std::result::Result<O, crate::Error>;+}++impl<O, E: Into<CudaError>> WrapErr<O> for std::result::Result<O, E> {+ fn w(self) -> std::result::Result<O, crate::Error> {+ self.map_err(|e| crate::Error::Cuda(Box::new(e.into())).bt())+ }+}diff --git a/vendor/candle-core/src/cuda_backend/mod.rs b/vendor/candle-core/src/cuda_backend/mod.rsnew file mode 100644index 0000000..93f315c--- /dev/null+++ b/vendor/candle-core/src/cuda_backend/mod.rs@@ -0,0 +1,2752 @@+//! Implementation of Backend traits for CUDA device+//!+use crate::backend::{BackendDevice, BackendStorage};+use crate::op::{BinaryOpT, CmpOp, ReduceOp, UnaryOpT};+use crate::{builder_arg as barg, CpuStorage, DType, Layout, Result, WithDType};+pub use candle_kernels as kernels;+pub use cudarc;+use cudarc::cublas::{Gemm, GemmConfig, StridedBatchedConfig};+use cudarc::driver::{+ CudaSlice, DevicePtr, DeviceRepr, LaunchConfig, PushKernelArg, ValidAsZeroBits,+};+use half::{bf16, f16};+use std::collections::HashMap;+use std::sync::{Arc, Mutex, OnceLock};++#[cfg(feature = "cudnn")]+pub mod cudnn;+#[cfg(feature = "cutile")]+pub mod cutile;+mod device;+mod error;+mod utils;+pub use device::{CudaDevice, DeviceId};+pub use error::{CudaError, WrapErr};+pub use utils::{Map1, Map1Any, Map2, Map2Any, Map2InPlace, Map3, S};++type ParamCache = HashMap<(DeviceId, Vec<usize>), Arc<CudaSlice<usize>>>;++static CUDA_PARAM_CACHE: OnceLock<Mutex<ParamCache>> = OnceLock::new();++pub enum SlicePtrOrNull<T> {+ Ptr(CudaSlice<T>),+ Cached(Arc<CudaSlice<T>>),+ Null,+}++impl<T: DeviceRepr> SlicePtrOrNull<T> {+ pub fn builder_arg<'a, 'b: 'a>(&'b self, builder: &mut cudarc::driver::LaunchArgs<'a>) {+ match self {+ SlicePtrOrNull::Ptr(slice) => builder.arg(slice),+ SlicePtrOrNull::Cached(slice) => builder.arg(slice.as_ref()),+ SlicePtrOrNull::Null => builder.arg(&0usize),+ };+ }+}++impl crate::scalar::Scalar {+ pub fn builder_arg<'a, 'b: 'a>(&'b self, builder: &mut cudarc::driver::LaunchArgs<'a>) {+ use crate::scalar::Scalar;+ match self {+ Scalar::U8(v) => builder.arg(v),+ Scalar::U32(v) => builder.arg(v),+ Scalar::I16(v) => builder.arg(v),+ Scalar::I32(v) => builder.arg(v),+ Scalar::I64(v) => builder.arg(v),+ Scalar::F32(v) => builder.arg(v),+ Scalar::F64(v) => builder.arg(v),+ Scalar::F16(v) => builder.arg(v),+ Scalar::BF16(v) => builder.arg(v),+ Scalar::F8E4M3(v) => builder.arg(v),+ };+ }+}++impl SlicePtrOrNull<usize> {+ pub fn params_from_vec(dev: &CudaDevice, params: Vec<usize>) -> Result<Self> {+ if device::cuda_graph_htod_cache_enabled() {+ let key = (dev.id(), params.clone());+ let cache = CUDA_PARAM_CACHE.get_or_init(|| Mutex::new(HashMap::new()));+ if let Some(slice) = cache.lock().unwrap().get(&key).cloned() {+ return Ok(SlicePtrOrNull::Cached(slice));+ }++ let is_capturing = dev.cuda_stream().capture_status()+ == Ok(cudarc::driver::sys::CUstreamCaptureStatus::CU_STREAM_CAPTURE_STATUS_ACTIVE);+ if is_capturing {+ crate::bail!(+ "h2d param cache miss during CUDA graph capture: \+ warmup did not populate all required parameter vectors"+ );+ }++ let slice = Arc::new(dev.cuda_stream().clone_htod(¶ms).w()?);+ let mut cache = cache.lock().unwrap();+ let slice = cache.entry(key).or_insert_with(|| slice).clone();+ return Ok(SlicePtrOrNull::Cached(slice));+ }++ Ok(SlicePtrOrNull::Ptr(dev.clone_htod(¶ms)?))+ }++ pub fn params_from_layout(dev: &CudaDevice, l: &Layout) -> Result<Self> {+ if l.is_contiguous() {+ Ok(SlicePtrOrNull::Null)+ } else {+ Self::params_from_vec(dev, [l.dims(), l.stride()].concat())+ }+ }+}++#[derive(Debug)]+pub enum CudaStorageSlice {+ U8(CudaSlice<u8>),+ U32(CudaSlice<u32>),+ I16(CudaSlice<i16>),+ I32(CudaSlice<i32>),+ I64(CudaSlice<i64>),+ BF16(CudaSlice<bf16>),+ F16(CudaSlice<f16>),+ F32(CudaSlice<f32>),+ F64(CudaSlice<f64>),+ F8E4M3(CudaSlice<float8::F8E4M3>),+ // Dummy types that store raw bytes+ F6E2M3(CudaSlice<u8>),+ F6E3M2(CudaSlice<u8>),+ F4(CudaSlice<u8>),+ F8E8M0(CudaSlice<u8>),+}++struct Clone;+impl Map1 for Clone {+ fn f<T: DeviceRepr>(+ &self,+ s: &CudaSlice<T>,+ _: &CudaDevice,+ _: &Layout,+ ) -> Result<CudaSlice<T>> {+ s.try_clone().w()+ }+}++pub fn kernel_name<T: WithDType>(root: &str) -> String {+ let dtype = T::DTYPE.as_str();+ format!("{root}_{dtype}")+}++struct Affine(f64, f64);+impl Map1 for Affine {+ fn f<T: DeviceRepr + WithDType>(+ &self,+ src: &CudaSlice<T>,+ dev: &CudaDevice,+ layout: &Layout,+ ) -> Result<CudaSlice<T>> {+ let shape = layout.shape();+ let dims = shape.dims();+ let el = shape.elem_count();+ let cfg = LaunchConfig::for_num_elems(el as u32);+ let ds = SlicePtrOrNull::params_from_layout(dev, layout)?;+ let src = &src.slice(layout.start_offset()..);+ let func = dev.get_or_load_func(&kernel_name::<T>("affine"), &kernels::AFFINE)?;+ // SAFETY: Set later by running the kernel.+ let out = unsafe { dev.alloc::<T>(el)? };+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ builder.arg(src);+ builder.arg(&out);+ barg!(builder, T::from_f64(self.0));+ barg!(builder, T::from_f64(self.1));+ // SAFETY: ffi.+ unsafe { builder.launch(cfg).w() }?;+ Ok(out)+ }+}++struct Elu(f64);+impl Map1 for Elu {+ fn f<T: DeviceRepr + WithDType>(+ &self,+ src: &CudaSlice<T>,+ dev: &CudaDevice,+ layout: &Layout,+ ) -> Result<CudaSlice<T>> {+ let shape = layout.shape();+ let dims = shape.dims();+ let el = shape.elem_count();+ let cfg = LaunchConfig::for_num_elems(el as u32);+ let ds = SlicePtrOrNull::params_from_layout(dev, layout)?;+ let src = &src.slice(layout.start_offset()..);+ let func = dev.get_or_load_func(&kernel_name::<T>("uelu"), &kernels::UNARY)?;+ // SAFETY: Set later by running the kernel.+ let out = unsafe { dev.alloc::<T>(el)? };+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ barg!(builder, T::from_f64(self.0));+ builder.arg(src);+ builder.arg(&out);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(out)+ }+}++#[allow(unused)]+struct Im2Col1D {+ l_k: usize,+ stride: usize,+ dilation: usize,+ padding: usize,+}++impl Im2Col1D {+ #[allow(unused)]+ fn l_out(&self, l: usize) -> usize {+ (l + 2 * self.padding - self.dilation * (self.l_k - 1) - 1) / self.stride + 1+ }+}++impl Map1 for Im2Col1D {+ fn f<T: DeviceRepr + WithDType>(+ &self,+ src: &CudaSlice<T>,+ dev: &CudaDevice,+ layout: &Layout,+ ) -> Result<CudaSlice<T>> {+ let shape = layout.shape();+ let dims = shape.dims();+ let l_out = self.l_out(dims[2]);+ let threads = dims[0] * l_out * dims[1];+ let cfg = LaunchConfig::for_num_elems(threads as u32);+ let ds = SlicePtrOrNull::params_from_vec(dev, [dims, layout.stride()].concat())?;+ let src = &src.slice(layout.start_offset()..);+ let func = dev.get_or_load_func(&kernel_name::<T>("im2col1d"), &kernels::CONV)?;+ // SAFETY: Set later by running the kernel.+ let dst = unsafe { dev.alloc::<T>(threads * self.l_k)? };+ let mut builder = func.builder();+ barg!(builder, threads);+ barg!(builder, l_out);+ barg!(builder, self.l_k);+ barg!(builder, self.stride);+ barg!(builder, self.padding);+ barg!(builder, self.dilation);+ ds.builder_arg(&mut builder);+ builder.arg(src);+ builder.arg(&dst);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(dst)+ }+}++#[allow(unused)]+struct Im2Col {+ h_k: usize,+ w_k: usize,+ stride: usize,+ dilation: usize,+ padding: usize,+}++impl Im2Col {+ #[allow(unused)]+ fn hw_out(&self, h: usize, w: usize) -> (usize, usize) {+ let h_out = (h + 2 * self.padding - self.dilation * (self.h_k - 1) - 1) / self.stride + 1;+ let w_out = (w + 2 * self.padding - self.dilation * (self.w_k - 1) - 1) / self.stride + 1;+ (h_out, w_out)+ }+}++impl Map1 for Im2Col {+ fn f<T: DeviceRepr + WithDType>(+ &self,+ src: &CudaSlice<T>,+ dev: &CudaDevice,+ layout: &Layout,+ ) -> Result<CudaSlice<T>> {+ let shape = layout.shape();+ let dims = shape.dims();+ let (h_out, w_out) = self.hw_out(dims[2], dims[3]);+ let dst_el = dims[0] * h_out * w_out * dims[1] * self.h_k * self.w_k;+ let cfg = LaunchConfig::for_num_elems(dst_el as u32);+ let ds = SlicePtrOrNull::params_from_vec(dev, [dims, layout.stride()].concat())?;+ let src = &src.slice(layout.start_offset()..);+ let func = dev.get_or_load_func(&kernel_name::<T>("im2col"), &kernels::CONV)?;+ // SAFETY: Set later by running the kernel.+ let dst = unsafe { dev.alloc::<T>(dst_el)? };+ let mut builder = func.builder();+ barg!(builder, dst_el);+ barg!(builder, h_out);+ barg!(builder, w_out);+ barg!(builder, self.h_k);+ barg!(builder, self.w_k);+ barg!(builder, self.stride);+ barg!(builder, self.padding);+ barg!(builder, self.dilation);+ ds.builder_arg(&mut builder);+ builder.arg(src);+ builder.arg(&dst);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(dst)+ }+}++struct Powf(f64);+impl Map1 for Powf {+ fn f<T: DeviceRepr + WithDType>(+ &self,+ src: &CudaSlice<T>,+ dev: &CudaDevice,+ layout: &Layout,+ ) -> Result<CudaSlice<T>> {+ let shape = layout.shape();+ let dims = shape.dims();+ let el = shape.elem_count();+ let cfg = LaunchConfig::for_num_elems(el as u32);+ let ds = SlicePtrOrNull::params_from_layout(dev, layout)?;+ let src = &src.slice(layout.start_offset()..);+ let func = dev.get_or_load_func(&kernel_name::<T>("upowf"), &kernels::UNARY)?;+ // SAFETY: Set later by running the kernel.+ let out = unsafe { dev.alloc::<T>(el)? };+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ barg!(builder, T::from_f64(self.0));+ builder.arg(src);+ builder.arg(&out);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(out)+ }+}++struct FastReduce<'a>(&'a [usize], ReduceOp);+impl Map1Any for FastReduce<'_> {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits, W: Fn(CudaSlice<T>) -> S>(+ &self,+ src: &CudaSlice<T>,+ dev: &CudaDevice,+ layout: &Layout,+ wrap: W,+ ) -> Result<S> {+ let src_stride = layout.stride();+ let src_dims = layout.shape().dims();+ let src_el: usize = src_dims.iter().product();+ // Source dims and strides with the sum dims at the end.+ let mut dims = vec![];+ let mut stride = vec![];+ let mut dst_el: usize = 1;+ for (dim_idx, &d) in src_dims.iter().enumerate() {+ if !self.0.contains(&dim_idx) {+ dst_el *= d;+ dims.push(d);+ stride.push(src_stride[dim_idx]);+ }+ }+ for &dim_idx in self.0.iter() {+ dims.push(src_dims[dim_idx]);+ stride.push(src_stride[dim_idx]);+ }+ let el_to_sum_per_block = src_el / dst_el;+ let ds =+ SlicePtrOrNull::params_from_vec(dev, [dims.as_slice(), stride.as_slice()].concat())?;+ let src = &src.slice(layout.start_offset()..);+ let (name, check_empty, return_index) = match self.1 {+ ReduceOp::Sum => ("fast_sum", false, false),+ ReduceOp::Min => ("fast_min", true, false),+ ReduceOp::Max => ("fast_max", true, false),+ ReduceOp::ArgMin => ("fast_argmin", true, true),+ ReduceOp::ArgMax => ("fast_argmax", true, true),+ };+ // For small reductions (e.g. MoE topk sum with el_to_sum=8), use a one-thread-per-output+ // kernel to avoid launching millions of blocks each doing almost nothing.+ let use_small_reduce = el_to_sum_per_block <= 32 && !return_index && name == "fast_sum";+ let (cfg, kernel_name_str) = if use_small_reduce {+ let threads = 256usize;+ let blocks = (dst_el + threads - 1) / threads;+ (+ LaunchConfig {+ grid_dim: (blocks as u32, 1, 1),+ block_dim: (threads as u32, 1, 1),+ shared_mem_bytes: 0,+ },+ "fast_sum_small",+ )+ } else {+ let block_dim = usize::min(1024, el_to_sum_per_block).next_power_of_two();+ (+ LaunchConfig {+ grid_dim: (dst_el as u32, 1, 1),+ block_dim: (block_dim as u32, 1, 1),+ shared_mem_bytes: 0,+ },+ name,+ )+ };+ if check_empty && layout.shape().elem_count() == 0 {+ Err(crate::Error::EmptyTensor { op: "reduce" }.bt())?+ }+ let func = dev.get_or_load_func(&kernel_name::<T>(kernel_name_str), &kernels::REDUCE)?;+ let kernel_numel = if use_small_reduce { dst_el } else { src_el };+ if return_index {+ // SAFETY: filled in by the follow up kernel.+ let out = unsafe { dev.alloc::<u32>(dst_el)? };+ let mut builder = func.builder();+ barg!(builder, kernel_numel);+ barg!(builder, el_to_sum_per_block);+ barg!(builder, src_dims.len());+ ds.builder_arg(&mut builder);+ builder.arg(src);+ builder.arg(&out);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(S::U32(out))+ } else {+ // SAFETY: filled in by the follow up kernel.+ let out = unsafe { dev.alloc::<T>(dst_el)? };+ let mut builder = func.builder();+ barg!(builder, kernel_numel);+ barg!(builder, el_to_sum_per_block);+ barg!(builder, src_dims.len());+ ds.builder_arg(&mut builder);+ builder.arg(src);+ builder.arg(&out);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(wrap(out))+ }+ }+}++impl<U: UnaryOpT> Map1 for U {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ src: &CudaSlice<T>,+ dev: &CudaDevice,+ layout: &Layout,+ ) -> Result<CudaSlice<T>> {+ let shape = layout.shape();+ let dims = shape.dims();+ let el_count = shape.elem_count();+ let cfg = LaunchConfig::for_num_elems(el_count as u32);+ let ds = SlicePtrOrNull::params_from_layout(dev, layout)?;+ let src = &src.slice(layout.start_offset()..);+ let func = dev.get_or_load_func(&kernel_name::<T>(U::KERNEL), &kernels::UNARY)?;+ // SAFETY: Set later by running the kernel.+ let mut out = unsafe { dev.alloc::<T>(el_count)? };+ let mut builder = func.builder();+ barg!(builder, el_count);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ builder.arg(src);+ builder.arg(&mut out);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(out)+ }+}++fn slice_ptr<T: DeviceRepr>(v: &CudaSlice<T>, lo: usize) -> (u64, cudarc::driver::SyncOnDrop<'_>) {+ let (_, guard) = v.device_ptr(v.stream());+ let (ptr, _) = v.slice(lo..).device_ptr(v.stream());+ (ptr, guard)+}++struct IndexSelect<'a>(&'a CudaStorage, &'a Layout, usize);+impl Map1 for IndexSelect<'_> {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ src: &CudaSlice<T>,+ dev: &CudaDevice,+ src_l: &Layout,+ ) -> Result<CudaSlice<T>> {+ let ids_l = &self.1;+ let (name, (ids, _guard)) = match &self.0.slice {+ CudaStorageSlice::U32(slice) => ("is_u32", slice_ptr(slice, ids_l.start_offset())),+ CudaStorageSlice::U8(slice) => ("is_u8", slice_ptr(slice, ids_l.start_offset())),+ CudaStorageSlice::I64(slice) => ("is_i64", slice_ptr(slice, ids_l.start_offset())),+ _ => Err(CudaError::UnexpectedDType {+ msg: "index_select ids should be u8, u32, or i64",+ expected: DType::U32,+ got: self.0.dtype(),+ })+ .w()?,+ };+ let ids_shape = ids_l.shape();+ let ids_dims = ids_shape.dims();+ let ds = SlicePtrOrNull::params_from_vec(dev, [ids_dims, ids_l.stride()].concat())?;+ let src = match src_l.contiguous_offsets() {+ Some((o1, o2)) => src.slice(o1..o2),+ None => Err(crate::Error::RequiresContiguous { op: "index-select" }.bt())?,+ };+ let left_size: usize = src_l.dims()[..self.2].iter().product();+ let right_size: usize = src_l.dims()[self.2 + 1..].iter().product();+ let src_dim_size = src_l.dims()[self.2];+ let ids_dim_size = ids_shape.elem_count();+ let dst_el = ids_shape.elem_count() * left_size * right_size;+ let cfg = LaunchConfig::for_num_elems(dst_el as u32);+ let func = dev.get_or_load_func(&kernel_name::<T>(name), &kernels::INDEXING)?;+ // SAFETY: Set later by running the kernel.+ let out = unsafe { dev.alloc::<T>(dst_el)? };+ let mut builder = func.builder();+ barg!(builder, dst_el);+ barg!(builder, ids_dims.len());+ ds.builder_arg(&mut builder);+ barg!(builder, ids);+ builder.arg(&src);+ builder.arg(&out);+ barg!(builder, left_size);+ barg!(builder, src_dim_size);+ barg!(builder, ids_dim_size);+ barg!(builder, right_size);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(out)+ }+}++struct Gather<'a>(&'a CudaStorage, &'a Layout, usize);+impl Map1 for Gather<'_> {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ src: &CudaSlice<T>,+ dev: &CudaDevice,+ src_l: &Layout,+ ) -> Result<CudaSlice<T>> {+ let ids = &self.0;+ let ids_l = &self.1;+ let dim = self.2;+ let (ids_o1, _) = match ids_l.contiguous_offsets() {+ Some(o12) => o12,+ None => Err(crate::Error::RequiresContiguous { op: "gather" }.bt())?,+ };+ let (name, (ids, _guard)) = match &ids.slice {+ CudaStorageSlice::U32(slice) => ("gather_u32", slice_ptr(slice, ids_o1)),+ CudaStorageSlice::U8(slice) => ("gather_u8", slice_ptr(slice, ids_o1)),+ CudaStorageSlice::I64(slice) => ("gather_i64", slice_ptr(slice, ids_o1)),+ _ => Err(CudaError::UnexpectedDType {+ msg: "gather ids should be u8/u32/i64",+ expected: DType::U32,+ got: ids.dtype(),+ })?,+ };+ let el = ids_l.shape().elem_count();+ let cfg = LaunchConfig::for_num_elems(el as u32);+ let src = match src_l.contiguous_offsets() {+ Some((o1, o2)) => src.slice(o1..o2),+ None => Err(crate::Error::RequiresContiguous { op: "gather" }.bt())?,+ };+ let left_sz: usize = src_l.dims()[..dim].iter().product();+ let right_sz: usize = src_l.dims()[dim + 1..].iter().product();+ let src_dim_sz = src_l.dims()[dim];+ let ids_dim_sz = ids_l.dims()[dim];+ let func = dev.get_or_load_func(&kernel_name::<T>(name), &kernels::INDEXING)?;+ // SAFETY: Set later by running the kernel.+ let out = unsafe { dev.alloc::<T>(el)? };+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, ids);+ builder.arg(&src);+ builder.arg(&out);+ barg!(builder, left_sz);+ barg!(builder, src_dim_sz);+ barg!(builder, ids_dim_sz);+ barg!(builder, right_sz);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(out)+ }+}++struct IndexAdd<'a>(&'a CudaStorage, &'a Layout, usize);+impl Map2InPlace for IndexAdd<'_> {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ dst: &mut CudaSlice<T>,+ dst_l: &Layout,+ src: &CudaSlice<T>,+ src_l: &Layout,+ dev: &CudaDevice,+ ) -> Result<()> {+ let ids = &self.0;+ let ids_l = &self.1;+ let dim = self.2;+ let (ids_o1, _) = match ids_l.contiguous_offsets() {+ Some(o12) => o12,+ None => Err(crate::Error::RequiresContiguous { op: "index-add" }.bt())?,+ };+ let (name, (ids, _guard)) = match &ids.slice {+ CudaStorageSlice::U32(slice) => ("ia_u32", slice_ptr(slice, ids_o1)),+ CudaStorageSlice::I64(slice) => ("ia_i64", slice_ptr(slice, ids_o1)),+ CudaStorageSlice::U8(slice) => ("ia_u8", slice_ptr(slice, ids_o1)),+ _ => Err(CudaError::UnexpectedDType {+ msg: "index-add ids should be u8/u32/i64",+ expected: DType::U32,+ got: ids.dtype(),+ })?,+ };+ let dst = match dst_l.contiguous_offsets() {+ Some((o1, o2)) => dst.slice(o1..o2),+ None => Err(crate::Error::RequiresContiguous { op: "index-add" }.bt())?,+ };+ let src = match src_l.contiguous_offsets() {+ Some((o1, o2)) => src.slice(o1..o2),+ None => Err(crate::Error::RequiresContiguous { op: "index-add" }.bt())?,+ };+ let left_sz: usize = src_l.dims()[..dim].iter().product();+ let right_sz: usize = src_l.dims()[dim + 1..].iter().product();+ let src_dim_sz = src_l.dims()[dim];+ let dst_dim_sz = dst_l.dims()[dim];+ let ids_dim_sz = ids_l.dims()[0];+ let cfg = LaunchConfig::for_num_elems((left_sz * right_sz) as u32);+ let func = dev.get_or_load_func(&kernel_name::<T>(name), &kernels::INDEXING)?;+ let mut builder = func.builder();+ barg!(builder, ids);+ barg!(builder, ids_dim_sz);+ builder.arg(&src);+ builder.arg(&dst);+ barg!(builder, left_sz, src_dim_sz, dst_dim_sz, right_sz);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(())+ }+}++struct Scatter<'a>(&'a CudaStorage, &'a Layout, usize);+impl Map2InPlace for Scatter<'_> {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ dst: &mut CudaSlice<T>,+ dst_l: &Layout,+ src: &CudaSlice<T>,+ src_l: &Layout,+ dev: &CudaDevice,+ ) -> Result<()> {+ let ids = &self.0;+ let ids_l = &self.1;+ let dim = self.2;+ let (ids_o1, _) = match ids_l.contiguous_offsets() {+ Some(o12) => o12,+ None => Err(crate::Error::RequiresContiguous { op: "scatter" }.bt())?,+ };+ let (name, (ids, _guard)) = match &ids.slice {+ CudaStorageSlice::U32(slice) => ("s_u32", slice_ptr(slice, ids_o1)),+ CudaStorageSlice::I64(slice) => ("s_i64", slice_ptr(slice, ids_o1)),+ CudaStorageSlice::U8(slice) => ("s_u8", slice_ptr(slice, ids_o1)),+ _ => Err(CudaError::UnexpectedDType {+ msg: "scatter ids should be u8/u32/i64",+ expected: DType::U32,+ got: ids.dtype(),+ })?,+ };+ let dst = match dst_l.contiguous_offsets() {+ Some((o1, o2)) => dst.slice(o1..o2),+ None => Err(crate::Error::RequiresContiguous { op: "scatter" }.bt())?,+ };+ let src = match src_l.contiguous_offsets() {+ Some((o1, o2)) => src.slice(o1..o2),+ None => Err(crate::Error::RequiresContiguous { op: "scatter" }.bt())?,+ };+ let left_sz: usize = src_l.dims()[..dim].iter().product();+ let right_sz: usize = src_l.dims()[dim + 1..].iter().product();+ let src_dim_sz = src_l.dims()[dim];+ let dst_dim_sz = dst_l.dims()[dim];+ let cfg = LaunchConfig::for_num_elems((left_sz * right_sz) as u32);+ let func = dev.get_or_load_func(&kernel_name::<T>(name), &kernels::INDEXING)?;+ let mut builder = func.builder();+ barg!(builder, ids);+ builder.arg(&src);+ builder.arg(&dst);+ barg!(builder, left_sz, src_dim_sz, dst_dim_sz, right_sz);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(())+ }+}++struct ScatterAdd<'a>(&'a CudaStorage, &'a Layout, usize);+impl Map2InPlace for ScatterAdd<'_> {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ dst: &mut CudaSlice<T>,+ dst_l: &Layout,+ src: &CudaSlice<T>,+ src_l: &Layout,+ dev: &CudaDevice,+ ) -> Result<()> {+ let ids = &self.0;+ let ids_l = &self.1;+ let dim = self.2;+ let (ids_o1, _) = match ids_l.contiguous_offsets() {+ Some(o12) => o12,+ None => Err(crate::Error::RequiresContiguous { op: "scatter-add" }.bt())?,+ };+ let (name, (ids, _guard)) = match &ids.slice {+ CudaStorageSlice::U32(slice) => ("sa_u32", slice_ptr(slice, ids_o1)),+ CudaStorageSlice::I64(slice) => ("sa_i64", slice_ptr(slice, ids_o1)),+ CudaStorageSlice::U8(slice) => ("sa_u8", slice_ptr(slice, ids_o1)),+ _ => Err(CudaError::UnexpectedDType {+ msg: "scatter-add ids should be u8/u32/i64",+ expected: DType::U32,+ got: ids.dtype(),+ })?,+ };+ let dst = match dst_l.contiguous_offsets() {+ Some((o1, o2)) => dst.slice(o1..o2),+ None => Err(crate::Error::RequiresContiguous { op: "scatter-add" }.bt())?,+ };+ let src = match src_l.contiguous_offsets() {+ Some((o1, o2)) => src.slice(o1..o2),+ None => Err(crate::Error::RequiresContiguous { op: "scatter-add" }.bt())?,+ };+ let left_sz: usize = src_l.dims()[..dim].iter().product();+ let right_sz: usize = src_l.dims()[dim + 1..].iter().product();+ let src_dim_sz = src_l.dims()[dim];+ let dst_dim_sz = dst_l.dims()[dim];+ let cfg = LaunchConfig::for_num_elems((left_sz * right_sz) as u32);+ let func = dev.get_or_load_func(&kernel_name::<T>(name), &kernels::INDEXING)?;+ let mut builder = func.builder();+ barg!(builder, ids);+ builder.arg(&src);+ builder.arg(&dst);+ barg!(builder, left_sz, src_dim_sz, dst_dim_sz, right_sz);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(())+ }+}++struct Conv1D<'a>(&'a crate::conv::ParamsConv1D);+impl Map2 for Conv1D<'_> {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ inp: &CudaSlice<T>,+ inp_l: &Layout,+ k: &CudaSlice<T>,+ k_l: &Layout,+ dev: &CudaDevice,+ ) -> Result<CudaSlice<T>> {+ // Kernel shape: (c_out, c_in_k, k_size)+ // Input shape: (b_size, c_in, l_in) or (c_in, l_in)+ let p = &self.0;+ let inp = &inp.slice(inp_l.start_offset()..);+ let k = &k.slice(k_l.start_offset()..);+ let shape = inp_l.shape();+ let dims = shape.dims();+ let el = shape.elem_count();+ let l_out = p.l_out();+ let dst_el = p.c_out * l_out * p.b_size;+ let cfg = LaunchConfig::for_num_elems(dst_el as u32);+ let func = dev.get_or_load_func(&kernel_name::<T>("conv1d"), &kernels::CONV)?;+ // SAFETY: Set later by running the kernel.+ let out = unsafe { dev.alloc::<T>(dst_el)? };+ let ds = if dims.len() == 3 {+ [dims, inp_l.stride(), k_l.dims(), k_l.stride()].concat()+ } else if dims.len() == 2 {+ [&[1], dims, &[1], inp_l.stride(), k_l.dims(), k_l.stride()].concat()+ } else {+ crate::bail!("unexpected input shape for conv1d {dims:?}")+ };+ let ds = SlicePtrOrNull::params_from_vec(dev, ds)?;+ let mut builder = func.builder();+ barg!(builder, el, l_out, p.stride, p.padding, p.dilation);+ ds.builder_arg(&mut builder);+ builder.arg(inp);+ builder.arg(k);+ builder.arg(&out);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(out)+ }+}++struct Conv2D<'a>(&'a crate::conv::ParamsConv2D);+impl Map2 for Conv2D<'_> {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ inp: &CudaSlice<T>,+ inp_l: &Layout,+ k: &CudaSlice<T>,+ k_l: &Layout,+ dev: &CudaDevice,+ ) -> Result<CudaSlice<T>> {+ // Kernel shape: (c_out, c_in_k, h_k, w_k)+ // Input shape: (b_size, c_in, h_in, w_in)+ let p = &self.0;+ let (out_w, out_h) = (p.out_w(), p.out_h());+ let dst_el = p.c_out * out_w * out_h * p.b_size;+ let inp = &inp.slice(inp_l.start_offset()..);+ let k = &k.slice(k_l.start_offset()..);+ let shape = inp_l.shape();+ let dims = shape.dims();+ let el = shape.elem_count();++ // SAFETY: Set later by running the kernel.+ let out = unsafe { dev.alloc::<T>(dst_el)? };+ let cfg = LaunchConfig::for_num_elems(dst_el as u32);+ let func = dev.get_or_load_func(&kernel_name::<T>("conv2d"), &kernels::CONV)?;+ let ds = if dims.len() == 4 {+ [dims, inp_l.stride(), k_l.dims(), k_l.stride()].concat()+ } else {+ crate::bail!("unexpected input shape for conv2d {dims:?}")+ };+ let ds = SlicePtrOrNull::params_from_vec(dev, ds)?;+ let mut builder = func.builder();+ barg!(builder, el, out_w, out_h, p.stride, p.padding, p.dilation);+ ds.builder_arg(&mut builder);+ builder.arg(inp);+ builder.arg(k);+ builder.arg(&out);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(out)+ }+}++struct Col2Im1D {+ stride: usize,+}++impl Map1 for Col2Im1D {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ col: &CudaSlice<T>,+ dev: &CudaDevice,+ l: &Layout,+ ) -> Result<CudaSlice<T>> {+ let (b_size, l_in, c_out, k_size) = l.shape().dims4()?;+ let stride = self.stride;+ let l_out = (l_in - 1) * stride + k_size;+ let dst_el = b_size * c_out * l_out;+ let mut im = unsafe { dev.alloc::<T>(dst_el)? };++ let cfg = LaunchConfig::for_num_elems(dst_el as u32);+ let func = dev.get_or_load_func(&kernel_name::<T>("col2im1d"), &kernels::CONV)?;+ let mut builder = func.builder();+ barg!(builder, dst_el, l_out, l_in, c_out, k_size, stride);+ builder.arg(col);+ builder.arg(&mut im);+ unsafe { builder.launch(cfg) }.w()?;+ Ok(im)+ }+}++struct ConvTranspose1D<'a>(&'a crate::conv::ParamsConvTranspose1D);+impl Map2 for ConvTranspose1D<'_> {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ inp: &CudaSlice<T>,+ inp_l: &Layout,+ k: &CudaSlice<T>,+ k_l: &Layout,+ dev: &CudaDevice,+ ) -> Result<CudaSlice<T>> {+ // Kernel shape: (c_in_k, c_out, l_k)+ // Input shape: (b_size, c_in, l_in)+ let p = &self.0;+ let l_out = p.l_out();+ let dst_el = p.c_out * l_out * p.b_size;+ let inp = &inp.slice(inp_l.start_offset()..);+ let k = &k.slice(k_l.start_offset()..);+ let shape = inp_l.shape();+ let dims = shape.dims();+ let el = shape.elem_count();++ // SAFETY: Set later by running the kernel.+ let out = unsafe { dev.alloc::<T>(dst_el)? };+ let cfg = LaunchConfig::for_num_elems(dst_el as u32);+ let func = dev.get_or_load_func(&kernel_name::<T>("conv_transpose1d"), &kernels::CONV)?;+ let ds = if dims.len() == 3 {+ [dims, inp_l.stride(), k_l.dims(), k_l.stride()].concat()+ } else {+ crate::bail!("unexpected input shape for conv_transpose1d {dims:?}")+ };+ let ds = SlicePtrOrNull::params_from_vec(dev, ds)?;+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, l_out);+ barg!(builder, p.stride);+ barg!(builder, p.padding);+ barg!(builder, p.output_padding);+ barg!(builder, p.dilation);+ ds.builder_arg(&mut builder);+ builder.arg(inp);+ builder.arg(k);+ builder.arg(&out);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(out)+ }+}++struct ConvTranspose2D<'a>(&'a crate::conv::ParamsConvTranspose2D);+impl Map2 for ConvTranspose2D<'_> {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ inp: &CudaSlice<T>,+ inp_l: &Layout,+ k: &CudaSlice<T>,+ k_l: &Layout,+ dev: &CudaDevice,+ ) -> Result<CudaSlice<T>> {+ // Kernel shape: (c_in_k, c_out, h_k, w_k)+ // Input shape: (b_size, c_in, h_in, w_in)+ let p = &self.0;+ let (out_w, out_h) = (p.out_w(), p.out_h());+ let dst_el = p.c_out * out_w * out_h * p.b_size;+ let inp = &inp.slice(inp_l.start_offset()..);+ let k = &k.slice(k_l.start_offset()..);+ let shape = inp_l.shape();+ let dims = shape.dims();+ let el = shape.elem_count();++ // SAFETY: Set later by running the kernel.+ let out = unsafe { dev.alloc::<T>(dst_el)? };+ let cfg = LaunchConfig::for_num_elems(dst_el as u32);+ let func = dev.get_or_load_func(&kernel_name::<T>("conv_transpose2d"), &kernels::CONV)?;+ let ds = if dims.len() == 4 {+ [dims, inp_l.stride(), k_l.dims(), k_l.stride()].concat()+ } else {+ crate::bail!("unexpected input shape for conv_transpose2d {dims:?}")+ };+ let ds = SlicePtrOrNull::params_from_vec(dev, ds)?;+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, out_w);+ barg!(builder, out_h);+ barg!(builder, p.stride);+ barg!(builder, p.padding);+ barg!(builder, p.output_padding);+ barg!(builder, p.dilation);+ ds.builder_arg(&mut builder);+ builder.arg(inp);+ builder.arg(k);+ builder.arg(&out);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(out)+ }+}++enum PoolOp {+ Max,+ Avg,+}++struct Pool2D {+ w_k: usize,+ h_k: usize,+ w_stride: usize,+ h_stride: usize,+ op: PoolOp,+}++impl Map1 for Pool2D {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ inp: &CudaSlice<T>,+ dev: &CudaDevice,+ inp_l: &Layout,+ ) -> Result<CudaSlice<T>> {+ // Input shape: (b_size, c, h, w)+ let inp = &inp.slice(inp_l.start_offset()..);+ let shape = inp_l.shape();+ let dims = shape.dims();+ let ds = if dims.len() == 4 {+ [dims, inp_l.stride()].concat()+ } else {+ crate::bail!("unexpected input shape for pool {dims:?}")+ };+ let el = shape.elem_count();+ let out_w = (dims[2] - self.w_k) / self.w_stride + 1;+ let out_h = (dims[3] - self.h_k) / self.h_stride + 1;+ let dst_el = out_w * out_h * dims[0] * dims[1];+ let cfg = LaunchConfig::for_num_elems(dst_el as u32);+ let kname = match self.op {+ PoolOp::Max => "max_pool2d",+ PoolOp::Avg => "avg_pool2d",+ };+ let func = dev.get_or_load_func(&kernel_name::<T>(kname), &kernels::CONV)?;+ // SAFETY: Set later by running the kernel.+ let out = unsafe { dev.alloc::<T>(dst_el)? };+ let ds = SlicePtrOrNull::params_from_vec(dev, ds)?;+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, self.w_k);+ barg!(builder, self.h_k);+ barg!(builder, self.w_stride);+ barg!(builder, self.h_stride);+ ds.builder_arg(&mut builder);+ builder.arg(inp);+ builder.arg(&out);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(out)+ }+}++struct UpsampleNearest2D(usize, usize);+impl Map1 for UpsampleNearest2D {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ inp: &CudaSlice<T>,+ dev: &CudaDevice,+ inp_l: &Layout,+ ) -> Result<CudaSlice<T>> {+ // Input shape: (b_size, c, h, w)+ let inp = &inp.slice(inp_l.start_offset()..);+ let shape = inp_l.shape();+ let dims = shape.dims();+ let ds = if dims.len() == 4 {+ [dims, inp_l.stride()].concat()+ } else {+ crate::bail!("unexpected input shape for upsample {dims:?}")+ };+ let (out_w, out_h) = (self.0, self.1);+ let dst_el = out_w * out_h * dims[0] * dims[1];+ let cfg = LaunchConfig::for_num_elems(dst_el as u32);+ let func = dev.get_or_load_func(&kernel_name::<T>("upsample_nearest2d"), &kernels::CONV)?;+ // SAFETY: Set later by running the kernel.+ let out = unsafe { dev.alloc::<T>(dst_el)? };+ let ds = SlicePtrOrNull::params_from_vec(dev, ds)?;+ let scale_w = dims[2] as f64 / out_w as f64;+ let scale_h = dims[3] as f64 / out_h as f64;+ let mut builder = func.builder();+ barg!(builder, out_w);+ barg!(builder, out_h);+ barg!(builder, scale_w);+ barg!(builder, scale_h);+ ds.builder_arg(&mut builder);+ builder.arg(inp);+ builder.arg(&out);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(out)+ }+}++struct UpsampleBilinear2D {+ out_w: usize,+ out_h: usize,+ align_corners: bool,+ scale_h_factor: Option<f64>,+ scale_w_factor: Option<f64>,+}++impl Map1 for UpsampleBilinear2D {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ inp: &CudaSlice<T>,+ dev: &CudaDevice,+ inp_l: &Layout,+ ) -> Result<CudaSlice<T>> {+ let inp = &inp.slice(inp_l.start_offset()..);+ let shape = inp_l.shape();+ let dims = shape.dims();+ let ds = if dims.len() == 4 {+ [dims, inp_l.stride()].concat()+ } else {+ crate::bail!("unexpected input shape for upsample_bilinear2d {dims:?}")+ };++ let (out_w, out_h) = (self.out_w, self.out_h);+ let dst_el = out_w * out_h * dims[0] * dims[1];+ let cfg = LaunchConfig::for_num_elems(dst_el as u32);+ let func =+ dev.get_or_load_func(&kernel_name::<T>("upsample_bilinear2d"), &kernels::CONV)?;++ // SAFETY: Set later by running the kernel.+ let out = unsafe { dev.alloc::<T>(dst_el)? };+ let ds = SlicePtrOrNull::params_from_vec(dev, ds)?;++ let mut builder = func.builder();+ barg!(builder, out_w);+ barg!(builder, out_h);+ barg!(builder, self.align_corners);+ barg!(builder, self.scale_h_factor.is_some());+ barg!(builder, self.scale_h_factor.unwrap_or(0.0));+ barg!(builder, self.scale_w_factor.is_some());+ barg!(builder, self.scale_w_factor.unwrap_or(0.0));+ ds.builder_arg(&mut builder);+ builder.arg(inp);+ builder.arg(&out);++ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(out)+ }+}++struct WhereCond<'a>(&'a CudaStorage, &'a Layout);+impl Map2 for WhereCond<'_> {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ t: &CudaSlice<T>,+ layout_t: &Layout,+ f: &CudaSlice<T>,+ layout_f: &Layout,+ dev: &CudaDevice,+ ) -> Result<CudaSlice<T>> {+ let ids_l = &self.1;+ let ((ids, _guard), name) = match &self.0.slice {+ CudaStorageSlice::U8(slice) => {+ let ptr = slice_ptr(slice, ids_l.start_offset());+ (ptr, "where_u8")+ }+ CudaStorageSlice::U32(slice) => {+ let ptr = slice_ptr(slice, ids_l.start_offset());+ (ptr, "where_u32")+ }+ CudaStorageSlice::I64(slice) => {+ let ptr = slice_ptr(slice, ids_l.start_offset());+ (ptr, "where_i64")+ }+ _ => Err(CudaError::UnexpectedDType {+ msg: "where conditions should be u8/u32/i64",+ expected: DType::U32,+ got: self.0.dtype(),+ })+ .w()?,+ };+ let shape = ids_l.shape();+ let dims = shape.dims();+ let el = shape.elem_count();+ let cfg = LaunchConfig::for_num_elems(el as u32);+ let ds = SlicePtrOrNull::params_from_vec(+ dev,+ [dims, ids_l.stride(), layout_t.stride(), layout_f.stride()].concat(),+ )?;+ let t = &t.slice(layout_t.start_offset()..);+ let f = &f.slice(layout_f.start_offset()..);+ let func = dev.get_or_load_func(&kernel_name::<T>(name), &kernels::TERNARY)?;+ // SAFETY: Set later by running the kernel.+ let out = unsafe { dev.alloc::<T>(el)? };+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ barg!(builder, ids);+ builder.arg(t);+ builder.arg(f);+ builder.arg(&out);+ // SAFETY: ffi+ unsafe { builder.launch(cfg) }.w()?;+ Ok(out)+ }+}++impl<U: crate::op::BinaryOpT> Map2 for U {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ lhs: &CudaSlice<T>,+ lhs_l: &Layout,+ rhs: &CudaSlice<T>,+ rhs_l: &Layout,+ dev: &CudaDevice,+ ) -> Result<CudaSlice<T>> {+ let shape = lhs_l.shape();+ let dims = shape.dims();+ let elem_count = shape.elem_count();+ let cfg = LaunchConfig::for_num_elems(elem_count as u32);+ let dims_and_strides = if lhs_l.is_contiguous() && rhs_l.is_contiguous() {+ SlicePtrOrNull::Null+ } else {+ SlicePtrOrNull::params_from_vec(dev, [dims, lhs_l.stride(), rhs_l.stride()].concat())?+ };+ let lhs = &lhs.slice(lhs_l.start_offset()..);+ let rhs = &rhs.slice(rhs_l.start_offset()..);+ let func = dev.get_or_load_func(&kernel_name::<T>(U::KERNEL), &kernels::BINARY)?;+ // SAFETY: Set later by running the kernel.+ let out = unsafe { dev.alloc::<T>(elem_count)? };+ let mut builder = func.builder();+ barg!(builder, elem_count);+ barg!(builder, dims.len());+ dims_and_strides.builder_arg(&mut builder);+ builder.arg(lhs);+ builder.arg(rhs);+ builder.arg(&out);+ // SAFETY: ffi+ unsafe { builder.launch(cfg) }.w()?;+ Ok(out)+ }+}++struct Cmp(CmpOp);+impl Map2Any for Cmp {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ lhs: &CudaSlice<T>,+ lhs_l: &Layout,+ rhs: &CudaSlice<T>,+ rhs_l: &Layout,+ dev: &CudaDevice,+ ) -> Result<S> {+ let shape = lhs_l.shape();+ let dims = shape.dims();+ let elem_count = shape.elem_count();+ let cfg = LaunchConfig::for_num_elems(elem_count as u32);+ let dims_and_strides = if lhs_l.is_contiguous() && rhs_l.is_contiguous() {+ SlicePtrOrNull::Null+ } else {+ SlicePtrOrNull::params_from_vec(dev, [dims, lhs_l.stride(), rhs_l.stride()].concat())?+ };+ let lhs = &lhs.slice(lhs_l.start_offset()..);+ let rhs = &rhs.slice(rhs_l.start_offset()..);+ let name = match self.0 {+ CmpOp::Eq => "eq",+ CmpOp::Ne => "ne",+ CmpOp::Lt => "lt",+ CmpOp::Le => "le",+ CmpOp::Gt => "gt",+ CmpOp::Ge => "ge",+ };+ let func = dev.get_or_load_func(&kernel_name::<T>(name), &kernels::BINARY)?;+ // SAFETY: Set later by running the kernel.+ let out = unsafe { dev.alloc::<u8>(elem_count)? };+ let mut builder = func.builder();+ barg!(builder, elem_count);+ barg!(builder, dims.len());+ dims_and_strides.builder_arg(&mut builder);+ builder.arg(lhs);+ builder.arg(rhs);+ builder.arg(&out);+ // SAFETY: ffi+ unsafe { builder.launch(cfg) }.w()?;+ Ok(S::U8(out))+ }+}++fn slice_src_and_dst<'a, T>(+ src: &'a CudaSlice<T>,+ src_l: &Layout,+ dst: &'a mut CudaSlice<T>,+ dst_offset: usize,+) -> (+ cudarc::driver::CudaView<'a, T>,+ cudarc::driver::CudaViewMut<'a, T>,+) {+ let src_offset = src_l.start_offset();+ // Copy exactly the number of elements the layout describes. Deriving the+ // count from the storage lengths alone over-copies whenever `src` is a view+ // whose storage continues past it -- anything produced by `narrow` -- and+ // that overruns into the elements sitting after `dst_offset` in `dst`. The+ // two storage bounds are kept as a defensive clamp.+ let to_copy = src_l+ .shape()+ .elem_count()+ .min(dst.len().saturating_sub(dst_offset))+ .min(src.len().saturating_sub(src_offset));+ let src = src.slice(src_offset..src_offset + to_copy);+ let dst = dst.slice_mut(dst_offset..dst_offset + to_copy);+ (src, dst)+}++#[derive(Debug)]+pub struct CudaStorage {+ pub slice: CudaStorageSlice,+ pub device: CudaDevice,+}++pub trait CudaDType: Sized {+ fn as_cuda_slice(s: &CudaStorage) -> Result<&CudaSlice<Self>>;+ fn as_cuda_slice_mut(s: &mut CudaStorage) -> Result<&mut CudaSlice<Self>>;+ fn wrap_cuda_slice(s: CudaSlice<Self>, dev: CudaDevice) -> CudaStorage;+}++macro_rules! cuda_dtype {+ ($ty:ty, $dtype:ident) => {+ impl CudaDType for $ty {+ fn as_cuda_slice(s: &CudaStorage) -> Result<&CudaSlice<Self>> {+ match &s.slice {+ CudaStorageSlice::$dtype(data) => Ok(&data),+ _ => Err(crate::Error::UnexpectedDType {+ expected: DType::$dtype,+ got: s.dtype(),+ msg: "unexpected dtype",+ }+ .bt()),+ }+ }++ fn as_cuda_slice_mut(s: &mut CudaStorage) -> Result<&mut CudaSlice<Self>> {+ match s.slice {+ CudaStorageSlice::$dtype(ref mut data) => Ok(data),+ _ => Err(crate::Error::UnexpectedDType {+ expected: DType::$dtype,+ got: s.dtype(),+ msg: "unexpected dtype",+ }+ .bt()),+ }+ }++ fn wrap_cuda_slice(slice: CudaSlice<Self>, device: CudaDevice) -> CudaStorage {+ let slice = CudaStorageSlice::$dtype(slice);+ CudaStorage { slice, device }+ }+ }+ };+}+cuda_dtype!(u8, U8);+cuda_dtype!(u32, U32);+cuda_dtype!(i16, I16);+cuda_dtype!(i32, I32);+cuda_dtype!(i64, I64);+cuda_dtype!(f16, F16);+cuda_dtype!(bf16, BF16);+cuda_dtype!(f32, F32);+cuda_dtype!(f64, F64);+cuda_dtype!(float8::F8E4M3, F8E4M3);++impl CudaStorage {+ pub fn wrap_cuda_slice<T: CudaDType>(slice: CudaSlice<T>, device: CudaDevice) -> CudaStorage {+ T::wrap_cuda_slice(slice, device)+ }++ pub fn as_cuda_slice<T: CudaDType>(&self) -> Result<&CudaSlice<T>> {+ T::as_cuda_slice(self)+ }++ pub fn as_cuda_slice_mut<T: CudaDType>(&mut self) -> Result<&mut CudaSlice<T>> {+ T::as_cuda_slice_mut(self)+ }++ pub fn transfer_to_device(&self, dst: &CudaDevice) -> Result<Self> {+ let dst_stream = dst.cuda_stream();+ let storage_slice = match self.dtype() {+ DType::U8 => {+ let cuda_slice = self.as_cuda_slice::<u8>()?;+ let result = dst_stream.clone_dtod(cuda_slice).w()?;+ CudaStorageSlice::U8(result)+ }+ DType::U32 => {+ let cuda_slice = self.as_cuda_slice::<u32>()?;+ let result = dst_stream.clone_dtod(cuda_slice).w()?;+ CudaStorageSlice::U32(result)+ }+ DType::I16 => {+ let cuda_slice = self.as_cuda_slice::<i16>()?;+ let result = dst_stream.clone_dtod(cuda_slice).w()?;+ CudaStorageSlice::I16(result)+ }+ DType::I32 => {+ let cuda_slice = self.as_cuda_slice::<i32>()?;+ let result = dst_stream.clone_dtod(cuda_slice).w()?;+ CudaStorageSlice::I32(result)+ }+ DType::I64 => {+ let cuda_slice = self.as_cuda_slice::<i64>()?;+ let result = dst_stream.clone_dtod(cuda_slice).w()?;+ CudaStorageSlice::I64(result)+ }+ DType::BF16 => {+ let cuda_slice = self.as_cuda_slice::<bf16>()?;+ let result = dst_stream.clone_dtod(cuda_slice).w()?;+ CudaStorageSlice::BF16(result)+ }+ DType::F16 => {+ let cuda_slice = self.as_cuda_slice::<f16>()?;+ let result = dst_stream.clone_dtod(cuda_slice).w()?;+ CudaStorageSlice::F16(result)+ }+ DType::F32 => {+ let cuda_slice = self.as_cuda_slice::<f32>()?;+ let result = dst_stream.clone_dtod(cuda_slice).w()?;+ CudaStorageSlice::F32(result)+ }+ DType::F64 => {+ let cuda_slice = self.as_cuda_slice::<f64>()?;+ let result = dst_stream.clone_dtod(cuda_slice).w()?;+ CudaStorageSlice::F64(result)+ }+ DType::F8E4M3 => {+ let cuda_slice = self.as_cuda_slice::<float8::F8E4M3>()?;+ let result = dst_stream.clone_dtod(cuda_slice).w()?;+ CudaStorageSlice::F8E4M3(result)+ }+ DType::F6E2M3 => {+ let cuda_slice = self.as_cuda_slice::<u8>()?;+ let result = dst_stream.clone_dtod(cuda_slice).w()?;+ CudaStorageSlice::F6E2M3(result)+ }+ DType::F6E3M2 => {+ let cuda_slice = self.as_cuda_slice::<u8>()?;+ let result = dst_stream.clone_dtod(cuda_slice).w()?;+ CudaStorageSlice::F6E3M2(result)+ }+ DType::F4 => {+ let cuda_slice = self.as_cuda_slice::<u8>()?;+ let result = dst_stream.clone_dtod(cuda_slice).w()?;+ CudaStorageSlice::F4(result)+ }+ DType::F8E8M0 => {+ let cuda_slice = self.as_cuda_slice::<u8>()?;+ let result = dst_stream.clone_dtod(cuda_slice).w()?;+ CudaStorageSlice::F8E8M0(result)+ }+ };++ Ok(Self {+ slice: storage_slice,+ device: dst.clone(),+ })+ }+}++fn gemm_config<T>(+ alpha: T,+ beta: T,+ (b, m, n, k): (usize, usize, usize, usize),+ lhs_l: &Layout,+ rhs_l: &Layout,+) -> Result<StridedBatchedConfig<T>> {+ // https://docs.nvidia.com/cuda/cublas/index.html#cublas-t-gemm+ use cudarc::cublas::sys::cublasOperation_t;++ let lhs_stride = lhs_l.stride();+ let rhs_stride = rhs_l.stride();+ let rhs_m1 = rhs_stride[rhs_stride.len() - 1];+ let rhs_m2 = rhs_stride[rhs_stride.len() - 2];+ let lhs_m1 = lhs_stride[lhs_stride.len() - 1];+ let lhs_m2 = lhs_stride[lhs_stride.len() - 2];+ // The a tensor has dims batching, k, n (rhs)+ // We also allow for the case where the stride on the minor dimension is not as expected but+ // there is a single element.+ let (lda, transa) = if (rhs_m1 == 1 || n == 1) && (rhs_m2 == n || k == 1) {+ (n as i32, cublasOperation_t::CUBLAS_OP_N)+ } else if (rhs_m1 == k || n == 1) && (rhs_m2 == 1 || k == 1) {+ (k as i32, cublasOperation_t::CUBLAS_OP_T)+ } else {+ Err(CudaError::MatMulNonContiguous {+ lhs_stride: lhs_l.clone(),+ rhs_stride: rhs_l.clone(),+ mnk: (m, n, k),+ })?+ };+ // The b tensor has dims batching, m, k (lhs)+ // We also allow for the case where the stride on the minor dimension is not as expected but+ // there is a single element.+ let (ldb, transb) = if (lhs_m1 == 1 || k == 1) && (lhs_m2 == k || m == 1) {+ (k as i32, cublasOperation_t::CUBLAS_OP_N)+ } else if (lhs_m1 == m || k == 1) && (lhs_m2 == 1 || m == 1) {+ (m as i32, cublasOperation_t::CUBLAS_OP_T)+ } else {+ Err(CudaError::MatMulNonContiguous {+ lhs_stride: lhs_l.clone(),+ rhs_stride: rhs_l.clone(),+ mnk: (m, n, k),+ })?+ };+ // The setup below was copied from:+ // https://github.com/lebedov/scikit-cuda/blob/7e7300474286019c917a6c8a4bca59405c64fbce/tests/test_cublas.py#L531+ let gemm = GemmConfig {+ alpha,+ beta,+ m: n as i32,+ n: m as i32,+ k: k as i32,+ lda,+ ldb,+ ldc: n as i32,+ transa,+ transb,+ };++ let stride_b: usize = match lhs_stride[..lhs_stride.len() - 2] {+ [s1, stride] if s1 == stride * lhs_l.dims()[1] => stride,+ [_, stride] if lhs_l.dims()[0] == 1 => stride,+ [stride, _] if lhs_l.dims()[1] == 1 => stride,+ [stride] => stride,+ [] => m * k,+ _ => Err(CudaError::MatMulNonContiguous {+ lhs_stride: lhs_l.clone(),+ rhs_stride: rhs_l.clone(),+ mnk: (m, n, k),+ })?,+ };+ let stride_a: usize = match rhs_stride[..rhs_stride.len() - 2] {+ [s1, stride] if s1 == stride * rhs_l.dims()[1] => stride,+ [_, stride] if rhs_l.dims()[0] == 1 => stride,+ [stride, _] if rhs_l.dims()[1] == 1 => stride,+ [stride] => stride,+ [] => n * k,+ _ => Err(CudaError::MatMulNonContiguous {+ lhs_stride: lhs_l.clone(),+ rhs_stride: rhs_l.clone(),+ mnk: (m, n, k),+ })?,+ };+ Ok(StridedBatchedConfig {+ batch_size: b as i32,+ gemm,+ stride_a: stride_a as i64,+ stride_b: stride_b as i64,+ stride_c: (m * n) as i64,+ })+}++impl BackendStorage for CudaStorage {+ type Device = CudaDevice;++ fn try_clone(&self, layout: &Layout) -> Result<Self> {+ let slice = Clone.map(&self.slice, self.device(), layout)?;+ let device = self.device.clone();+ Ok(Self { slice, device })+ }++ fn dtype(&self) -> DType {+ match self.slice {+ CudaStorageSlice::U8(_) => DType::U8,+ CudaStorageSlice::U32(_) => DType::U32,+ CudaStorageSlice::I16(_) => DType::I16,+ CudaStorageSlice::I32(_) => DType::I32,+ CudaStorageSlice::I64(_) => DType::I64,+ CudaStorageSlice::BF16(_) => DType::BF16,+ CudaStorageSlice::F16(_) => DType::F16,+ CudaStorageSlice::F32(_) => DType::F32,+ CudaStorageSlice::F64(_) => DType::F64,+ CudaStorageSlice::F8E4M3(_) => DType::F8E4M3,+ CudaStorageSlice::F6E2M3(_) => DType::F6E2M3,+ CudaStorageSlice::F6E3M2(_) => DType::F6E3M2,+ CudaStorageSlice::F4(_) => DType::F4,+ CudaStorageSlice::F8E8M0(_) => DType::F8E8M0,+ }+ }++ fn device(&self) -> &CudaDevice {+ &self.device+ }++ fn const_set(&mut self, s: crate::scalar::Scalar, layout: &Layout) -> Result<()> {+ let dev = &self.device;+ let shape = layout.shape();+ let dims = shape.dims();+ let el_count = shape.elem_count();+ let cfg = LaunchConfig::for_num_elems(el_count as u32);+ let ds = SlicePtrOrNull::params_from_layout(dev, layout)?;+ let src_o = layout.start_offset();+ let ((src, _guard_src), kernel_name) = match &mut self.slice {+ S::U8(s) => (slice_ptr(s, src_o), "const_set_u8"),+ S::U32(s) => (slice_ptr(s, src_o), "const_set_u32"),+ S::I16(s) => (slice_ptr(s, src_o), "const_set_i16"),+ S::I32(s) => (slice_ptr(s, src_o), "const_set_i32"),+ S::I64(s) => (slice_ptr(s, src_o), "const_set_i64"),+ S::BF16(s) => (slice_ptr(s, src_o), "const_set_bf16"),+ S::F16(s) => (slice_ptr(s, src_o), "const_set_f16"),+ S::F32(s) => (slice_ptr(s, src_o), "const_set_f32"),+ S::F64(s) => (slice_ptr(s, src_o), "const_set_f64"),+ S::F8E4M3(s) => (slice_ptr(s, src_o), "const_set_f8_e4m3"),+ S::F4(_) | S::F6E2M3(_) | S::F6E3M2(_) | S::F8E8M0(_) => {+ return Err(CudaError::UnsupportedDtype {+ dtype: self.dtype(),+ op: "const_set",+ }+ .into());+ }+ };++ let func = dev.get_or_load_func(kernel_name, &kernels::FILL)?;+ let mut builder = func.builder();+ barg!(builder, el_count);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ s.builder_arg(&mut builder);+ barg!(builder, src);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(())+ }++ fn to_dtype(&self, layout: &Layout, dtype: DType) -> Result<Self> {+ let shape = layout.shape();+ let dims = shape.dims();+ let el = shape.elem_count();+ let cfg = LaunchConfig::for_num_elems(el as u32);+ let dev = self.device();+ let ds = SlicePtrOrNull::params_from_layout(dev, layout)?;+ let start_o = layout.start_offset();+ // This returns an i64 rather than a &i64, this is useful to get around some temporary+ // lifetime issue and is safe as long as self.slice does not go out of scope before inp+ // is used.+ let (inp, _guard) = match &self.slice {+ CudaStorageSlice::U8(inp) => slice_ptr(inp, start_o),+ CudaStorageSlice::U32(inp) => slice_ptr(inp, start_o),+ CudaStorageSlice::I16(inp) => slice_ptr(inp, start_o),+ CudaStorageSlice::I32(inp) => slice_ptr(inp, start_o),+ CudaStorageSlice::I64(inp) => slice_ptr(inp, start_o),+ CudaStorageSlice::BF16(inp) => slice_ptr(inp, start_o),+ CudaStorageSlice::F16(inp) => slice_ptr(inp, start_o),+ CudaStorageSlice::F32(inp) => slice_ptr(inp, start_o),+ CudaStorageSlice::F64(inp) => slice_ptr(inp, start_o),+ CudaStorageSlice::F8E4M3(inp) => slice_ptr(inp, start_o),+ CudaStorageSlice::F4(_)+ | CudaStorageSlice::F6E2M3(_)+ | CudaStorageSlice::F6E3M2(_)+ | CudaStorageSlice::F8E8M0(_) => {+ return Err(CudaError::UnsupportedDtype {+ dtype: self.dtype(),+ op: "to_dtype",+ }+ .into());+ }+ };+ let inp = &inp;++ let kernel_name = format!("cast_{}_{}", self.dtype().as_str(), dtype.as_str());+ let func = dev.get_or_load_func(&kernel_name, &kernels::CAST)?;+ let slice = match dtype {+ DType::U8 => {+ let out = unsafe { dev.alloc::<u8>(el)? };+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ barg!(builder, *inp);+ builder.arg(&out);+ unsafe { builder.launch(cfg) }.w()?;+ CudaStorageSlice::U8(out)+ }+ DType::U32 => {+ let out = unsafe { dev.alloc::<u32>(el)? };+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ barg!(builder, *inp);+ builder.arg(&out);+ unsafe { builder.launch(cfg) }.w()?;+ CudaStorageSlice::U32(out)+ }+ DType::I64 => {+ let out = unsafe { dev.alloc::<i64>(el)? };+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ barg!(builder, *inp);+ builder.arg(&out);+ unsafe { builder.launch(cfg) }.w()?;+ CudaStorageSlice::I64(out)+ }+ DType::BF16 => {+ let out = unsafe { dev.alloc::<bf16>(el)? };+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ barg!(builder, *inp);+ builder.arg(&out);+ unsafe { builder.launch(cfg) }.w()?;+ CudaStorageSlice::BF16(out)+ }+ DType::F16 => {+ let out = unsafe { dev.alloc::<f16>(el)? };+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ barg!(builder, *inp);+ builder.arg(&out);+ unsafe { builder.launch(cfg) }.w()?;+ CudaStorageSlice::F16(out)+ }+ DType::F32 => {+ let out = unsafe { dev.alloc::<f32>(el)? };+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ barg!(builder, *inp);+ builder.arg(&out);+ unsafe { builder.launch(cfg) }.w()?;+ CudaStorageSlice::F32(out)+ }+ DType::F64 => {+ let out = unsafe { dev.alloc::<f64>(el)? };+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ barg!(builder, *inp);+ builder.arg(&out);+ unsafe { builder.launch(cfg) }.w()?;+ CudaStorageSlice::F64(out)+ }+ DType::F8E4M3 => {+ let out = unsafe { dev.alloc::<float8::F8E4M3>(el)? };+ let mut builder = func.builder();+ barg!(builder, el);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ barg!(builder, *inp);+ builder.arg(&out);+ unsafe { builder.launch(cfg) }.w()?;+ CudaStorageSlice::F8E4M3(out)+ }+ DType::I16 | DType::I32 => {+ return Err(CudaError::InternalError("i16,i32 dtypes are not supported").into())+ }+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ return Err(+ CudaError::InternalError("Dummy types not supported in CUDA backend").into(),+ )+ }+ };+ Ok(Self {+ slice,+ device: dev.clone(),+ })+ }++ fn affine(&self, layout: &Layout, mul: f64, add: f64) -> Result<Self> {+ let device = self.device().clone();+ let slice = Affine(mul, add).map(&self.slice, &device, layout)?;+ Ok(Self { slice, device })+ }++ fn powf(&self, layout: &Layout, e: f64) -> Result<Self> {+ let device = self.device().clone();+ let slice = Powf(e).map(&self.slice, &device, layout)?;+ Ok(Self { slice, device })+ }++ fn elu(&self, layout: &Layout, alpha: f64) -> Result<Self> {+ let device = self.device().clone();+ let slice = Elu(alpha).map(&self.slice, &device, layout)?;+ Ok(Self { slice, device })+ }++ fn reduce_op(&self, op: ReduceOp, layout: &Layout, sum_dims: &[usize]) -> Result<Self> {+ let device = self.device().clone();+ let slice = FastReduce(sum_dims, op).map(&self.slice, &device, layout)?;+ Ok(Self { slice, device })+ }++ fn cmp(&self, op: CmpOp, rhs: &Self, lhs_l: &Layout, rhs_l: &Layout) -> Result<Self> {+ let device = self.device().clone();+ let slice = Cmp(op).map(&self.slice, lhs_l, &rhs.slice, rhs_l, &device)?;+ Ok(Self { slice, device })+ }++ fn unary_impl<U: UnaryOpT>(&self, layout: &Layout) -> Result<Self> {+ let device = self.device().clone();+ let slice = U::V.map(&self.slice, &device, layout)?;+ Ok(Self { slice, device })+ }++ fn binary_impl<B: BinaryOpT>(+ &self,+ rhs: &Self,+ lhs_l: &Layout,+ rhs_l: &Layout,+ ) -> Result<Self> {+ let device = self.device().clone();+ let slice = B::V.map(&self.slice, lhs_l, &rhs.slice, rhs_l, &device)?;+ Ok(Self { slice, device })+ }++ fn to_cpu_storage(&self) -> Result<CpuStorage> {+ if device::cuda_graph_htod_cache_enabled() {+ let is_capturing = self.device.cuda_stream().capture_status()+ == Ok(cudarc::driver::sys::CUstreamCaptureStatus::CU_STREAM_CAPTURE_STATUS_ACTIVE);+ if is_capturing {+ crate::bail!(+ "to_cpu_storage during CUDA graph capture: host/device copies are not \+ permitted while capturing"+ );+ }+ }+ match &self.slice {+ CudaStorageSlice::U8(slice) => {+ let cpu_storage = slice.stream().clone_dtoh(slice).w()?;+ Ok(CpuStorage::U8(cpu_storage))+ }+ CudaStorageSlice::U32(slice) => {+ let cpu_storage = slice.stream().clone_dtoh(slice).w()?;+ Ok(CpuStorage::U32(cpu_storage))+ }+ CudaStorageSlice::I16(slice) => {+ let cpu_storage = slice.stream().clone_dtoh(slice).w()?;+ Ok(CpuStorage::I16(cpu_storage))+ }+ CudaStorageSlice::I32(slice) => {+ let cpu_storage = slice.stream().clone_dtoh(slice).w()?;+ Ok(CpuStorage::I32(cpu_storage))+ }+ CudaStorageSlice::I64(slice) => {+ let cpu_storage = slice.stream().clone_dtoh(slice).w()?;+ Ok(CpuStorage::I64(cpu_storage))+ }+ CudaStorageSlice::BF16(slice) => {+ let cpu_storage = slice.stream().clone_dtoh(slice).w()?;+ Ok(CpuStorage::BF16(cpu_storage))+ }+ CudaStorageSlice::F16(slice) => {+ let cpu_storage = slice.stream().clone_dtoh(slice).w()?;+ Ok(CpuStorage::F16(cpu_storage))+ }+ CudaStorageSlice::F32(slice) => {+ let cpu_storage = slice.stream().clone_dtoh(slice).w()?;+ Ok(CpuStorage::F32(cpu_storage))+ }+ CudaStorageSlice::F64(slice) => {+ let cpu_storage = slice.stream().clone_dtoh(slice).w()?;+ Ok(CpuStorage::F64(cpu_storage))+ }+ CudaStorageSlice::F8E4M3(slice) => {+ let cpu_storage = slice.stream().clone_dtoh(slice).w()?;+ Ok(CpuStorage::F8E4M3(cpu_storage))+ }+ CudaStorageSlice::F4(_)+ | CudaStorageSlice::F6E2M3(_)+ | CudaStorageSlice::F6E3M2(_)+ | CudaStorageSlice::F8E8M0(_) => Err(CudaError::UnsupportedDtype {+ dtype: self.dtype(),+ op: "to_cpu_storage",+ }+ .into()),+ }+ }++ fn where_cond(+ &self,+ layout: &Layout,+ t: &Self,+ t_l: &Layout,+ f: &Self,+ f_l: &Layout,+ ) -> Result<Self> {+ let device = self.device().clone();+ let slice = WhereCond(self, layout).map(&t.slice, t_l, &f.slice, f_l, &device)?;+ Ok(Self { slice, device })+ }++ #[cfg(not(feature = "cudnn"))]+ fn conv1d(+ &self,+ l: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &crate::conv::ParamsConv1D,+ ) -> Result<Self> {+ const USE_IM2COL_CONV1D: bool = true;++ let device = self.device().clone();+ if !USE_IM2COL_CONV1D {+ let slice = Conv1D(params).map(&self.slice, l, &kernel.slice, kernel_l, &device)?;+ return Ok(Self { slice, device });+ }++ let col = Im2Col1D {+ l_k: params.k_size,+ stride: params.stride,+ dilation: params.dilation,+ padding: params.padding,+ }+ .map(&self.slice, &device, l)?;+ let col = Self { slice: col, device };+ let l_out = params.l_out();+ let b = params.b_size;+ let n = params.c_out;+ let k = params.k_size * params.c_in;+ let m = l_out;+ let col_l = Layout::contiguous((b * m, k));+ let res = if kernel_l.is_contiguous() {+ let kernel_l =+ Layout::contiguous_with_offset((n, k), kernel_l.start_offset()).transpose(0, 1)?;+ col.matmul(kernel, (1, b * m, n, k), &col_l, &kernel_l)?+ } else {+ // Make the kernel contiguous if not already the case.+ let mut kernel_c = unsafe {+ self.device()+ .alloc_uninit(kernel_l.shape(), kernel.dtype())?+ };+ kernel.copy_strided_src(&mut kernel_c, 0, kernel_l)?;+ let kernel_l =+ Layout::contiguous_with_offset((n, k), kernel_l.start_offset()).transpose(0, 1)?;+ col.matmul(kernel, (1, b * m, n, k), &col_l, &kernel_l)?+ };+ let res_l = Layout::contiguous((b, l_out, n)).transpose(1, 2)?;+ let mut res_t = unsafe { self.device().alloc_uninit(res_l.shape(), res.dtype())? };+ res.copy_strided_src(&mut res_t, 0, &res_l)?;+ Ok(res_t)+ }++ #[cfg(feature = "cudnn")]+ fn conv1d(+ &self,+ inp_l: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &crate::conv::ParamsConv1D,+ ) -> Result<Self> {+ let device = self.device().clone();+ if !kernel_l.is_contiguous() {+ let slice = Conv1D(params).map(&self.slice, inp_l, &kernel.slice, kernel_l, &device)?;+ return Ok(Self { slice, device });+ }+ let l_out = params.l_out();+ let dst_el = params.c_out * l_out * params.b_size;+ let slice = match (&self.slice, &kernel.slice) {+ (S::U8(inp), S::U8(k)) => {+ let inp = &inp.slice(inp_l.start_offset()..);+ let k = &k.slice(kernel_l.start_offset()..);+ let mut out = unsafe { device.alloc::<u8>(dst_el)? };+ crate::cudnn::launch_conv1d::<u8, u8>(inp, inp_l, k, &mut out, params, &device)+ .map_err(crate::Error::wrap)?;+ S::U8(out)+ }+ (S::BF16(inp), S::BF16(k)) => {+ let inp = &inp.slice(inp_l.start_offset()..);+ let k = &k.slice(kernel_l.start_offset()..);+ let mut out = unsafe { device.alloc::<bf16>(dst_el)? };+ // Only PSEUDO_BFLOAT16_CONFIG is supported in cudnn, there is no "true bfloat16"+ // version.+ // https://docs.nvidia.com/deeplearning/cudnn/latest/api/cudnn-cnn-library.html#id88+ crate::cudnn::launch_conv1d::<bf16, f32>(inp, inp_l, k, &mut out, params, &device)+ .map_err(crate::Error::wrap)?;+ S::BF16(out)+ }+ (S::F16(inp), S::F16(k)) => {+ let inp = &inp.slice(inp_l.start_offset()..);+ let k = &k.slice(kernel_l.start_offset()..);+ let mut out = unsafe { device.alloc::<f16>(dst_el)? };+ crate::cudnn::launch_conv1d::<f16, f16>(inp, inp_l, k, &mut out, params, &device)+ .map_err(crate::Error::wrap)?;+ S::F16(out)+ }+ (S::F32(inp), S::F32(k)) => {+ let inp = &inp.slice(inp_l.start_offset()..);+ let k = &k.slice(kernel_l.start_offset()..);+ let mut out = unsafe { device.alloc::<f32>(dst_el)? };+ crate::cudnn::launch_conv1d::<f32, f32>(inp, inp_l, k, &mut out, params, &device)+ .map_err(crate::Error::wrap)?;+ S::F32(out)+ }+ (S::F64(inp), S::F64(k)) => {+ let inp = &inp.slice(inp_l.start_offset()..);+ let k = &k.slice(kernel_l.start_offset()..);+ let mut out = unsafe { device.alloc::<f64>(dst_el)? };+ crate::cudnn::launch_conv1d::<f64, f64>(inp, inp_l, k, &mut out, params, &device)+ .map_err(crate::Error::wrap)?;+ S::F64(out)+ }+ (S::U32(_), S::U32(_)) => Err(CudaError::InternalError("conv1d does not support u32"))?,+ (S::I16(_), S::I16(_)) => Err(CudaError::InternalError("conv1d does not support i16"))?,+ (S::I32(_), S::I32(_)) => Err(CudaError::InternalError("conv1d does not support i32"))?,+ (S::I64(_), S::I64(_)) => Err(CudaError::InternalError("conv1d does not support i64"))?,+ (S::F8E4M3(_), S::F8E4M3(_)) => {+ Err(CudaError::InternalError("conv1d does not support f8e4m3"))?+ }+ _ => Err(CudaError::InternalError("dtype mismatch in conv1d"))?,+ };+ Ok(Self { slice, device })+ }++ fn conv_transpose1d(+ &self,+ l: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &crate::conv::ParamsConvTranspose1D,+ ) -> Result<Self> {+ const USE_COL2IM_CONV1D_TR: bool = true;++ let device = self.device().clone();+ let can_use_col2im = kernel_l.is_contiguous()+ && params.dilation == 1+ && params.padding == 0+ && params.output_padding == 0;+ let slice = if USE_COL2IM_CONV1D_TR && can_use_col2im {+ let (b_size, c_in, l_in) = l.shape().dims3()?;+ let (c_in2, c_out, k_size) = kernel_l.shape().dims3()?;+ if !kernel_l.is_contiguous() {+ crate::bail!(+ "convtr1d: the second argument (kernel) has to be contiguous {kernel_l:?}"+ )+ }+ if c_in != c_in2 {+ crate::bail!(+ "convtr1d: shape mismatch on c_in {:?} {:?}",+ l.shape(),+ kernel_l.shape()+ )+ }+ let col = {+ // This merges the last two dimensions of the kernel together.+ let kernel_l_mm = Layout::new(+ (b_size, c_in, k_size * c_out).into(),+ vec![0, k_size * c_out, 1],+ kernel_l.start_offset(),+ );+ self.matmul(+ kernel,+ (+ b_size,+ /* m */ l_in,+ /* n */ c_out * k_size,+ /* k */ c_in,+ ),+ &l.transpose(1, 2)?,+ &kernel_l_mm,+ )?+ };+ let col_l = Layout::contiguous((b_size, l_in, c_out, k_size));+ Col2Im1D {+ stride: params.stride,+ }+ .map(&col.slice, &device, &col_l)?+ } else {+ ConvTranspose1D(params).map(&self.slice, l, &kernel.slice, kernel_l, &device)?+ };+ Ok(Self { slice, device })+ }++ #[cfg(not(feature = "cudnn"))]+ fn conv2d(+ &self,+ l: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &crate::conv::ParamsConv2D,+ ) -> Result<Self> {+ const USE_IM2COL_CONV2D: bool = true;++ let device = self.device().clone();+ if !USE_IM2COL_CONV2D {+ let slice = Conv2D(params).map(&self.slice, l, &kernel.slice, kernel_l, &device)?;+ return Ok(Self { slice, device });+ }++ let col = Im2Col {+ h_k: params.k_h,+ w_k: params.k_w,+ stride: params.stride,+ dilation: params.dilation,+ padding: params.padding,+ }+ .map(&self.slice, &device, l)?;+ let col = Self { slice: col, device };+ let h_out = params.out_h();+ let w_out = params.out_w();+ let b = params.b_size;+ let n = params.c_out;+ let k = params.k_h * params.k_w * params.c_in;+ let m = h_out * w_out;+ let col_l = Layout::contiguous((b * m, k));+ let res = if kernel_l.is_contiguous() {+ let kernel_l =+ Layout::contiguous_with_offset((n, k), kernel_l.start_offset()).transpose(0, 1)?;+ col.matmul(kernel, (1, b * m, n, k), &col_l, &kernel_l)?+ } else {+ // Make the kernel contiguous if not already the case. copy_strided_src writes the+ // materialized kernel starting at offset 0, so the matmul layout must use offset 0,+ // not the original (strided) kernel's start offset.+ let mut kernel_c = unsafe {+ self.device()+ .alloc_uninit(kernel_l.shape(), kernel.dtype())?+ };+ kernel.copy_strided_src(&mut kernel_c, 0, kernel_l)?;+ let kernel_l = Layout::contiguous_with_offset((n, k), 0).transpose(0, 1)?;+ col.matmul(&kernel_c, (1, b * m, n, k), &col_l, &kernel_l)?+ };+ let res_l = Layout::contiguous((b, h_out, w_out, n))+ .transpose(1, 2)?+ .transpose(1, 3)?;+ let mut res_t = unsafe { self.device().alloc_uninit(res_l.shape(), res.dtype())? };+ res.copy_strided_src(&mut res_t, 0, &res_l)?;+ Ok(res_t)+ }++ #[cfg(feature = "cudnn")]+ fn conv2d(+ &self,+ inp_l: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &crate::conv::ParamsConv2D,+ ) -> Result<Self> {+ let device = self.device().clone();+ if !kernel_l.is_contiguous() {+ let slice = Conv2D(params).map(&self.slice, inp_l, &kernel.slice, kernel_l, &device)?;+ return Ok(Self { slice, device });+ }+ let (out_w, out_h) = (params.out_w(), params.out_h());+ let dst_el = params.c_out * out_w * out_h * params.b_size;+ let slice = match (&self.slice, &kernel.slice) {+ (S::U8(inp), S::U8(k)) => {+ let inp = &inp.slice(inp_l.start_offset()..);+ let k = &k.slice(kernel_l.start_offset()..);+ let mut out = unsafe { device.alloc::<u8>(dst_el)? };+ crate::cudnn::launch_conv2d::<u8, u8>(inp, inp_l, k, &mut out, params, &device)+ .map_err(crate::Error::wrap)?;+ S::U8(out)+ }+ (S::BF16(inp), S::BF16(k)) => {+ let inp = &inp.slice(inp_l.start_offset()..);+ let k = &k.slice(kernel_l.start_offset()..);+ let mut out = unsafe { device.alloc::<bf16>(dst_el)? };+ // Only PSEUDO_BFLOAT16_CONFIG is supported in cudnn, there is no "true bfloat16"+ // version.+ // https://docs.nvidia.com/deeplearning/cudnn/latest/api/cudnn-cnn-library.html#id88+ crate::cudnn::launch_conv2d::<bf16, f32>(inp, inp_l, k, &mut out, params, &device)+ .map_err(crate::Error::wrap)?;+ S::BF16(out)+ }+ (S::F16(inp), S::F16(k)) => {+ let inp = &inp.slice(inp_l.start_offset()..);+ let k = &k.slice(kernel_l.start_offset()..);+ let mut out = unsafe { device.alloc::<f16>(dst_el)? };+ crate::cudnn::launch_conv2d::<f16, f16>(inp, inp_l, k, &mut out, params, &device)+ .map_err(crate::Error::wrap)?;+ S::F16(out)+ }+ (S::F32(inp), S::F32(k)) => {+ let inp = &inp.slice(inp_l.start_offset()..);+ let k = &k.slice(kernel_l.start_offset()..);+ let mut out = unsafe { device.alloc::<f32>(dst_el)? };+ crate::cudnn::launch_conv2d::<f32, f32>(inp, inp_l, k, &mut out, params, &device)+ .map_err(crate::Error::wrap)?;+ S::F32(out)+ }+ (S::F64(inp), S::F64(k)) => {+ let inp = &inp.slice(inp_l.start_offset()..);+ let k = &k.slice(kernel_l.start_offset()..);+ let mut out = unsafe { device.alloc::<f64>(dst_el)? };+ crate::cudnn::launch_conv2d::<f64, f64>(inp, inp_l, k, &mut out, params, &device)+ .map_err(crate::Error::wrap)?;+ S::F64(out)+ }+ (S::U32(_), S::U32(_)) => Err(CudaError::InternalError("conv2d does not support u32"))?,+ (S::I16(_), S::I16(_)) => Err(CudaError::InternalError("conv2d does not support i16"))?,+ (S::I32(_), S::I32(_)) => Err(CudaError::InternalError("conv2d does not support i32"))?,+ (S::I64(_), S::I64(_)) => Err(CudaError::InternalError("conv2d does not support i64"))?,+ (S::F8E4M3(_), S::F8E4M3(_)) => {+ Err(CudaError::InternalError("conv2d does not support f8e4m3"))?+ }+ _ => Err(CudaError::InternalError("dtype mismatch in conv2d"))?,+ };+ Ok(Self { slice, device })+ }++ fn conv_transpose2d(+ &self,+ l: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &crate::conv::ParamsConvTranspose2D,+ ) -> Result<Self> {+ let device = self.device().clone();+ let slice =+ ConvTranspose2D(params).map(&self.slice, l, &kernel.slice, kernel_l, &device)?;+ Ok(Self { slice, device })+ }++ fn avg_pool2d(&self, l: &Layout, k: (usize, usize), stride: (usize, usize)) -> Result<Self> {+ let device = self.device().clone();+ let slice = Pool2D {+ w_k: k.0,+ h_k: k.1,+ w_stride: stride.0,+ h_stride: stride.1,+ op: PoolOp::Avg,+ }+ .map(&self.slice, &device, l)?;+ Ok(Self { slice, device })+ }++ fn max_pool2d(&self, l: &Layout, k: (usize, usize), stride: (usize, usize)) -> Result<Self> {+ let device = self.device().clone();+ let slice = Pool2D {+ w_k: k.0,+ h_k: k.1,+ w_stride: stride.0,+ h_stride: stride.1,+ op: PoolOp::Max,+ }+ .map(&self.slice, &device, l)?;+ Ok(Self { slice, device })+ }++ fn upsample_nearest1d(&self, _: &Layout, _out_sz: usize) -> Result<Self> {+ crate::bail!("upsample-nearest1d is not supported on cuda")+ }++ fn upsample_nearest2d(&self, l: &Layout, out_w: usize, out_h: usize) -> Result<Self> {+ let device = self.device().clone();+ let slice = UpsampleNearest2D(out_w, out_h).map(&self.slice, &device, l)?;+ Ok(Self { slice, device })+ }++ fn upsample_bilinear2d(+ &self,+ l: &Layout,+ out_h: usize,+ out_w: usize,+ align_corners: bool,+ scale_h: Option<f64>,+ scale_w: Option<f64>,+ ) -> Result<Self> {+ let device = self.device().clone();+ let slice = UpsampleBilinear2D {+ out_w,+ out_h,+ align_corners,+ scale_h_factor: scale_h,+ scale_w_factor: scale_w,+ }+ .map(&self.slice, &device, l)?;+ Ok(Self { slice, device })+ }++ fn index_select(&self, ids: &Self, l: &Layout, ids_l: &Layout, dim: usize) -> Result<Self> {+ let device = self.device().clone();+ let slice = IndexSelect(ids, ids_l, dim).map(&self.slice, &device, l)?;+ Ok(Self { slice, device })+ }+ fn gather(&self, l: &Layout, ids: &Self, ids_l: &Layout, dim: usize) -> Result<Self> {+ let device = self.device().clone();+ let slice = Gather(ids, ids_l, dim).map(&self.slice, &device, l)?;+ Ok(Self { slice, device })+ }+ fn scatter_set(+ &mut self,+ l: &Layout,+ ids: &Self,+ ids_l: &Layout,+ src: &Self,+ src_l: &Layout,+ dim: usize,+ ) -> Result<()> {+ let device = self.device().clone();+ Scatter(ids, ids_l, dim).map(&mut self.slice, l, &src.slice, src_l, &device)+ }+ fn scatter_add_set(+ &mut self,+ l: &Layout,+ ids: &Self,+ ids_l: &Layout,+ src: &Self,+ src_l: &Layout,+ dim: usize,+ ) -> Result<()> {+ let device = self.device().clone();+ ScatterAdd(ids, ids_l, dim).map(&mut self.slice, l, &src.slice, src_l, &device)+ }+ fn index_add(+ &self,+ l: &Layout,+ ids: &Self,+ ids_l: &Layout,+ src: &Self,+ src_l: &Layout,+ dim: usize,+ ) -> Result<Self> {+ let device = self.device().clone();+ let mut acc = unsafe { device.alloc_uninit(l.shape(), self.dtype())? };+ self.copy_strided_src(&mut acc, 0, l)?;+ IndexAdd(ids, ids_l, dim).map(&mut acc.slice, l, &src.slice, src_l, &device)?;+ Ok(acc)+ }++ fn matmul(+ &self,+ rhs: &Self,+ (b, m, n, k): (usize, usize, usize, usize),+ lhs_l: &Layout,+ rhs_l: &Layout,+ ) -> Result<Self> {+ let elem_count = b * m * n;+ let dev = &self.device;+ let slice = match (&self.slice, &rhs.slice) {+ (CudaStorageSlice::BF16(lhs), CudaStorageSlice::BF16(rhs)) => {+ let lhs = &lhs.slice(lhs_l.start_offset()..);+ let rhs = &rhs.slice(rhs_l.start_offset()..);+ let cfg = gemm_config(bf16::ONE, bf16::ZERO, (b, m, n, k), lhs_l, rhs_l)?;+ let mut out = unsafe { dev.alloc::<bf16>(elem_count)? };+ unsafe { gemm_strided_batched_bf16(&self.device.blas, cfg, rhs, lhs, &mut out) }+ .w()?;+ CudaStorageSlice::BF16(out)+ }+ (CudaStorageSlice::F16(lhs), CudaStorageSlice::F16(rhs)) => {+ let lhs = &lhs.slice(lhs_l.start_offset()..);+ let rhs = &rhs.slice(rhs_l.start_offset()..);+ let cfg = gemm_config(f16::ONE, f16::ZERO, (b, m, n, k), lhs_l, rhs_l)?;+ let mut out = unsafe { dev.alloc::<f16>(elem_count)? };+ unsafe { gemm_strided_batched_f16(&self.device.blas, cfg, rhs, lhs, &mut out) }+ .w()?;+ CudaStorageSlice::F16(out)+ }+ (CudaStorageSlice::F32(lhs), CudaStorageSlice::F32(rhs)) => {+ let lhs = &lhs.slice(lhs_l.start_offset()..);+ let rhs = &rhs.slice(rhs_l.start_offset()..);+ let cfg = gemm_config(1., 0., (b, m, n, k), lhs_l, rhs_l)?;+ let mut out = unsafe { dev.alloc::<f32>(elem_count)? };+ unsafe { gemm_strided_batched_f32(&self.device.blas, cfg, rhs, lhs, &mut out) }+ .w()?;+ CudaStorageSlice::F32(out)+ }+ (CudaStorageSlice::F64(lhs), CudaStorageSlice::F64(rhs)) => {+ let lhs = &lhs.slice(lhs_l.start_offset()..);+ let rhs = &rhs.slice(rhs_l.start_offset()..);+ let cfg = gemm_config(1., 0., (b, m, n, k), lhs_l, rhs_l)?;+ let mut out = unsafe { dev.alloc::<f64>(elem_count)? };+ unsafe {+ self.device+ .blas+ .gemm_strided_batched(cfg, rhs, lhs, &mut out)+ }+ .w()?;+ CudaStorageSlice::F64(out)+ }+ _ => Err(CudaError::InternalError("dtype mismatch in matmul op"))?,+ };+ let device = dev.clone();+ Ok(Self { slice, device })+ }++ fn copy2d(+ &self,+ dst: &mut Self,+ d1: usize,+ d2: usize,+ src_s: usize,+ dst_s: usize,+ src_o: usize,+ dst_o: usize,+ ) -> Result<()> {+ let dev = &self.device;+ let d1 = d1 as u32;+ let d2 = d2 as u32;+ // Nothing to copy so we exit early to avoid launching a kernel and some potential invalid+ // argument with a null pointer.+ if d1 == 0 || d2 == 0 {+ return Ok(());+ }+ let dst_s = dst_s as u32;+ let src_s = src_s as u32;+ let ((src, _guard_src), (dst, _guard_dst), kname) = match (&self.slice, &mut dst.slice) {+ (S::U8(s), S::U8(d)) => (slice_ptr(s, src_o), slice_ptr(d, dst_o), "copy2d_u8"),+ (S::U32(s), S::U32(d)) => (slice_ptr(s, src_o), slice_ptr(d, dst_o), "copy2d_u32"),+ (S::I16(s), S::I16(d)) => (slice_ptr(s, src_o), slice_ptr(d, dst_o), "copy2d_i16"),+ (S::I32(s), S::I32(d)) => (slice_ptr(s, src_o), slice_ptr(d, dst_o), "copy2d_i32"),+ (S::I64(s), S::I64(d)) => (slice_ptr(s, src_o), slice_ptr(d, dst_o), "copy2d_i64"),+ (S::BF16(s), S::BF16(d)) => (slice_ptr(s, src_o), slice_ptr(d, dst_o), "copy2d_bf16"),+ (S::F16(s), S::F16(d)) => (slice_ptr(s, src_o), slice_ptr(d, dst_o), "copy2d_f16"),+ (S::F32(s), S::F32(d)) => (slice_ptr(s, src_o), slice_ptr(d, dst_o), "copy2d_f32"),+ (S::F64(s), S::F64(d)) => (slice_ptr(s, src_o), slice_ptr(d, dst_o), "copy2d_f64"),+ (S::F8E4M3(s), S::F8E4M3(d)) => (slice_ptr(s, src_o), slice_ptr(d, dst_o), "copy2d_u8"),+ (S::F8E8M0(s), S::F8E8M0(d)) => (slice_ptr(s, src_o), slice_ptr(d, dst_o), "copy2d_u8"),+ _ => Err(CudaError::InternalError("dtype mismatch in copy2d"))?,+ };+ let func = dev.get_or_load_func(kname, &kernels::FILL)?;+ let cfg = LaunchConfig::for_num_elems(d1 * d2);+ let mut builder = func.builder();+ barg!(builder, src);+ barg!(builder, dst);+ barg!(builder, d1);+ barg!(builder, d2);+ builder.arg(&src_s);+ builder.arg(&dst_s);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ Ok(())+ }++ fn copy_strided_src(&self, dst: &mut Self, dst_offset: usize, src_l: &Layout) -> Result<()> {+ let src_shape = src_l.shape();+ let dims = src_shape.dims();+ let el_count = src_shape.elem_count();+ if el_count == 0 {+ return Ok(());+ }+ let cfg = LaunchConfig::for_num_elems(el_count as u32);+ let dev = &self.device;+ let ds = SlicePtrOrNull::params_from_layout(dev, src_l)?;+ match (&self.slice, &mut dst.slice) {+ (CudaStorageSlice::BF16(src), CudaStorageSlice::BF16(dst)) => {+ let (src, mut dst) = slice_src_and_dst(src, src_l, dst, dst_offset);+ if src_l.is_contiguous() {+ dev.memcpy_dtod(&src, &mut dst)?+ } else {+ let func = dev.get_or_load_func("ucopy_bf16", &kernels::UNARY)?;+ let mut builder = func.builder();+ barg!(builder, el_count);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ builder.arg(&src);+ builder.arg(&mut dst);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ }+ }+ (CudaStorageSlice::F16(src), CudaStorageSlice::F16(dst)) => {+ let (src, mut dst) = slice_src_and_dst(src, src_l, dst, dst_offset);+ if src_l.is_contiguous() {+ dev.memcpy_dtod(&src, &mut dst)?+ } else {+ let func = dev.get_or_load_func("ucopy_f16", &kernels::UNARY)?;+ let mut builder = func.builder();+ barg!(builder, el_count);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ builder.arg(&src);+ builder.arg(&mut dst);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ }+ }+ (CudaStorageSlice::F32(src), CudaStorageSlice::F32(dst)) => {+ let (src, mut dst) = slice_src_and_dst(src, src_l, dst, dst_offset);+ if src_l.is_contiguous() {+ dev.memcpy_dtod(&src, &mut dst)?+ } else {+ let func = dev.get_or_load_func("ucopy_f32", &kernels::UNARY)?;+ let mut builder = func.builder();+ barg!(builder, el_count);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ builder.arg(&src);+ builder.arg(&mut dst);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ }+ }+ (CudaStorageSlice::U8(src), CudaStorageSlice::U8(dst)) => {+ let (src, mut dst) = slice_src_and_dst(src, src_l, dst, dst_offset);+ if src_l.is_contiguous() {+ dev.memcpy_dtod(&src, &mut dst)?+ } else {+ let func = dev.get_or_load_func("ucopy_u8", &kernels::UNARY)?;+ let mut builder = func.builder();+ barg!(builder, el_count);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ builder.arg(&src);+ builder.arg(&mut dst);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ }+ }+ (CudaStorageSlice::U32(src), CudaStorageSlice::U32(dst)) => {+ let (src, mut dst) = slice_src_and_dst(src, src_l, dst, dst_offset);+ if src_l.is_contiguous() {+ dev.memcpy_dtod(&src, &mut dst)?+ } else {+ let func = dev.get_or_load_func("ucopy_u32", &kernels::UNARY)?;+ let mut builder = func.builder();+ barg!(builder, el_count);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ builder.arg(&src);+ builder.arg(&mut dst);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ }+ }+ (CudaStorageSlice::I16(src), CudaStorageSlice::I16(dst)) => {+ let (src, mut dst) = slice_src_and_dst(src, src_l, dst, dst_offset);+ if src_l.is_contiguous() {+ dev.memcpy_dtod(&src, &mut dst)?+ } else {+ let func = dev.get_or_load_func("ucopy_i16", &kernels::UNARY)?;+ let mut builder = func.builder();+ barg!(builder, el_count);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ builder.arg(&src);+ builder.arg(&mut dst);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ }+ }+ (CudaStorageSlice::I32(src), CudaStorageSlice::I32(dst)) => {+ let (src, mut dst) = slice_src_and_dst(src, src_l, dst, dst_offset);+ if src_l.is_contiguous() {+ dev.memcpy_dtod(&src, &mut dst)?+ } else {+ let func = dev.get_or_load_func("ucopy_i32", &kernels::UNARY)?;+ let mut builder = func.builder();+ barg!(builder, el_count);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ builder.arg(&src);+ builder.arg(&mut dst);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ }+ }+ (CudaStorageSlice::I64(src), CudaStorageSlice::I64(dst)) => {+ let (src, mut dst) = slice_src_and_dst(src, src_l, dst, dst_offset);+ if src_l.is_contiguous() {+ dev.memcpy_dtod(&src, &mut dst)?+ } else {+ let func = dev.get_or_load_func("ucopy_i64", &kernels::UNARY)?;+ let mut builder = func.builder();+ barg!(builder, el_count);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ builder.arg(&src);+ builder.arg(&mut dst);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ }+ }+ (CudaStorageSlice::F64(src), CudaStorageSlice::F64(dst)) => {+ let (src, mut dst) = slice_src_and_dst(src, src_l, dst, dst_offset);+ if src_l.is_contiguous() {+ dev.memcpy_dtod(&src, &mut dst)?+ } else {+ let func = dev.get_or_load_func("ucopy_f64", &kernels::UNARY)?;+ let mut builder = func.builder();+ barg!(builder, el_count);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ builder.arg(&src);+ builder.arg(&mut dst);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ }+ }+ (CudaStorageSlice::F8E4M3(src), CudaStorageSlice::F8E4M3(dst)) => {+ let (src, mut dst) = slice_src_and_dst(src, src_l, dst, dst_offset);+ if src_l.is_contiguous() {+ dev.memcpy_dtod(&src, &mut dst)?+ } else {+ let func = dev.get_or_load_func("ucopy_f8e4m3", &kernels::UNARY)?;+ let mut builder = func.builder();+ barg!(builder, el_count);+ barg!(builder, dims.len());+ ds.builder_arg(&mut builder);+ builder.arg(&src);+ builder.arg(&mut dst);+ // SAFETY: ffi.+ unsafe { builder.launch(cfg) }.w()?;+ }+ }+ _ => Err(CudaError::InternalError(+ "dtype mismatch in copy_strided op",+ ))?,+ }+ Ok(())+ }+}++// Default for the reduced precision setting is false, similar to pytorch.+// https://github.com/pytorch/pytorch/issues/123157+static MM_F16_REDUCED_PRECISION: std::sync::atomic::AtomicBool =+ std::sync::atomic::AtomicBool::new(false);+static MM_BF16_REDUCED_PRECISION: std::sync::atomic::AtomicBool =+ std::sync::atomic::AtomicBool::new(false);+static MM_F32_REDUCED_PRECISION: std::sync::atomic::AtomicBool =+ std::sync::atomic::AtomicBool::new(false);++/// This bool controls whether reduced precision reductions (e.g., with tf32 accumulation type) are+/// allowed with f32 GEMMs.+pub fn gemm_reduced_precision_f32() -> bool {+ MM_F32_REDUCED_PRECISION.load(std::sync::atomic::Ordering::Relaxed)+}++/// This bool controls whether reduced precision reductions (e.g., with tf32 accumulation type) are+/// allowed with f32 GEMMs.+pub fn set_gemm_reduced_precision_f32(b: bool) {+ MM_F32_REDUCED_PRECISION.store(b, std::sync::atomic::Ordering::Relaxed)+}++/// This bool controls whether reduced precision reductions (e.g., with fp16 accumulation type) are+/// allowed with f16 GEMMs.+pub fn gemm_reduced_precision_f16() -> bool {+ MM_F16_REDUCED_PRECISION.load(std::sync::atomic::Ordering::Relaxed)+}++/// This bool controls whether reduced precision reductions (e.g., with fp16 accumulation type) are+/// allowed with f16 GEMMs.+pub fn set_gemm_reduced_precision_f16(b: bool) {+ MM_F16_REDUCED_PRECISION.store(b, std::sync::atomic::Ordering::Relaxed)+}++/// This bool controls whether reduced precision reductions (e.g., with fp16 accumulation type) are+/// allowed with bf16 GEMMs.+pub fn gemm_reduced_precision_bf16() -> bool {+ MM_BF16_REDUCED_PRECISION.load(std::sync::atomic::Ordering::Relaxed)+}++/// This bool controls whether reduced precision reductions (e.g., with fp16 accumulation type) are+/// allowed with bf16 GEMMs.+pub fn set_gemm_reduced_precision_bf16(b: bool) {+ MM_BF16_REDUCED_PRECISION.store(b, std::sync::atomic::Ordering::Relaxed)+}++unsafe fn gemm_strided_batched_f32(+ cublas: &cudarc::cublas::CudaBlas,+ cfg: StridedBatchedConfig<f32>,+ a: &cudarc::driver::CudaView<f32>,+ b: &cudarc::driver::CudaView<f32>,+ c: &mut CudaSlice<f32>,+) -> std::result::Result<(), cudarc::cublas::result::CublasError> {+ use cudarc::cublas::sys;+ use cudarc::driver::DevicePtrMut;++ let compute_type = if gemm_reduced_precision_f32() {+ sys::cublasComputeType_t::CUBLAS_COMPUTE_32F_FAST_TF32+ } else {+ sys::cublasComputeType_t::CUBLAS_COMPUTE_32F+ };+ let alpha = &cfg.gemm.alpha as *const f32 as *const _;+ let beta = &cfg.gemm.beta as *const f32 as *const _;++ let stream = c.stream().clone();+ let (a, _guard_a) = a.device_ptr(&stream);+ let (b, _guard_b) = b.device_ptr(&stream);+ let (c, _guard_c) = c.device_ptr_mut(&stream);++ cudarc::cublas::result::gemm_strided_batched_ex(+ *cublas.handle(),+ cfg.gemm.transa,+ cfg.gemm.transb,+ cfg.gemm.m,+ cfg.gemm.n,+ cfg.gemm.k,+ alpha,+ a as *const _,+ sys::cudaDataType_t::CUDA_R_32F,+ cfg.gemm.lda,+ cfg.stride_a,+ b as *const _,+ sys::cudaDataType_t::CUDA_R_32F,+ cfg.gemm.ldb,+ cfg.stride_b,+ beta,+ c as *mut _,+ sys::cudaDataType_t::CUDA_R_32F,+ cfg.gemm.ldc,+ cfg.stride_c,+ cfg.batch_size,+ compute_type,+ sys::cublasGemmAlgo_t::CUBLAS_GEMM_DEFAULT_TENSOR_OP,+ )+}++unsafe fn gemm_strided_batched_f16(+ cublas: &cudarc::cublas::CudaBlas,+ cfg: StridedBatchedConfig<f16>,+ a: &cudarc::driver::CudaView<f16>,+ b: &cudarc::driver::CudaView<f16>,+ c: &mut CudaSlice<f16>,+) -> std::result::Result<(), cudarc::cublas::result::CublasError> {+ use cudarc::cublas::sys;+ use cudarc::driver::DevicePtrMut;++ let alpha = cfg.gemm.alpha;+ let beta = cfg.gemm.beta;+ let alpha_f32: f32 = cfg.gemm.alpha.to_f32();+ let beta_f32: f32 = cfg.gemm.beta.to_f32();+ let (compute_type, alpha, beta) = if gemm_reduced_precision_f16() {+ (+ sys::cublasComputeType_t::CUBLAS_COMPUTE_16F,+ (&alpha) as *const f16 as *const _,+ (&beta) as *const f16 as *const _,+ )+ } else {+ (+ sys::cublasComputeType_t::CUBLAS_COMPUTE_32F,+ (&alpha_f32) as *const f32 as *const _,+ (&beta_f32) as *const f32 as *const _,+ )+ };++ let stream = c.stream().clone();+ let (a, _guard_a) = a.device_ptr(&stream);+ let (b, _guard_b) = b.device_ptr(&stream);+ let (c, _guard_c) = c.device_ptr_mut(&stream);+ cudarc::cublas::result::gemm_strided_batched_ex(+ *cublas.handle(),+ cfg.gemm.transa,+ cfg.gemm.transb,+ cfg.gemm.m,+ cfg.gemm.n,+ cfg.gemm.k,+ alpha,+ a as *const _,+ sys::cudaDataType_t::CUDA_R_16F,+ cfg.gemm.lda,+ cfg.stride_a,+ b as *const _,+ sys::cudaDataType_t::CUDA_R_16F,+ cfg.gemm.ldb,+ cfg.stride_b,+ beta,+ c as *mut _,+ sys::cudaDataType_t::CUDA_R_16F,+ cfg.gemm.ldc,+ cfg.stride_c,+ cfg.batch_size,+ compute_type,+ sys::cublasGemmAlgo_t::CUBLAS_GEMM_DEFAULT_TENSOR_OP,+ )+}++unsafe fn gemm_strided_batched_bf16(+ cublas: &cudarc::cublas::CudaBlas,+ cfg: StridedBatchedConfig<bf16>,+ a: &cudarc::driver::CudaView<bf16>,+ b: &cudarc::driver::CudaView<bf16>,+ c: &mut CudaSlice<bf16>,+) -> std::result::Result<(), cudarc::cublas::result::CublasError> {+ use cudarc::cublas::sys;+ use cudarc::driver::DevicePtrMut;++ let alpha_f32: f32 = cfg.gemm.alpha.to_f32();+ let beta_f32: f32 = cfg.gemm.beta.to_f32();+ // The type for alpha and beta depends on the computeType.+ // https://docs.nvidia.com/cuda/cublas/index.html#cublasgemmstridedbatchedex+ let (compute_type, alpha, beta) = if gemm_reduced_precision_bf16() {+ (+ sys::cublasComputeType_t::CUBLAS_COMPUTE_32F_FAST_16BF,+ (&alpha_f32) as *const f32 as *const _,+ (&beta_f32) as *const f32 as *const _,+ )+ } else {+ (+ sys::cublasComputeType_t::CUBLAS_COMPUTE_32F,+ (&alpha_f32) as *const f32 as *const _,+ (&beta_f32) as *const f32 as *const _,+ )+ };++ let stream = c.stream().clone();+ let (a, _guard_a) = a.device_ptr(&stream);+ let (b, _guard_b) = b.device_ptr(&stream);+ let (c, _guard_c) = c.device_ptr_mut(&stream);+ cudarc::cublas::result::gemm_strided_batched_ex(+ *cublas.handle(),+ cfg.gemm.transa,+ cfg.gemm.transb,+ cfg.gemm.m,+ cfg.gemm.n,+ cfg.gemm.k,+ alpha,+ a as *const _,+ sys::cudaDataType_t::CUDA_R_16BF,+ cfg.gemm.lda,+ cfg.stride_a,+ b as *const _,+ sys::cudaDataType_t::CUDA_R_16BF,+ cfg.gemm.ldb,+ cfg.stride_b,+ beta,+ c as *mut _,+ sys::cudaDataType_t::CUDA_R_16BF,+ cfg.gemm.ldc,+ cfg.stride_c,+ cfg.batch_size,+ compute_type,+ sys::cublasGemmAlgo_t::CUBLAS_GEMM_DEFAULT_TENSOR_OP,+ )+}diff --git a/vendor/candle-core/src/cuda_backend/utils.rs b/vendor/candle-core/src/cuda_backend/utils.rsnew file mode 100644index 0000000..10f8876--- /dev/null+++ b/vendor/candle-core/src/cuda_backend/utils.rs@@ -0,0 +1,194 @@+/// Helper functions to plug cuda kernels in candle.+use crate::{Layout, Result, WithDType};+pub use cudarc;+use cudarc::driver::{CudaSlice, DeviceRepr, ValidAsZeroBits};++use super::{CudaDevice, CudaError, WrapErr};++pub type S = super::CudaStorageSlice;++pub trait Map1 {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ src: &CudaSlice<T>,+ dev: &CudaDevice,+ layout: &Layout,+ ) -> Result<CudaSlice<T>>;++ fn map(&self, s: &S, d: &CudaDevice, l: &Layout) -> Result<S> {+ let out = match s {+ S::U8(s) => S::U8(self.f(s, d, l)?),+ S::U32(s) => S::U32(self.f(s, d, l)?),+ S::I16(s) => S::I16(self.f(s, d, l)?),+ S::I32(s) => S::I32(self.f(s, d, l)?),+ S::I64(s) => S::I64(self.f(s, d, l)?),+ S::BF16(s) => S::BF16(self.f(s, d, l)?),+ S::F16(s) => S::F16(self.f(s, d, l)?),+ S::F32(s) => S::F32(self.f(s, d, l)?),+ S::F64(s) => S::F64(self.f(s, d, l)?),+ S::F8E4M3(s) => S::F8E4M3(self.f(s, d, l)?),+ S::F4(_) | S::F6E2M3(_) | S::F6E3M2(_) | S::F8E8M0(_) => {+ crate::bail!("Map1 does not uspport this dtype.");+ }+ };+ Ok(out)+ }+}++pub trait Map2 {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ src1: &CudaSlice<T>,+ layout1: &Layout,+ src2: &CudaSlice<T>,+ layout2: &Layout,+ dev: &CudaDevice,+ ) -> Result<CudaSlice<T>>;++ fn map(&self, s1: &S, l1: &Layout, s2: &S, l2: &Layout, d: &CudaDevice) -> Result<S> {+ let out = match (s1, s2) {+ (S::U8(s1), S::U8(s2)) => S::U8(self.f(s1, l1, s2, l2, d)?),+ (S::U32(s1), S::U32(s2)) => S::U32(self.f(s1, l1, s2, l2, d)?),+ (S::I16(s1), S::I16(s2)) => S::I16(self.f(s1, l1, s2, l2, d)?),+ (S::I32(s1), S::I32(s2)) => S::I32(self.f(s1, l1, s2, l2, d)?),+ (S::I64(s1), S::I64(s2)) => S::I64(self.f(s1, l1, s2, l2, d)?),+ (S::BF16(s1), S::BF16(s2)) => S::BF16(self.f(s1, l1, s2, l2, d)?),+ (S::F16(s1), S::F16(s2)) => S::F16(self.f(s1, l1, s2, l2, d)?),+ (S::F32(s1), S::F32(s2)) => S::F32(self.f(s1, l1, s2, l2, d)?),+ (S::F64(s1), S::F64(s2)) => S::F64(self.f(s1, l1, s2, l2, d)?),+ (S::F8E4M3(s1), S::F8E4M3(s2)) => S::F8E4M3(self.f(s1, l1, s2, l2, d)?),+ _ => Err(CudaError::InternalError("dtype mismatch in binary op"))?,+ };+ Ok(out)+ }+}++pub trait Map3 {+ #[allow(clippy::too_many_arguments)]+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ src1: &CudaSlice<T>,+ layout1: &Layout,+ src2: &CudaSlice<T>,+ layout2: &Layout,+ src3: &CudaSlice<T>,+ layout3: &Layout,+ dev: &CudaDevice,+ ) -> Result<CudaSlice<T>>;++ #[allow(clippy::too_many_arguments)]+ fn map(+ &self,+ s1: &S,+ l1: &Layout,+ s2: &S,+ l2: &Layout,+ s3: &S,+ l3: &Layout,+ d: &CudaDevice,+ ) -> Result<S> {+ let out = match (s1, s2, s3) {+ (S::U8(s1), S::U8(s2), S::U8(s3)) => S::U8(self.f(s1, l1, s2, l2, s3, l3, d)?),+ (S::U32(s1), S::U32(s2), S::U32(s3)) => S::U32(self.f(s1, l1, s2, l2, s3, l3, d)?),+ (S::I64(s1), S::I64(s2), S::I64(s3)) => S::I64(self.f(s1, l1, s2, l2, s3, l3, d)?),+ (S::BF16(s1), S::BF16(s2), S::BF16(s3)) => S::BF16(self.f(s1, l1, s2, l2, s3, l3, d)?),+ (S::F16(s1), S::F16(s2), S::F16(s3)) => S::F16(self.f(s1, l1, s2, l2, s3, l3, d)?),+ (S::F32(s1), S::F32(s2), S::F32(s3)) => S::F32(self.f(s1, l1, s2, l2, s3, l3, d)?),+ (S::F64(s1), S::F64(s2), S::F64(s3)) => S::F64(self.f(s1, l1, s2, l2, s3, l3, d)?),+ (S::F8E4M3(s1), S::F8E4M3(s2), S::F8E4M3(s3)) => {+ S::F8E4M3(self.f(s1, l1, s2, l2, s3, l3, d)?)+ }+ _ => Err(CudaError::InternalError("dtype mismatch in ternary op"))?,+ };+ Ok(out)+ }+}++pub trait Map2InPlace {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ dst: &mut CudaSlice<T>,+ dst_l: &Layout,+ src: &CudaSlice<T>,+ src_l: &Layout,+ dev: &CudaDevice,+ ) -> Result<()>;++ fn map(+ &self,+ dst: &mut S,+ dst_l: &Layout,+ src: &S,+ src_l: &Layout,+ d: &CudaDevice,+ ) -> Result<()> {+ match (dst, src) {+ (S::U8(dst), S::U8(src)) => self.f(dst, dst_l, src, src_l, d),+ (S::U32(dst), S::U32(src)) => self.f(dst, dst_l, src, src_l, d),+ (S::I16(dst), S::I16(src)) => self.f(dst, dst_l, src, src_l, d),+ (S::I32(dst), S::I32(src)) => self.f(dst, dst_l, src, src_l, d),+ (S::I64(dst), S::I64(src)) => self.f(dst, dst_l, src, src_l, d),+ (S::BF16(dst), S::BF16(src)) => self.f(dst, dst_l, src, src_l, d),+ (S::F16(dst), S::F16(src)) => self.f(dst, dst_l, src, src_l, d),+ (S::F32(dst), S::F32(src)) => self.f(dst, dst_l, src, src_l, d),+ (S::F64(dst), S::F64(src)) => self.f(dst, dst_l, src, src_l, d),+ (S::F8E4M3(dst), S::F8E4M3(src)) => self.f(dst, dst_l, src, src_l, d),+ _ => Err(CudaError::InternalError("dtype mismatch in binary op"))?,+ }+ }+}++pub trait Map1Any {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits, W: Fn(CudaSlice<T>) -> S>(+ &self,+ src: &CudaSlice<T>,+ dev: &CudaDevice,+ layout: &Layout,+ wrap: W,+ ) -> Result<S>;++ fn map(&self, s: &S, d: &CudaDevice, l: &Layout) -> Result<S> {+ let out = match s {+ S::U8(s) => self.f(s, d, l, S::U8)?,+ S::U32(s) => self.f(s, d, l, S::U32)?,+ S::I16(s) => self.f(s, d, l, S::I16)?,+ S::I32(s) => self.f(s, d, l, S::I32)?,+ S::I64(s) => self.f(s, d, l, S::I64)?,+ S::BF16(s) => self.f(s, d, l, S::BF16)?,+ S::F16(s) => self.f(s, d, l, S::F16)?,+ S::F32(s) => self.f(s, d, l, S::F32)?,+ S::F64(s) => self.f(s, d, l, S::F64)?,+ S::F8E4M3(s) => self.f(s, d, l, S::F8E4M3)?,+ S::F4(_) | S::F6E2M3(_) | S::F6E3M2(_) | S::F8E8M0(_) => {+ crate::bail!("Map1 does not uspport this dtype.");+ }+ };+ Ok(out)+ }+}++pub trait Map2Any {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits>(+ &self,+ src1: &CudaSlice<T>,+ layout1: &Layout,+ src2: &CudaSlice<T>,+ layout2: &Layout,+ dev: &CudaDevice,+ ) -> Result<S>;++ fn map(&self, s1: &S, l1: &Layout, s2: &S, l2: &Layout, d: &CudaDevice) -> Result<S> {+ let out = match (s1, s2) {+ (S::U8(s1), S::U8(s2)) => self.f(s1, l1, s2, l2, d)?,+ (S::U32(s1), S::U32(s2)) => self.f(s1, l1, s2, l2, d)?,+ (S::I64(s1), S::I64(s2)) => self.f(s1, l1, s2, l2, d)?,+ (S::BF16(s1), S::BF16(s2)) => self.f(s1, l1, s2, l2, d)?,+ (S::F16(s1), S::F16(s2)) => self.f(s1, l1, s2, l2, d)?,+ (S::F32(s1), S::F32(s2)) => self.f(s1, l1, s2, l2, d)?,+ (S::F64(s1), S::F64(s2)) => self.f(s1, l1, s2, l2, d)?,+ (S::F8E4M3(s1), S::F8E4M3(s2)) => self.f(s1, l1, s2, l2, d)?,+ _ => Err(CudaError::InternalError("dtype mismatch in binary op")).w()?,+ };+ Ok(out)+ }+}diff --git a/vendor/candle-core/src/custom_op.rs b/vendor/candle-core/src/custom_op.rsnew file mode 100644index 0000000..a29fbfc--- /dev/null+++ b/vendor/candle-core/src/custom_op.rs@@ -0,0 +1,688 @@+use crate::layout::LayoutRelation;+use crate::op::{BackpropOp, Op};+use crate::tensor::from_storage;+use crate::{bail, CpuStorage, CudaStorage, Layout, MetalStorage, Result, Shape, Storage, Tensor};+use std::sync::Arc;++/// Unary ops that can be defined in user-land.+pub trait CustomOp1 {+ // Box<dyn> does not support const yet, so use a function to get the name.+ fn name(&self) -> &'static str;++ /// The forward pass, as run on a cpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn cpu_fwd(&self, storage: &CpuStorage, layout: &Layout) -> Result<(CpuStorage, Shape)>;++ /// The forward pass, as run on a gpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn cuda_fwd(&self, _storage: &CudaStorage, _layout: &Layout) -> Result<(CudaStorage, Shape)> {+ Err(crate::Error::Cuda(+ format!("no cuda implementation for {}", self.name()).into(),+ ))+ }++ /// The forward pass, as run on a metal gpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn metal_fwd(+ &self,+ _storage: &MetalStorage,+ _layout: &Layout,+ ) -> Result<(MetalStorage, Shape)> {+ Err(crate::Error::Metal(+ format!("no metal implementation for {}", self.name()).into(),+ ))+ }++ /// This function takes as argument the argument `arg` used in the forward pass, the result+ /// produced by the forward operation `res` and the gradient of the result `grad_res`.+ /// The function should return the gradient of the argument.+ fn bwd(&self, _arg: &Tensor, _res: &Tensor, _grad_res: &Tensor) -> Result<Option<Tensor>> {+ Err(crate::Error::BackwardNotSupported { op: self.name() })+ }+}++pub trait CustomOp2 {+ fn name(&self) -> &'static str;++ /// The forward pass, as run on a cpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn cpu_fwd(+ &self,+ s1: &CpuStorage,+ l1: &Layout,+ s2: &CpuStorage,+ l2: &Layout,+ ) -> Result<(CpuStorage, Shape)>;++ /// The forward pass, as run on a gpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn cuda_fwd(+ &self,+ _: &CudaStorage,+ _: &Layout,+ _: &CudaStorage,+ _: &Layout,+ ) -> Result<(CudaStorage, Shape)> {+ Err(crate::Error::Cuda(+ format!("no cuda implementation for {}", self.name()).into(),+ ))+ }++ /// The forward pass, as run on a metal gpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn metal_fwd(+ &self,+ _: &MetalStorage,+ _: &Layout,+ _: &MetalStorage,+ _: &Layout,+ ) -> Result<(MetalStorage, Shape)> {+ Err(crate::Error::Metal(+ format!("no metal implementation for {}", self.name()).into(),+ ))+ }++ fn bwd(+ &self,+ _arg1: &Tensor,+ _arg2: &Tensor,+ _res: &Tensor,+ _grad_res: &Tensor,+ ) -> Result<(Option<Tensor>, Option<Tensor>)> {+ Err(crate::Error::BackwardNotSupported { op: self.name() })+ }+}++pub trait CustomOp3 {+ fn name(&self) -> &'static str;++ /// The forward pass, as run on a cpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn cpu_fwd(+ &self,+ s1: &CpuStorage,+ l1: &Layout,+ s2: &CpuStorage,+ l2: &Layout,+ s3: &CpuStorage,+ l3: &Layout,+ ) -> Result<(CpuStorage, Shape)>;++ /// The forward pass, as run on a gpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn cuda_fwd(+ &self,+ _: &CudaStorage,+ _: &Layout,+ _: &CudaStorage,+ _: &Layout,+ _: &CudaStorage,+ _: &Layout,+ ) -> Result<(CudaStorage, Shape)> {+ Err(crate::Error::Cuda(+ format!("no cuda implementation for {}", self.name()).into(),+ ))+ }++ /// The forward pass, as run on a metal gpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn metal_fwd(+ &self,+ _: &MetalStorage,+ _: &Layout,+ _: &MetalStorage,+ _: &Layout,+ _: &MetalStorage,+ _: &Layout,+ ) -> Result<(MetalStorage, Shape)> {+ Err(crate::Error::Metal(+ format!("no metal implementation for {}", self.name()).into(),+ ))+ }++ fn bwd(+ &self,+ _arg1: &Tensor,+ _arg2: &Tensor,+ _arg3: &Tensor,+ _res: &Tensor,+ _grad_res: &Tensor,+ ) -> Result<(Option<Tensor>, Option<Tensor>, Option<Tensor>)> {+ Err(crate::Error::BackwardNotSupported { op: self.name() })+ }+}++impl Tensor {+ /// Applies a unary custom op without backward support+ pub fn apply_op1_no_bwd<C: CustomOp1>(&self, c: &C) -> Result<Self> {+ let (storage, shape) = self.storage().apply_op1(self.layout(), c)?;+ Ok(from_storage(storage, shape, BackpropOp::none(), false))+ }++ /// Applies a binary custom op without backward support+ pub fn apply_op2_no_bwd<C: CustomOp2>(&self, rhs: &Self, c: &C) -> Result<Self> {+ let (storage, shape) =+ self.storage()+ .apply_op2(self.layout(), &rhs.storage(), rhs.layout(), c)?;+ Ok(from_storage(storage, shape, BackpropOp::none(), false))+ }++ /// Applies a ternary custom op without backward support+ pub fn apply_op3_no_bwd<C: CustomOp3>(&self, t2: &Self, t3: &Self, c: &C) -> Result<Self> {+ let (storage, shape) = self.storage().apply_op3(+ self.layout(),+ &t2.storage(),+ t2.layout(),+ &t3.storage(),+ t3.layout(),+ c,+ )?;+ Ok(from_storage(storage, shape, BackpropOp::none(), false))+ }++ /// Applies a unary custom op.+ pub fn apply_op1_arc(&self, c: Arc<Box<dyn CustomOp1 + Send + Sync>>) -> Result<Self> {+ let (storage, shape) = self+ .storage()+ .apply_op1(self.layout(), c.as_ref().as_ref())?;+ let op = BackpropOp::new1(self, |s| Op::CustomOp1(s, c.clone()));+ Ok(from_storage(storage, shape, op, false))+ }++ pub fn apply_op1<C: 'static + CustomOp1 + Send + Sync>(&self, c: C) -> Result<Self> {+ self.apply_op1_arc(Arc::new(Box::new(c)))+ }++ /// Applies a binary custom op.+ pub fn apply_op2_arc(+ &self,+ rhs: &Self,+ c: Arc<Box<dyn CustomOp2 + Send + Sync>>,+ ) -> Result<Self> {+ let (storage, shape) = self.storage().apply_op2(+ self.layout(),+ &rhs.storage(),+ rhs.layout(),+ c.as_ref().as_ref(),+ )?;+ let op = BackpropOp::new2(self, rhs, |t1, t2| Op::CustomOp2(t1, t2, c.clone()));+ Ok(from_storage(storage, shape, op, false))+ }++ pub fn apply_op2<C: 'static + CustomOp2 + Send + Sync>(&self, r: &Self, c: C) -> Result<Self> {+ self.apply_op2_arc(r, Arc::new(Box::new(c)))+ }++ /// Applies a ternary custom op.+ pub fn apply_op3_arc(+ &self,+ t2: &Self,+ t3: &Self,+ c: Arc<Box<dyn CustomOp3 + Send + Sync>>,+ ) -> Result<Self> {+ let (storage, shape) = self.storage().apply_op3(+ self.layout(),+ &t2.storage(),+ t2.layout(),+ &t3.storage(),+ t3.layout(),+ c.as_ref().as_ref(),+ )?;+ let op = BackpropOp::new3(self, t2, t3, |t1, t2, t3| {+ Op::CustomOp3(t1, t2, t3, c.clone())+ });+ Ok(from_storage(storage, shape, op, false))+ }++ pub fn apply_op3<C: 'static + CustomOp3 + Send + Sync>(+ &self,+ t2: &Self,+ t3: &Self,+ c: C,+ ) -> Result<Self> {+ self.apply_op3_arc(t2, t3, Arc::new(Box::new(c)))+ }+}++// In place ops.++/// Unary ops that can be defined in user-land.+/// These ops work in place and as such back-prop is unsupported.+pub trait InplaceOp1 {+ // Box<dyn> does not support const yet, so use a function to get the name.+ fn name(&self) -> &'static str;++ /// The forward pass, as run on a cpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn cpu_fwd(&self, storage: &mut CpuStorage, layout: &Layout) -> Result<()>;++ /// The forward pass, as run on a gpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn cuda_fwd(&self, _storage: &mut CudaStorage, _layout: &Layout) -> Result<()> {+ Err(crate::Error::Cuda(+ format!("no cuda implementation for {}", self.name()).into(),+ ))+ }++ /// The forward pass, as run on a metal gpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn metal_fwd(&self, _storage: &mut MetalStorage, _layout: &Layout) -> Result<()> {+ Err(crate::Error::Metal(+ format!("no metal implementation for {}", self.name()).into(),+ ))+ }+}++/// In-place ops that can be defined in user-land.+/// These ops work in-place and as such back-propagation is unsupported.+pub trait InplaceOpN<const N: usize> {+ fn name(&self) -> &'static str;++ /// Defines the source access pattern of this in-place op.+ /// Defaults to `None`, which rejects all in-place source aliasing.+ fn src_access_pattern(&self) -> Option<AccessPattern> {+ None+ }++ fn cpu_fwd(+ &self,+ dst: &mut CpuStorage,+ dst_l: &Layout,+ srcs: [(&CpuStorage, &Layout); N],+ ) -> Result<()> {+ let _ = (dst, dst_l, srcs);+ bail!("no cpu implementation for {}", self.name())+ }++ fn cpu_fwd_aliased(+ &self,+ dst: &mut CpuStorage,+ dst_l: &Layout,+ srcs: [(Src<'_, CpuStorage>, &Layout); N],+ ) -> Result<()> {+ let _ = (dst, dst_l, srcs);+ bail!("no aliased cpu implementation for {}", self.name())+ }++ fn cuda_fwd(+ &self,+ dst: &mut CudaStorage,+ dst_l: &Layout,+ srcs: [(&CudaStorage, &Layout); N],+ ) -> Result<()> {+ let _ = (dst, dst_l, srcs);+ bail!("no cuda implementation for {}", self.name())+ }++ fn cuda_fwd_aliased(+ &self,+ dst: &mut CudaStorage,+ dst_l: &Layout,+ srcs: [(Src<'_, CudaStorage>, &Layout); N],+ ) -> Result<()> {+ let _ = (dst, dst_l, srcs);+ bail!("no aliased cpu implementation for {}", self.name())+ }++ fn metal_fwd(+ &self,+ dst: &mut MetalStorage,+ dst_l: &Layout,+ srcs: [(&MetalStorage, &Layout); N],+ ) -> Result<()> {+ let _ = (dst, dst_l, srcs);+ bail!("no metal implementation for {}", self.name())+ }++ fn metal_fwd_aliased(+ &self,+ dst: &mut MetalStorage,+ dst_l: &Layout,+ srcs: [(Src<'_, MetalStorage>, &Layout); N],+ ) -> Result<()> {+ let _ = (dst, dst_l, srcs);+ bail!("no aliased metal implementation for {}", self.name())+ }+}++#[derive(Debug)]+pub enum Src<'a, S> {+ Distinct(&'a S),+ Aliased(LayoutRelation),+}++impl<S> Copy for Src<'_, S> {}++impl<S> Clone for Src<'_, S> {+ fn clone(&self) -> Self {+ *self+ }+}++// If all sources are distinct we return the entire array of tensors without the `Src` wrapper.+pub(crate) fn all_distinct<'a, B, const N: usize>(+ srcs: &[(Src<'a, B>, &'a Layout); N],+) -> Option<[(&'a B, &'a Layout); N]> {+ srcs.iter()+ .all(|(s, _)| matches!(s, Src::Distinct(_)))+ .then(|| {+ std::array::from_fn(|i| match srcs[i] {+ (Src::Distinct(s), l) => (s, l),+ _ => unreachable!("checked immediately above"),+ })+ })+}++/// Indicates which indices a kernel reads, relative to the destination indices it writes.+///+/// When used with [`crate::LayoutRelation`] we can describe safe access patterns.+/// For example `Elementwise` is safe to use with both `LayoutRelation::Identical` and `LayoutRelation::Disjoint`,+/// while `Arbitrary` is only guaranteed to be safe with `LayoutRelation::Disjoint`.+#[derive(Copy, Clone, PartialEq, Eq, Debug)]+pub enum AccessPattern {+ /// Reads only source index `i` when writing destination index `i`.+ Elementwise,+ /// Reads arbitrary source indices.+ Arbitrary,+}++impl AccessPattern {+ fn supports(self, rel: LayoutRelation) -> bool {+ matches!(+ (self, rel),+ (+ AccessPattern::Elementwise,+ LayoutRelation::Identical | LayoutRelation::Disjoint+ ) | (AccessPattern::Arbitrary, LayoutRelation::Disjoint)+ )+ }+}++macro_rules! forward_op1 {+ ($fwd:ident, $fwd_aliased:ident, $storage:ty) => {+ fn $fwd(+ &self,+ dst: &mut $storage,+ dl: &Layout,+ _: [(&$storage, &Layout); 0],+ ) -> Result<()> {+ InplaceOp1::$fwd(self, dst, dl)+ }++ fn $fwd_aliased(+ &self,+ dst: &mut $storage,+ dl: &Layout,+ _: [(Src<'_, $storage>, &Layout); 0],+ ) -> Result<()> {+ InplaceOp1::$fwd(self, dst, dl)+ }+ };+}++impl<C: InplaceOp1> InplaceOpN<0> for C {+ fn name(&self) -> &'static str {+ InplaceOp1::name(self)+ }++ forward_op1!(cpu_fwd, cpu_fwd_aliased, CpuStorage);+ forward_op1!(cuda_fwd, cuda_fwd_aliased, CudaStorage);+ forward_op1!(metal_fwd, metal_fwd_aliased, MetalStorage);+}++macro_rules! forward_op2 {+ ($fwd:ident, $fwd_aliased:ident, $storage:ty) => {+ fn $fwd(+ &self,+ dst: &mut $storage,+ dl: &Layout,+ srcs: [(&$storage, &Layout); 1],+ ) -> Result<()> {+ let [(s, sl)] = srcs;+ InplaceOp2::$fwd(self, dst, dl, s, sl)+ }++ fn $fwd_aliased(+ &self,+ dst: &mut $storage,+ dl: &Layout,+ srcs: [(Src<'_, $storage>, &Layout); 1],+ ) -> Result<()> {+ match srcs {+ [(Src::Distinct(s), sl)] => InplaceOp2::$fwd(self, dst, dl, s, sl),+ _ => bail!(+ "{}: aliased input requires migrating to InplaceOpN",+ self.name()+ ),+ }+ }+ };+}++impl<C: InplaceOp2> InplaceOpN<1> for C {+ fn name(&self) -> &'static str {+ InplaceOp2::name(self)+ }++ forward_op2!(cpu_fwd, cpu_fwd_aliased, CpuStorage);+ forward_op2!(cuda_fwd, cuda_fwd_aliased, CudaStorage);+ forward_op2!(metal_fwd, metal_fwd_aliased, MetalStorage);+}++pub trait InplaceOp2 {+ fn name(&self) -> &'static str;++ /// The forward pass, as run on a cpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn cpu_fwd(&self, s1: &mut CpuStorage, l1: &Layout, s2: &CpuStorage, l2: &Layout)+ -> Result<()>;++ /// The forward pass, as run on a gpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn cuda_fwd(+ &self,+ s1: &mut CudaStorage,+ l1: &Layout,+ s2: &CudaStorage,+ l2: &Layout,+ ) -> Result<()> {+ _ = (s1, l1, s2, l2);+ Err(crate::Error::Cuda(+ format!("no cuda implementation for {}", self.name()).into(),+ ))+ }++ /// The forward pass, as run on a metal gpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn metal_fwd(+ &self,+ s1: &mut MetalStorage,+ l1: &Layout,+ s2: &MetalStorage,+ l2: &Layout,+ ) -> Result<()> {+ _ = (s1, l1, s2, l2);+ Err(crate::Error::Metal(+ format!("no metal implementation for {}", self.name()).into(),+ ))+ }+}++macro_rules! forward_op3 {+ ($fwd:ident, $fwd_aliased:ident, $storage:ty) => {+ fn $fwd(+ &self,+ dst: &mut $storage,+ dl: &Layout,+ srcs: [(&$storage, &Layout); 2],+ ) -> Result<()> {+ let [(s1, l1), (s2, l2)] = srcs;+ InplaceOp3::$fwd(self, dst, dl, s1, l1, s2, l2)+ }++ fn $fwd_aliased(+ &self,+ dst: &mut $storage,+ dl: &Layout,+ srcs: [(Src<'_, $storage>, &Layout); 2],+ ) -> Result<()> {+ match srcs {+ [(Src::Distinct(s1), l1), (Src::Distinct(s2), l2)] => {+ InplaceOp3::$fwd(self, dst, dl, s1, l1, s2, l2)+ }+ _ => bail!(+ "{}: aliased input requires migrating to InplaceOpN",+ self.name()+ ),+ }+ }+ };+}++impl<C: InplaceOp3> InplaceOpN<2> for C {+ fn name(&self) -> &'static str {+ InplaceOp3::name(self)+ }++ forward_op3!(cpu_fwd, cpu_fwd_aliased, CpuStorage);+ forward_op3!(cuda_fwd, cuda_fwd_aliased, CudaStorage);+ forward_op3!(metal_fwd, metal_fwd_aliased, MetalStorage);+}++pub trait InplaceOp3 {+ fn name(&self) -> &'static str;++ /// The forward pass, as run on a cpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn cpu_fwd(+ &self,+ s1: &mut CpuStorage,+ l1: &Layout,+ s2: &CpuStorage,+ l2: &Layout,+ s3: &CpuStorage,+ l3: &Layout,+ ) -> Result<()>;++ /// The forward pass, as run on a gpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn cuda_fwd(+ &self,+ _: &mut CudaStorage,+ _: &Layout,+ _: &CudaStorage,+ _: &Layout,+ _: &CudaStorage,+ _: &Layout,+ ) -> Result<()> {+ Err(crate::Error::Cuda(+ format!("no cuda implementation for {}", self.name()).into(),+ ))+ }++ /// The forward pass, as run on a metal gpu device. Note that the storage can use arbitrary strides,+ /// offsets etc so the associated layout should be used to access it.+ fn metal_fwd(+ &self,+ _: &mut MetalStorage,+ _: &Layout,+ _: &MetalStorage,+ _: &Layout,+ _: &MetalStorage,+ _: &Layout,+ ) -> Result<()> {+ Err(crate::Error::Metal(+ format!("no metal implementation for {}", self.name()).into(),+ ))+ }+}++impl Tensor {+ /// Applies a custom op in-place for the `self` tensor.+ ///+ /// Tensors sharing underlying storage with `self` are classified and passed as [`Src::Aliased`].+ /// Separate tensors are locked and passed as [`Src::Distinct`].+ fn inplace_op<const N: usize, C: InplaceOpN<N>>(&self, srcs: [&Self; N], c: &C) -> Result<()> {+ let name = c.name();++ // Ensure writes cannot collide with themselves+ if self.layout().has_internal_overlap() {+ bail!("{name}: dst has repeated elements (zero-stride). Can not write in-place")+ }++ // Classify srcs wrt dst+ let access = c.src_access_pattern();+ let mut rels: [Option<LayoutRelation>; N] = [None; N];+ for i in 0..N {+ if !self.same_storage(srcs[i]) {+ continue;+ }+ let rel = Layout::relation(self.layout(), srcs[i].layout());+ match access {+ Some(a) if a.supports(rel) => rels[i] = Some(rel),+ Some(a) => bail!(+ "src {i} shares storage with dst ({rel:?}), which is not supported for the access pattern of `{name}` ({a:?})."+ ),+ None => bail!(+ "src {i} shares storage with dst, and `{name}` does not support aliased operands."+ ),+ }+ }++ // Acquire locks in order sorted by `Tensor::storage_key`.+ // Avoids deadlock from cycle of waiting locks.+ let dst_key = self.storage_key();++ let mut order: [usize; N] = std::array::from_fn(|i| i);+ order.sort_unstable_by_key(|&i| srcs[i].storage_key());++ let mut guards: [Option<_>; N] = std::array::from_fn(|_| None);+ let mut dst: Option<_> = None;++ for &i in order.iter() {+ if rels[i].is_some() {+ continue; // Aliased. Read through `dst`+ }+ let key = srcs[i].storage_key();+ if key > dst_key && dst.is_none() {+ dst = Some(self.storage_mut());+ }+ // Two sources sharing the same allocation, but distinct from dst.+ // `read_recursive` allows for shared read access+ guards[i] = Some(srcs[i].storage());+ }+ // If `dst` is not yet set we acquire it from `self` now.+ let mut dst = match dst {+ Some(g) => g,+ None => self.storage_mut(),+ };++ let operands: [(Src<'_, Storage>, &Layout); N] = std::array::from_fn(|i| {+ let s = match (&guards[i], rels[i]) {+ (Some(g), None) => Src::Distinct(&**g),+ (None, Some(rel)) => Src::Aliased(rel),+ _ => unreachable!(+ "Source is either distinct or aliased. Other match patterns should be impossible"+ ),+ };+ (s, srcs[i].layout())+ });++ dst.inplace_op(self.layout(), operands, c)+ }++ /// Applies a unary custom op in place.+ pub fn inplace_op1<C: InplaceOp1>(&self, c: &C) -> Result<()> {+ self.inplace_op([], c)+ }++ /// Applies a binary custom op in place (for the first tensor).+ pub fn inplace_op2<C: InplaceOpN<1>>(&self, rhs: &Self, c: &C) -> Result<()> {+ self.inplace_op([rhs], c)+ }++ /// Applies a ternary custom op in place (for the first tensor).+ pub fn inplace_op3<C: InplaceOpN<2>>(&self, t2: &Self, t3: &Self, c: &C) -> Result<()> {+ self.inplace_op([t2, t3], c)+ }+}diff --git a/vendor/candle-core/src/device.rs b/vendor/candle-core/src/device.rsnew file mode 100644index 0000000..f5b14c1--- /dev/null+++ b/vendor/candle-core/src/device.rs@@ -0,0 +1,518 @@+use crate::backend::BackendDevice;+use crate::cpu_backend::CpuDevice;+use crate::{CpuStorage, DType, Result, Shape, Storage, WithDType};++/// A `DeviceLocation` represents a physical device whereas multiple `Device`+/// can live on the same location (typically for cuda devices).+#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]+pub enum DeviceLocation {+ Cpu,+ Cuda { gpu_id: usize },+ Metal { gpu_id: usize },+}++/// Cpu, Cuda, or Metal+#[derive(Debug, Clone)]+pub enum Device {+ Cpu,+ Cuda(crate::CudaDevice),+ Metal(crate::MetalDevice),+}++pub trait NdArray {+ fn shape(&self) -> Result<Shape>;++ fn to_cpu_storage(&self) -> CpuStorage;+}++impl<S: WithDType> NdArray for S {+ fn shape(&self) -> Result<Shape> {+ Ok(Shape::from(()))+ }++ fn to_cpu_storage(&self) -> CpuStorage {+ S::to_cpu_storage(&[*self])+ }+}++impl<S: WithDType, const N: usize> NdArray for &[S; N] {+ fn shape(&self) -> Result<Shape> {+ Ok(Shape::from(self.len()))+ }++ fn to_cpu_storage(&self) -> CpuStorage {+ S::to_cpu_storage(self.as_slice())+ }+}++impl<S: WithDType> NdArray for &[S] {+ fn shape(&self) -> Result<Shape> {+ Ok(Shape::from(self.len()))+ }++ fn to_cpu_storage(&self) -> CpuStorage {+ S::to_cpu_storage(self)+ }+}++impl<S: WithDType, const N: usize, const M: usize> NdArray for &[[S; N]; M] {+ fn shape(&self) -> Result<Shape> {+ Ok(Shape::from((M, N)))+ }++ fn to_cpu_storage(&self) -> CpuStorage {+ S::to_cpu_storage_owned(self.concat())+ }+}++impl<S: WithDType, const N1: usize, const N2: usize, const N3: usize> NdArray+ for &[[[S; N3]; N2]; N1]+{+ fn shape(&self) -> Result<Shape> {+ Ok(Shape::from((N1, N2, N3)))+ }++ fn to_cpu_storage(&self) -> CpuStorage {+ let mut vec = Vec::with_capacity(N1 * N2 * N3);+ for i1 in 0..N1 {+ for i2 in 0..N2 {+ vec.extend(self[i1][i2])+ }+ }+ S::to_cpu_storage_owned(vec)+ }+}++impl<S: WithDType, const N1: usize, const N2: usize, const N3: usize, const N4: usize> NdArray+ for &[[[[S; N4]; N3]; N2]; N1]+{+ fn shape(&self) -> Result<Shape> {+ Ok(Shape::from((N1, N2, N3, N4)))+ }++ fn to_cpu_storage(&self) -> CpuStorage {+ let mut vec = Vec::with_capacity(N1 * N2 * N3 * N4);+ for i1 in 0..N1 {+ for i2 in 0..N2 {+ for i3 in 0..N3 {+ vec.extend(self[i1][i2][i3])+ }+ }+ }+ S::to_cpu_storage_owned(vec)+ }+}++impl<S: WithDType> NdArray for Vec<S> {+ fn shape(&self) -> Result<Shape> {+ Ok(Shape::from(self.len()))+ }++ fn to_cpu_storage(&self) -> CpuStorage {+ S::to_cpu_storage(self.as_slice())+ }+}++impl<S: WithDType> NdArray for Vec<&[S]> {+ fn shape(&self) -> Result<Shape> {+ if self.is_empty() {+ crate::bail!("empty array")+ }+ let n = self.len();+ let m = self[0].len();+ for v in self.iter() {+ if v.len() != m {+ crate::bail!("two elements have different len {m} {}", v.len())+ }+ }+ Ok(Shape::from((n, m)))+ }++ fn to_cpu_storage(&self) -> CpuStorage {+ let data = self.iter().copied().flatten().copied().collect::<Vec<_>>();+ S::to_cpu_storage_owned(data)+ }+}++impl<S: WithDType> NdArray for Vec<Vec<S>> {+ fn shape(&self) -> Result<Shape> {+ if self.is_empty() {+ crate::bail!("empty array")+ }+ let n = self.len();+ let m = self[0].len();+ for v in self.iter() {+ if v.len() != m {+ crate::bail!("two elements have different len {m} {}", v.len())+ }+ }+ Ok(Shape::from((n, m)))+ }++ fn to_cpu_storage(&self) -> CpuStorage {+ let len: usize = self.iter().map(|v| v.len()).sum();+ let mut dst = Vec::with_capacity(len);+ for v in self.iter() {+ dst.extend(v.iter().copied());+ }+ S::to_cpu_storage_owned(dst)+ }+}++impl<S: WithDType> NdArray for Vec<Vec<Vec<S>>> {+ fn shape(&self) -> Result<Shape> {+ if self.is_empty() {+ crate::bail!("empty array")+ }+ let shape0 = self[0].shape()?;+ let n = self.len();+ for v in self.iter() {+ let shape = v.shape()?;+ if shape != shape0 {+ crate::bail!("two elements have different shapes {shape:?} {shape0:?}")+ }+ }+ Ok(Shape::from([[n].as_slice(), shape0.dims()].concat()))+ }++ fn to_cpu_storage(&self) -> CpuStorage {+ if self.is_empty() {+ return S::to_cpu_storage_owned(vec![]);+ }+ let len: usize = self+ .iter()+ .map(|v| v.iter().map(|v| v.len()).sum::<usize>())+ .sum();+ let mut dst = Vec::with_capacity(len);+ for v1 in self.iter() {+ for v2 in v1.iter() {+ dst.extend(v2.iter().copied());+ }+ }+ S::to_cpu_storage_owned(dst)+ }+}++impl<S: WithDType> NdArray for Vec<Vec<Vec<Vec<S>>>> {+ fn shape(&self) -> Result<Shape> {+ if self.is_empty() {+ crate::bail!("empty array")+ }+ let shape0 = self[0].shape()?;+ let n = self.len();+ for v in self.iter() {+ let shape = v.shape()?;+ if shape != shape0 {+ crate::bail!("two elements have different shapes {shape:?} {shape0:?}")+ }+ }+ Ok(Shape::from([[n].as_slice(), shape0.dims()].concat()))+ }++ fn to_cpu_storage(&self) -> CpuStorage {+ let len: usize = self+ .iter()+ .map(|v| {+ v.iter()+ .map(|v| v.iter().map(|v| v.len()).sum::<usize>())+ .sum::<usize>()+ })+ .sum();+ let mut dst = Vec::with_capacity(len);+ for v1 in self.iter() {+ for v2 in v1.iter() {+ for v3 in v2.iter() {+ dst.extend(v3.iter().copied());+ }+ }+ }+ S::to_cpu_storage_owned(dst)+ }+}++impl Device {+ pub fn new_cuda(ordinal: usize) -> Result<Self> {+ Ok(Self::Cuda(crate::CudaDevice::new(ordinal)?))+ }++ pub fn as_cuda_device(&self) -> Result<&crate::CudaDevice> {+ match self {+ Self::Cuda(d) => Ok(d),+ Self::Cpu => crate::bail!("expected a cuda device, got cpu"),+ Self::Metal(_) => crate::bail!("expected a cuda device, got Metal"),+ }+ }++ pub fn as_metal_device(&self) -> Result<&crate::MetalDevice> {+ match self {+ Self::Cuda(_) => crate::bail!("expected a metal device, got cuda"),+ Self::Cpu => crate::bail!("expected a metal device, got cpu"),+ Self::Metal(d) => Ok(d),+ }+ }++ pub fn new_cuda_with_stream(ordinal: usize) -> Result<Self> {+ Ok(Self::Cuda(crate::CudaDevice::new_with_stream(ordinal)?))+ }++ pub fn new_metal(ordinal: usize) -> Result<Self> {+ Ok(Self::Metal(crate::MetalDevice::new(ordinal)?))+ }++ /// Run `f` with device specific context.+ ///+ /// On CPU this installs candle's private rayon thread pool for the+ /// duration of `f`, keeping worker threads warm across the many short+ /// parallel sections in a model forward pass. Currently noop for other backends.+ pub fn with_context<F, R>(&self, f: F) -> R+ where+ F: FnOnce() -> R + Send,+ R: Send,+ {+ match self {+ Self::Cpu => crate::utils::with_threadpool(f),+ _ => f(),+ }+ }++ pub fn set_seed(&self, seed: u64) -> Result<()> {+ match self {+ Self::Cpu => CpuDevice.set_seed(seed),+ Self::Cuda(c) => c.set_seed(seed),+ Self::Metal(m) => m.set_seed(seed),+ }+ }++ pub fn get_current_seed(&self) -> Result<u64> {+ match self {+ Self::Cpu => CpuDevice.get_current_seed(),+ Self::Cuda(c) => c.get_current_seed(),+ Self::Metal(m) => m.get_current_seed(),+ }+ }++ pub fn same_device(&self, rhs: &Self) -> bool {+ match (self, rhs) {+ (Self::Cpu, Self::Cpu) => true,+ (Self::Cuda(lhs), Self::Cuda(rhs)) => lhs.same_device(rhs),+ (Self::Metal(lhs), Self::Metal(rhs)) => lhs.same_device(rhs),+ _ => false,+ }+ }++ pub fn location(&self) -> DeviceLocation {+ match self {+ Self::Cpu => DeviceLocation::Cpu,+ Self::Cuda(device) => device.location(),+ Device::Metal(device) => device.location(),+ }+ }++ pub fn is_cpu(&self) -> bool {+ matches!(self, Self::Cpu)+ }++ pub fn is_cuda(&self) -> bool {+ matches!(self, Self::Cuda(_))+ }++ pub fn is_metal(&self) -> bool {+ matches!(self, Self::Metal(_))+ }++ pub fn supports_bf16(&self) -> bool {+ match self {+ Self::Cuda(_) | Self::Metal(_) => true,+ Self::Cpu => false,+ }+ }++ /// Return `BF16` for devices that support it, otherwise default to `F32`.+ pub fn bf16_default_to_f32(&self) -> DType {+ if self.supports_bf16() {+ DType::BF16+ } else {+ DType::F32+ }+ }++ pub fn cuda_if_available(ordinal: usize) -> Result<Self> {+ if crate::utils::cuda_is_available() {+ Self::new_cuda(ordinal)+ } else {+ Ok(Self::Cpu)+ }+ }++ pub fn metal_if_available(ordinal: usize) -> Result<Self> {+ if crate::utils::metal_is_available() {+ Self::new_metal(ordinal)+ } else {+ Ok(Self::Cpu)+ }+ }++ pub(crate) fn rand_uniform_f64(+ &self,+ lo: f64,+ up: f64,+ shape: &Shape,+ dtype: DType,+ ) -> Result<Storage> {+ match self {+ Device::Cpu => {+ let storage = CpuDevice.rand_uniform(shape, dtype, lo, up)?;+ Ok(Storage::Cpu(storage))+ }+ Device::Cuda(device) => {+ // TODO: Remove the special case if we start supporting generating f16/bf16 directly.+ if dtype == DType::F16 || dtype == DType::BF16 {+ let storage = device.rand_uniform(shape, DType::F32, lo, up)?;+ Storage::Cuda(storage).to_dtype(&crate::Layout::contiguous(shape), dtype)+ } else {+ let storage = device.rand_uniform(shape, dtype, lo, up)?;+ Ok(Storage::Cuda(storage))+ }+ }+ Device::Metal(device) => {+ let storage = device.rand_uniform(shape, dtype, lo, up)?;+ Ok(Storage::Metal(storage))+ }+ }+ }++ pub(crate) fn rand_uniform<T: crate::FloatDType>(+ &self,+ lo: T,+ up: T,+ shape: &Shape,+ ) -> Result<Storage> {+ self.rand_uniform_f64(lo.to_f64(), up.to_f64(), shape, T::DTYPE)+ }++ pub(crate) fn rand_normal_f64(+ &self,+ mean: f64,+ std: f64,+ shape: &Shape,+ dtype: DType,+ ) -> Result<Storage> {+ match self {+ Device::Cpu => {+ let storage = CpuDevice.rand_normal(shape, dtype, mean, std)?;+ Ok(Storage::Cpu(storage))+ }+ Device::Cuda(device) => {+ // TODO: Remove the special case if we start supporting generating f16/bf16 directly.+ if dtype == DType::F16 || dtype == DType::BF16 {+ let storage = device.rand_normal(shape, DType::F32, mean, std)?;+ Storage::Cuda(storage).to_dtype(&crate::Layout::contiguous(shape), dtype)+ } else {+ let storage = device.rand_normal(shape, dtype, mean, std)?;+ Ok(Storage::Cuda(storage))+ }+ }+ Device::Metal(device) => {+ let storage = device.rand_normal(shape, dtype, mean, std)?;+ Ok(Storage::Metal(storage))+ }+ }+ }++ pub(crate) fn rand_normal<T: crate::FloatDType>(+ &self,+ mean: T,+ std: T,+ shape: &Shape,+ ) -> Result<Storage> {+ self.rand_normal_f64(mean.to_f64(), std.to_f64(), shape, T::DTYPE)+ }++ pub(crate) fn zeros(&self, shape: &Shape, dtype: DType) -> Result<Storage> {+ match self {+ Device::Cpu => {+ let storage = CpuDevice.zeros_impl(shape, dtype)?;+ Ok(Storage::Cpu(storage))+ }+ Device::Cuda(device) => {+ let storage = device.zeros_impl(shape, dtype)?;+ Ok(Storage::Cuda(storage))+ }+ Device::Metal(device) => {+ let storage = device.zeros_impl(shape, dtype)?;+ Ok(Storage::Metal(storage))+ }+ }+ }++ pub(crate) unsafe fn alloc_uninit(&self, shape: &Shape, dtype: DType) -> Result<Storage> {+ match self {+ Device::Cpu => {+ let storage = CpuDevice.alloc_uninit(shape, dtype)?;+ Ok(Storage::Cpu(storage))+ }+ Device::Cuda(device) => {+ let storage = device.alloc_uninit(shape, dtype)?;+ Ok(Storage::Cuda(storage))+ }+ Device::Metal(device) => {+ let storage = device.alloc_uninit(shape, dtype)?;+ Ok(Storage::Metal(storage))+ }+ }+ }++ pub(crate) fn storage_from_slice<D: WithDType>(&self, data: &[D]) -> Result<Storage> {+ match self {+ Device::Cpu => Ok(Storage::Cpu(data.to_cpu_storage())),+ Device::Cuda(device) => {+ let storage = device.storage_from_slice(data)?;+ Ok(Storage::Cuda(storage))+ }+ Device::Metal(device) => {+ let storage = device.storage_from_slice(data)?;+ Ok(Storage::Metal(storage))+ }+ }+ }++ pub(crate) fn storage<A: NdArray>(&self, array: A) -> Result<Storage> {+ match self {+ Device::Cpu => Ok(Storage::Cpu(array.to_cpu_storage())),+ Device::Cuda(device) => {+ let storage = array.to_cpu_storage();+ let storage = device.storage_from_cpu_storage_owned(storage)?;+ Ok(Storage::Cuda(storage))+ }+ Device::Metal(device) => {+ let storage = array.to_cpu_storage();+ let storage = device.storage_from_cpu_storage_owned(storage)?;+ Ok(Storage::Metal(storage))+ }+ }+ }++ pub(crate) fn storage_owned<S: WithDType>(&self, data: Vec<S>) -> Result<Storage> {+ match self {+ Device::Cpu => Ok(Storage::Cpu(S::to_cpu_storage_owned(data))),+ Device::Cuda(device) => {+ let storage = S::to_cpu_storage_owned(data);+ let storage = device.storage_from_cpu_storage_owned(storage)?;+ Ok(Storage::Cuda(storage))+ }+ Device::Metal(device) => {+ let storage = S::to_cpu_storage_owned(data);+ let storage = device.storage_from_cpu_storage_owned(storage)?;+ Ok(Storage::Metal(storage))+ }+ }+ }++ pub fn synchronize(&self) -> Result<()> {+ match self {+ Self::Cpu => Ok(()),+ Self::Cuda(d) => d.synchronize(),+ Self::Metal(d) => d.synchronize(),+ }+ }+}diff --git a/vendor/candle-core/src/display.rs b/vendor/candle-core/src/display.rsnew file mode 100644index 0000000..a9b5394--- /dev/null+++ b/vendor/candle-core/src/display.rs@@ -0,0 +1,558 @@+//! Pretty printing of tensors+//!+//! This implementation should be in line with the [PyTorch version](https://github.com/pytorch/pytorch/blob/7b419e8513a024e172eae767e24ec1b849976b13/torch/_tensor_str.py).+//!+use crate::{DType, Result, Tensor, WithDType};+use half::{bf16, f16};++impl Tensor {+ fn fmt_dt<T: WithDType + std::fmt::Display>(+ &self,+ f: &mut std::fmt::Formatter,+ ) -> std::fmt::Result {+ let device_str = match self.device().location() {+ crate::DeviceLocation::Cpu => "".to_owned(),+ crate::DeviceLocation::Cuda { gpu_id } => {+ format!(", cuda:{gpu_id}")+ }+ crate::DeviceLocation::Metal { gpu_id } => {+ format!(", metal:{gpu_id}")+ }+ };++ write!(f, "Tensor[")?;+ match self.dims() {+ [] => {+ if let Ok(v) = self.to_scalar::<T>() {+ write!(f, "{v}")?+ }+ }+ [s] if *s < 10 => {+ if let Ok(vs) = self.to_vec1::<T>() {+ for (i, v) in vs.iter().enumerate() {+ if i > 0 {+ write!(f, ", ")?;+ }+ write!(f, "{v}")?;+ }+ }+ }+ dims => {+ write!(f, "dims ")?;+ for (i, d) in dims.iter().enumerate() {+ if i > 0 {+ write!(f, ", ")?;+ }+ write!(f, "{d}")?;+ }+ }+ }+ write!(f, "; {}{}]", self.dtype().as_str(), device_str)+ }+}++impl std::fmt::Debug for Tensor {+ fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {+ match self.dtype() {+ DType::U8 => self.fmt_dt::<u8>(f),+ DType::U32 => self.fmt_dt::<u32>(f),+ DType::I16 => self.fmt_dt::<i16>(f),+ DType::I32 => self.fmt_dt::<i32>(f),+ DType::I64 => self.fmt_dt::<i64>(f),+ DType::BF16 => self.fmt_dt::<bf16>(f),+ DType::F16 => self.fmt_dt::<f16>(f),+ DType::F32 => self.fmt_dt::<f32>(f),+ DType::F64 => self.fmt_dt::<f64>(f),+ DType::F8E4M3 => self.fmt_dt::<float8::F8E4M3>(f),+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ write!(+ f,+ "Tensor[{:?}; dtype={}, unsupported dummy type]",+ self.shape(),+ self.dtype().as_str()+ )+ }+ }+ }+}++/// Options for Tensor pretty printing+#[derive(Debug, Clone)]+pub struct PrinterOptions {+ pub precision: usize,+ pub threshold: usize,+ pub edge_items: usize,+ pub line_width: usize,+ pub sci_mode: Option<bool>,+}++static PRINT_OPTS: std::sync::Mutex<PrinterOptions> =+ std::sync::Mutex::new(PrinterOptions::const_default());++impl PrinterOptions {+ // We cannot use the default trait as it's not const.+ const fn const_default() -> Self {+ Self {+ precision: 4,+ threshold: 1000,+ edge_items: 3,+ line_width: 80,+ sci_mode: None,+ }+ }+}++pub fn print_options() -> &'static std::sync::Mutex<PrinterOptions> {+ &PRINT_OPTS+}++pub fn set_print_options(options: PrinterOptions) {+ *PRINT_OPTS.lock().unwrap() = options+}++pub fn set_print_options_default() {+ *PRINT_OPTS.lock().unwrap() = PrinterOptions::const_default()+}++pub fn set_print_options_short() {+ *PRINT_OPTS.lock().unwrap() = PrinterOptions {+ precision: 2,+ threshold: 1000,+ edge_items: 2,+ line_width: 80,+ sci_mode: None,+ }+}++pub fn set_print_options_full() {+ *PRINT_OPTS.lock().unwrap() = PrinterOptions {+ precision: 4,+ threshold: usize::MAX,+ edge_items: 3,+ line_width: 80,+ sci_mode: None,+ }+}++pub fn set_line_width(line_width: usize) {+ PRINT_OPTS.lock().unwrap().line_width = line_width+}++pub fn set_precision(precision: usize) {+ PRINT_OPTS.lock().unwrap().precision = precision+}++pub fn set_edge_items(edge_items: usize) {+ PRINT_OPTS.lock().unwrap().edge_items = edge_items+}++pub fn set_threshold(threshold: usize) {+ PRINT_OPTS.lock().unwrap().threshold = threshold+}++pub fn set_sci_mode(sci_mode: Option<bool>) {+ PRINT_OPTS.lock().unwrap().sci_mode = sci_mode+}++struct FmtSize {+ current_size: usize,+}++impl FmtSize {+ fn new() -> Self {+ Self { current_size: 0 }+ }++ fn final_size(self) -> usize {+ self.current_size+ }+}++impl std::fmt::Write for FmtSize {+ fn write_str(&mut self, s: &str) -> std::fmt::Result {+ self.current_size += s.len();+ Ok(())+ }+}++trait TensorFormatter {+ type Elem: WithDType;++ fn fmt<T: std::fmt::Write>(&self, v: Self::Elem, max_w: usize, f: &mut T) -> std::fmt::Result;++ fn max_width(&self, to_display: &Tensor) -> usize {+ let mut max_width = 1;+ if let Ok(vs) = to_display.flatten_all().and_then(|t| t.to_vec1()) {+ for &v in vs.iter() {+ let mut fmt_size = FmtSize::new();+ let _res = self.fmt(v, 1, &mut fmt_size);+ max_width = usize::max(max_width, fmt_size.final_size())+ }+ }+ max_width+ }++ fn write_newline_indent(i: usize, f: &mut std::fmt::Formatter) -> std::fmt::Result {+ writeln!(f)?;+ for _ in 0..i {+ write!(f, " ")?+ }+ Ok(())+ }++ fn fmt_tensor(+ &self,+ t: &Tensor,+ indent: usize,+ max_w: usize,+ summarize: bool,+ po: &PrinterOptions,+ f: &mut std::fmt::Formatter,+ ) -> std::fmt::Result {+ let dims = t.dims();+ let edge_items = po.edge_items;+ write!(f, "[")?;+ match dims {+ [] => {+ if let Ok(v) = t.to_scalar::<Self::Elem>() {+ self.fmt(v, max_w, f)?+ }+ }+ [v] if summarize && *v > 2 * edge_items => {+ if let Ok(vs) = t+ .narrow(0, 0, edge_items)+ .and_then(|t| t.to_vec1::<Self::Elem>())+ {+ for v in vs.into_iter() {+ self.fmt(v, max_w, f)?;+ write!(f, ", ")?;+ }+ }+ write!(f, "...")?;+ if let Ok(vs) = t+ .narrow(0, v - edge_items, edge_items)+ .and_then(|t| t.to_vec1::<Self::Elem>())+ {+ for v in vs.into_iter() {+ write!(f, ", ")?;+ self.fmt(v, max_w, f)?;+ }+ }+ }+ [_] => {+ let elements_per_line = usize::max(1, po.line_width / (max_w + 2));+ if let Ok(vs) = t.to_vec1::<Self::Elem>() {+ for (i, v) in vs.into_iter().enumerate() {+ if i > 0 {+ if i % elements_per_line == 0 {+ write!(f, ",")?;+ Self::write_newline_indent(indent, f)?+ } else {+ write!(f, ", ")?;+ }+ }+ self.fmt(v, max_w, f)?+ }+ }+ }+ _ => {+ if summarize && dims[0] > 2 * edge_items {+ for i in 0..edge_items {+ match t.get(i) {+ Ok(t) => self.fmt_tensor(&t, indent + 1, max_w, summarize, po, f)?,+ Err(e) => write!(f, "{e:?}")?,+ }+ write!(f, ",")?;+ Self::write_newline_indent(indent, f)?+ }+ write!(f, "...")?;+ Self::write_newline_indent(indent, f)?;+ for i in dims[0] - edge_items..dims[0] {+ match t.get(i) {+ Ok(t) => self.fmt_tensor(&t, indent + 1, max_w, summarize, po, f)?,+ Err(e) => write!(f, "{e:?}")?,+ }+ if i + 1 != dims[0] {+ write!(f, ",")?;+ Self::write_newline_indent(indent, f)?+ }+ }+ } else {+ for i in 0..dims[0] {+ match t.get(i) {+ Ok(t) => self.fmt_tensor(&t, indent + 1, max_w, summarize, po, f)?,+ Err(e) => write!(f, "{e:?}")?,+ }+ if i + 1 != dims[0] {+ write!(f, ",")?;+ Self::write_newline_indent(indent, f)?+ }+ }+ }+ }+ }+ write!(f, "]")?;+ Ok(())+ }+}++struct FloatFormatter<S: WithDType> {+ int_mode: bool,+ sci_mode: bool,+ precision: usize,+ _phantom: std::marker::PhantomData<S>,+}++impl<S> FloatFormatter<S>+where+ S: WithDType + num_traits::Float + std::fmt::Display,+{+ fn new(t: &Tensor, po: &PrinterOptions) -> Result<Self> {+ let mut int_mode = true;+ let mut sci_mode = false;++ // Rather than containing all values, this should only include+ // values that end up being displayed according to [threshold].+ let values = t+ .flatten_all()?+ .to_vec1()?+ .into_iter()+ .filter(|v: &S| v.is_finite() && !v.is_zero())+ .collect::<Vec<_>>();+ if !values.is_empty() {+ let mut nonzero_finite_min = S::max_value();+ let mut nonzero_finite_max = S::min_value();+ for &v in values.iter() {+ let v = v.abs();+ if v < nonzero_finite_min {+ nonzero_finite_min = v+ }+ if v > nonzero_finite_max {+ nonzero_finite_max = v+ }+ }++ for &value in values.iter() {+ if value.ceil() != value {+ int_mode = false;+ break;+ }+ }+ if let Some(v1) = S::from(1000.) {+ if let Some(v2) = S::from(1e8) {+ if let Some(v3) = S::from(1e-4) {+ sci_mode = nonzero_finite_max / nonzero_finite_min > v1+ || nonzero_finite_max > v2+ || nonzero_finite_min < v3+ }+ }+ }+ }++ match po.sci_mode {+ None => {}+ Some(v) => sci_mode = v,+ }+ Ok(Self {+ int_mode,+ sci_mode,+ precision: po.precision,+ _phantom: std::marker::PhantomData,+ })+ }+}++impl<S> TensorFormatter for FloatFormatter<S>+where+ S: WithDType + num_traits::Float + std::fmt::Display + std::fmt::LowerExp,+{+ type Elem = S;++ fn fmt<T: std::fmt::Write>(&self, v: Self::Elem, max_w: usize, f: &mut T) -> std::fmt::Result {+ if self.sci_mode {+ write!(+ f,+ "{v:width$.prec$e}",+ v = v,+ width = max_w,+ prec = self.precision+ )+ } else if self.int_mode {+ if v.is_finite() {+ write!(f, "{v:width$.0}.", v = v, width = max_w - 1)+ } else {+ write!(f, "{v:max_w$.0}")+ }+ } else {+ write!(+ f,+ "{v:width$.prec$}",+ v = v,+ width = max_w,+ prec = self.precision+ )+ }+ }+}++struct IntFormatter<S: WithDType> {+ _phantom: std::marker::PhantomData<S>,+}++impl<S: WithDType> IntFormatter<S> {+ fn new() -> Self {+ Self {+ _phantom: std::marker::PhantomData,+ }+ }+}++impl<S> TensorFormatter for IntFormatter<S>+where+ S: WithDType + std::fmt::Display,+{+ type Elem = S;++ fn fmt<T: std::fmt::Write>(&self, v: Self::Elem, max_w: usize, f: &mut T) -> std::fmt::Result {+ write!(f, "{v:max_w$}")+ }+}++fn get_summarized_data(t: &Tensor, edge_items: usize) -> Result<Tensor> {+ let dims = t.dims();+ if dims.is_empty() {+ Ok(t.clone())+ } else if dims.len() == 1 {+ if dims[0] > 2 * edge_items {+ Tensor::cat(+ &[+ t.narrow(0, 0, edge_items)?,+ t.narrow(0, dims[0] - edge_items, edge_items)?,+ ],+ 0,+ )+ } else {+ Ok(t.clone())+ }+ } else if dims[0] > 2 * edge_items {+ let mut vs: Vec<_> = (0..edge_items)+ .map(|i| get_summarized_data(&t.get(i)?, edge_items))+ .collect::<Result<Vec<_>>>()?;+ for i in (dims[0] - edge_items)..dims[0] {+ vs.push(get_summarized_data(&t.get(i)?, edge_items)?)+ }+ Tensor::cat(&vs, 0)+ } else {+ let vs: Vec<_> = (0..dims[0])+ .map(|i| get_summarized_data(&t.get(i)?, edge_items))+ .collect::<Result<Vec<_>>>()?;+ Tensor::cat(&vs, 0)+ }+}++impl std::fmt::Display for Tensor {+ fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {+ let po = PRINT_OPTS.lock().unwrap();+ let summarize = self.elem_count() > po.threshold;+ let to_display = if summarize {+ match get_summarized_data(self, po.edge_items) {+ Ok(v) => v,+ Err(err) => return write!(f, "{err:?}"),+ }+ } else {+ self.clone()+ };+ match self.dtype() {+ DType::U8 => {+ let tf: IntFormatter<u8> = IntFormatter::new();+ let max_w = tf.max_width(&to_display);+ tf.fmt_tensor(self, 1, max_w, summarize, &po, f)?;+ writeln!(f)?;+ }+ DType::U32 => {+ let tf: IntFormatter<u32> = IntFormatter::new();+ let max_w = tf.max_width(&to_display);+ tf.fmt_tensor(self, 1, max_w, summarize, &po, f)?;+ writeln!(f)?;+ }+ DType::I16 => {+ let tf: IntFormatter<i16> = IntFormatter::new();+ let max_w = tf.max_width(&to_display);+ tf.fmt_tensor(self, 1, max_w, summarize, &po, f)?;+ writeln!(f)?;+ }+ DType::I32 => {+ let tf: IntFormatter<i32> = IntFormatter::new();+ let max_w = tf.max_width(&to_display);+ tf.fmt_tensor(self, 1, max_w, summarize, &po, f)?;+ writeln!(f)?;+ }+ DType::I64 => {+ let tf: IntFormatter<i64> = IntFormatter::new();+ let max_w = tf.max_width(&to_display);+ tf.fmt_tensor(self, 1, max_w, summarize, &po, f)?;+ writeln!(f)?;+ }+ DType::BF16 => {+ if let Ok(tf) = FloatFormatter::<bf16>::new(&to_display, &po) {+ let max_w = tf.max_width(&to_display);+ tf.fmt_tensor(self, 1, max_w, summarize, &po, f)?;+ writeln!(f)?;+ }+ }+ DType::F16 => {+ if let Ok(tf) = FloatFormatter::<f16>::new(&to_display, &po) {+ let max_w = tf.max_width(&to_display);+ tf.fmt_tensor(self, 1, max_w, summarize, &po, f)?;+ writeln!(f)?;+ }+ }+ DType::F64 => {+ if let Ok(tf) = FloatFormatter::<f64>::new(&to_display, &po) {+ let max_w = tf.max_width(&to_display);+ tf.fmt_tensor(self, 1, max_w, summarize, &po, f)?;+ writeln!(f)?;+ }+ }+ DType::F32 => {+ if let Ok(tf) = FloatFormatter::<f32>::new(&to_display, &po) {+ let max_w = tf.max_width(&to_display);+ tf.fmt_tensor(self, 1, max_w, summarize, &po, f)?;+ writeln!(f)?;+ }+ }+ DType::F8E4M3 => {+ if let Ok(tf) = FloatFormatter::<float8::F8E4M3>::new(&to_display, &po) {+ let max_w = tf.max_width(&to_display);+ tf.fmt_tensor(self, 1, max_w, summarize, &po, f)?;+ writeln!(f)?;+ }+ }+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ writeln!(+ f,+ "Dummy type {} (not supported for display)",+ self.dtype().as_str()+ )?;+ }+ };++ let device_str = match self.device().location() {+ crate::DeviceLocation::Cpu => "".to_owned(),+ crate::DeviceLocation::Cuda { gpu_id } => {+ format!(", cuda:{gpu_id}")+ }+ crate::DeviceLocation::Metal { gpu_id } => {+ format!(", metal:{gpu_id}")+ }+ };++ write!(+ f,+ "Tensor[{:?}, {}{}]",+ self.dims(),+ self.dtype().as_str(),+ device_str+ )+ }+}diff --git a/vendor/candle-core/src/dtype.rs b/vendor/candle-core/src/dtype.rsnew file mode 100644index 0000000..f6ac3c8--- /dev/null+++ b/vendor/candle-core/src/dtype.rs@@ -0,0 +1,295 @@+//! Types for elements that can be stored and manipulated using tensors.+#![allow(clippy::redundant_closure_call)]+use crate::backend::BackendStorage;+use crate::{CpuStorage, CpuStorageRef, Error, Result};++/// The different types of elements allowed in tensors.+#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]+#[non_exhaustive]+pub enum DType {+ // Unsigned 8 bits integer.+ U8,+ // Unsigned 32 bits integer.+ U32,+ // Signed 16 bits integer.+ I16,+ // Signed 32 bits integer.+ I32,+ // Signed 64 bits integer.+ I64,+ // Brain floating-point using half precision (16 bits).+ BF16,+ // Floating-point using half precision (16 bits).+ F16,+ // Floating-point using single precision (32 bits).+ F32,+ // Floating-point using double precision (64 bits).+ F64,+ // 8-bit floating point with 4-bit exponent and 3-bit mantissa.+ F8E4M3,+ /// 6-bit float with 2 exponent bits and 3 mantissa bits (MX6 format)+ F6E2M3,+ /// 6-bit float with 3 exponent bits and 2 mantissa bits (MX6 format)+ F6E3M2,+ /// 4-bit float (MX4 format)+ F4,+ /// 8-bit float with 8 exponent bits and 0 mantissa bits+ F8E8M0,+}++#[derive(Debug, PartialEq, Eq)]+pub struct DTypeParseError(String);++impl std::fmt::Display for DTypeParseError {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ write!(f, "cannot parse '{}' as a dtype", self.0)+ }+}++impl std::error::Error for DTypeParseError {}++impl std::str::FromStr for DType {+ type Err = DTypeParseError;+ fn from_str(s: &str) -> std::result::Result<Self, Self::Err> {+ match s {+ "u8" => Ok(Self::U8),+ "u32" => Ok(Self::U32),+ "i16" => Ok(Self::I16),+ "i32" => Ok(Self::I32),+ "i64" => Ok(Self::I64),+ "bf16" => Ok(Self::BF16),+ "f16" => Ok(Self::F16),+ "f32" => Ok(Self::F32),+ "f64" => Ok(Self::F64),+ "f8e4m3" => Ok(Self::F8E4M3),+ "f6e2m3" => Ok(Self::F6E2M3),+ "f6e3m2" => Ok(Self::F6E3M2),+ "f4" => Ok(Self::F4),+ "f8e8m0" => Ok(Self::F8E8M0),+ _ => Err(DTypeParseError(s.to_string())),+ }+ }+}++impl DType {+ /// String representation for dtypes.+ pub fn as_str(&self) -> &'static str {+ match self {+ Self::U8 => "u8",+ Self::U32 => "u32",+ Self::I16 => "i16",+ Self::I32 => "i32",+ Self::I64 => "i64",+ Self::BF16 => "bf16",+ Self::F16 => "f16",+ Self::F32 => "f32",+ Self::F64 => "f64",+ Self::F8E4M3 => "f8e4m3",+ Self::F6E2M3 => "f6e2m3",+ Self::F6E3M2 => "f6e3m2",+ Self::F4 => "f4",+ Self::F8E8M0 => "f8e8m0",+ }+ }++ /// The size used by each element in bytes, i.e. 1 for `U8`, 4 for `F32`.+ pub fn size_in_bytes(&self) -> usize {+ match self {+ Self::U8 => 1,+ Self::U32 => 4,+ Self::I16 => 2,+ Self::I32 => 4,+ Self::I64 => 8,+ Self::BF16 => 2,+ Self::F16 => 2,+ Self::F32 => 4,+ Self::F64 => 8,+ Self::F8E4M3 => 1,+ Self::F6E2M3 => 0, // 6 bits+ Self::F6E3M2 => 0, // 6 bits+ Self::F4 => 0, // 4 bits+ Self::F8E8M0 => 1,+ }+ }++ pub fn is_int(&self) -> bool {+ match self {+ Self::U8 | Self::U32 | Self::I16 | Self::I32 | Self::I64 => true,+ Self::BF16+ | Self::F16+ | Self::F32+ | Self::F64+ | Self::F8E4M3+ | Self::F6E2M3+ | Self::F6E3M2+ | Self::F4+ | Self::F8E8M0 => false,+ }+ }++ pub fn is_float(&self) -> bool {+ match self {+ Self::U8 | Self::U32 | Self::I16 | Self::I32 | Self::I64 => false,+ Self::BF16+ | Self::F16+ | Self::F32+ | Self::F64+ | Self::F8E4M3+ | Self::F6E2M3+ | Self::F6E3M2+ | Self::F4+ | Self::F8E8M0 => true,+ }+ }+}++pub trait WithDType:+ Sized+ + Copy+ + num_traits::NumAssign+ + std::cmp::PartialOrd+ + std::fmt::Display+ + 'static+ + Send+ + Sync+ + std::any::Any+ + crate::cpu::kernels::VecOps+{+ const DTYPE: DType;++ fn from_f64(v: f64) -> Self;+ fn to_f64(self) -> f64;+ fn to_scalar(self) -> crate::scalar::Scalar;+ fn cpu_storage_ref(data: &[Self]) -> CpuStorageRef<'_>;+ fn to_cpu_storage_owned(data: Vec<Self>) -> CpuStorage;++ fn to_cpu_storage(data: &[Self]) -> CpuStorage {+ Self::to_cpu_storage_owned(data.to_vec())+ }++ fn cpu_storage_as_slice(s: &CpuStorage) -> Result<&[Self]>;+ fn cpu_storage_data(s: CpuStorage) -> Result<Vec<Self>>;+}++macro_rules! with_dtype {+ ($ty:ty, $dtype:ident, $from_f64:expr, $to_f64:expr) => {+ impl WithDType for $ty {+ const DTYPE: DType = DType::$dtype;++ fn from_f64(v: f64) -> Self {+ $from_f64(v)+ }++ fn to_f64(self) -> f64 {+ $to_f64(self)+ }++ fn to_scalar(self) -> crate::scalar::Scalar {+ crate::scalar::Scalar::$dtype(self)+ }++ fn cpu_storage_ref(data: &[Self]) -> CpuStorageRef<'_> {+ CpuStorageRef::$dtype(data)+ }++ fn to_cpu_storage_owned(data: Vec<Self>) -> CpuStorage {+ CpuStorage::$dtype(data)+ }++ fn cpu_storage_data(s: CpuStorage) -> Result<Vec<Self>> {+ match s {+ CpuStorage::$dtype(data) => Ok(data),+ _ => Err(Error::UnexpectedDType {+ expected: DType::$dtype,+ got: s.dtype(),+ msg: "unexpected dtype",+ }+ .bt()),+ }+ }++ fn cpu_storage_as_slice(s: &CpuStorage) -> Result<&[Self]> {+ match s {+ CpuStorage::$dtype(data) => Ok(data),+ _ => Err(Error::UnexpectedDType {+ expected: DType::$dtype,+ got: s.dtype(),+ msg: "unexpected dtype",+ }+ .bt()),+ }+ }+ }+ };+}+use float8::F8E4M3 as f8e4m3;+use half::{bf16, f16};++with_dtype!(u8, U8, |v: f64| v as u8, |v: u8| v as f64);+with_dtype!(u32, U32, |v: f64| v as u32, |v: u32| v as f64);+with_dtype!(i16, I16, |v: f64| v as i16, |v: i16| v as f64);+with_dtype!(i32, I32, |v: f64| v as i32, |v: i32| v as f64);+with_dtype!(i64, I64, |v: f64| v as i64, |v: i64| v as f64);+with_dtype!(f16, F16, f16::from_f64, f16::to_f64);+with_dtype!(bf16, BF16, bf16::from_f64, bf16::to_f64);+with_dtype!(f32, F32, |v: f64| v as f32, |v: f32| v as f64);+with_dtype!(f64, F64, |v: f64| v, |v: f64| v);+with_dtype!(f8e4m3, F8E4M3, f8e4m3::from_f64, |v: f8e4m3| v.to_f64());++pub trait IntDType: WithDType + num_traits::Bounded {+ fn is_true(&self) -> bool;+ fn as_usize(&self) -> usize;+}++impl IntDType for i64 {+ fn is_true(&self) -> bool {+ *self != 0+ }+ fn as_usize(&self) -> usize {+ *self as usize+ }+}++impl IntDType for u32 {+ fn is_true(&self) -> bool {+ *self != 0+ }+ fn as_usize(&self) -> usize {+ *self as usize+ }+}++impl IntDType for u8 {+ fn is_true(&self) -> bool {+ *self != 0+ }+ fn as_usize(&self) -> usize {+ *self as usize+ }+}++impl IntDType for i16 {+ fn is_true(&self) -> bool {+ *self != 0+ }+ fn as_usize(&self) -> usize {+ *self as usize+ }+}++impl IntDType for i32 {+ fn is_true(&self) -> bool {+ *self != 0+ }+ fn as_usize(&self) -> usize {+ *self as usize+ }+}++pub trait FloatDType: WithDType {}++impl FloatDType for f16 {}+impl FloatDType for bf16 {}+impl FloatDType for f32 {}+impl FloatDType for f64 {}+impl FloatDType for f8e4m3 {}diff --git a/vendor/candle-core/src/dummy_cuda_backend.rs b/vendor/candle-core/src/dummy_cuda_backend.rsnew file mode 100644index 0000000..2aa4585--- /dev/null+++ b/vendor/candle-core/src/dummy_cuda_backend.rs@@ -0,0 +1,317 @@+//! Implementation of the Cuda backend when Cuda support has not been compiled in.+//!+#![allow(dead_code)]+use crate::op::{BinaryOpT, CmpOp, ReduceOp, UnaryOpT};+use crate::{CpuStorage, DType, Error, Layout, Result, Shape};++#[derive(Debug, Clone)]+pub struct CudaDevice;++#[derive(Debug)]+pub struct CudaStorage;++impl CudaStorage {+ pub fn transfer_to_device(&self, _dst: &CudaDevice) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }+}++macro_rules! fail {+ () => {+ unimplemented!("cuda support has not been enabled, add `cuda` feature to enable.")+ };+}+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]+pub struct DeviceId(usize);++impl CudaDevice {+ pub fn new_with_stream(_: usize) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }+ pub fn id(&self) -> DeviceId {+ DeviceId(0)+ }+}++impl crate::backend::BackendStorage for CudaStorage {+ type Device = CudaDevice;++ fn try_clone(&self, _: &Layout) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn dtype(&self) -> DType {+ fail!()+ }++ fn device(&self) -> &Self::Device {+ fail!()+ }++ fn const_set(&mut self, _: crate::scalar::Scalar, _: &Layout) -> Result<()> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn to_cpu_storage(&self) -> Result<CpuStorage> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn affine(&self, _: &Layout, _: f64, _: f64) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn powf(&self, _: &Layout, _: f64) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn elu(&self, _: &Layout, _: f64) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn reduce_op(&self, _: ReduceOp, _: &Layout, _: &[usize]) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn cmp(&self, _: CmpOp, _: &Self, _: &Layout, _: &Layout) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn to_dtype(&self, _: &Layout, _: DType) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn unary_impl<B: UnaryOpT>(&self, _: &Layout) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn binary_impl<B: BinaryOpT>(&self, _: &Self, _: &Layout, _: &Layout) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn where_cond(&self, _: &Layout, _: &Self, _: &Layout, _: &Self, _: &Layout) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn conv1d(+ &self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: &crate::conv::ParamsConv1D,+ ) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn conv_transpose1d(+ &self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: &crate::conv::ParamsConvTranspose1D,+ ) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn conv2d(+ &self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: &crate::conv::ParamsConv2D,+ ) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn conv_transpose2d(+ &self,+ _l: &Layout,+ _kernel: &Self,+ _kernel_l: &Layout,+ _params: &crate::conv::ParamsConvTranspose2D,+ ) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn index_select(&self, _: &Self, _: &Layout, _: &Layout, _: usize) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }+ fn gather(&self, _: &Layout, _: &Self, _: &Layout, _: usize) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn scatter_set(+ &mut self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: usize,+ ) -> Result<()> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn scatter_add_set(+ &mut self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: usize,+ ) -> Result<()> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn index_add(+ &self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: usize,+ ) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn matmul(+ &self,+ _: &Self,+ _: (usize, usize, usize, usize),+ _: &Layout,+ _: &Layout,+ ) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn copy_strided_src(&self, _: &mut Self, _: usize, _: &Layout) -> Result<()> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn copy2d(+ &self,+ _: &mut Self,+ _: usize,+ _: usize,+ _: usize,+ _: usize,+ _: usize,+ _: usize,+ ) -> Result<()> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn avg_pool2d(&self, _: &Layout, _: (usize, usize), _: (usize, usize)) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn max_pool2d(&self, _: &Layout, _: (usize, usize), _: (usize, usize)) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn upsample_nearest1d(&self, _: &Layout, _: usize) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn upsample_nearest2d(&self, _: &Layout, _: usize, _: usize) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn upsample_bilinear2d(+ &self,+ _: &Layout,+ _: usize,+ _: usize,+ _: bool,+ _: Option<f64>,+ _: Option<f64>,+ ) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }+}++impl crate::backend::BackendDevice for CudaDevice {+ type Storage = CudaStorage;+ fn new(_: usize) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn set_seed(&self, _: u64) -> Result<()> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn get_current_seed(&self) -> Result<u64> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn location(&self) -> crate::DeviceLocation {+ fail!()+ }++ fn same_device(&self, _: &Self) -> bool {+ fail!()+ }++ fn zeros_impl(&self, _shape: &Shape, _dtype: DType) -> Result<Self::Storage> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ unsafe fn alloc_uninit(&self, _shape: &Shape, _dtype: DType) -> Result<Self::Storage> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn storage_from_slice<T: crate::WithDType>(&self, _: &[T]) -> Result<Self::Storage> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn storage_from_cpu_storage(&self, _: &CpuStorage) -> Result<Self::Storage> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn storage_from_cpu_storage_owned(&self, _: CpuStorage) -> Result<Self::Storage> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn rand_uniform(&self, _: &Shape, _: DType, _: f64, _: f64) -> Result<Self::Storage> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn rand_normal(&self, _: &Shape, _: DType, _: f64, _: f64) -> Result<Self::Storage> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ fn synchronize(&self) -> Result<()> {+ Ok(())+ }+}++/// This bool controls whether reduced precision reductions (e.g., with fp16 accumulation type) are+/// allowed with f16 GEMMs.+pub fn gemm_reduced_precision_f16() -> bool {+ true+}++/// This bool controls whether reduced precision reductions (e.g., with fp16 accumulation type) are+/// allowed with f16 GEMMs.+pub fn set_gemm_reduced_precision_f16(_: bool) {}++/// This bool controls whether reduced precision reductions (e.g., with fp16 accumulation type) are+/// allowed with bf16 GEMMs.+pub fn gemm_reduced_precision_bf16() -> bool {+ true+}++/// This bool controls whether reduced precision reductions (e.g., with fp16 accumulation type) are+/// allowed with bf16 GEMMs.+pub fn set_gemm_reduced_precision_bf16(_: bool) {}++/// This bool controls whether reduced precision reductions (e.g., with tf32 accumulation type) are+/// allowed with f32 GEMMs.+pub fn gemm_reduced_precision_f32() -> bool {+ true+}++/// This bool controls whether reduced precision reductions (e.g., with tf32 accumulation type) are+/// allowed with f32 GEMMs.+pub fn set_gemm_reduced_precision_f32(_b: bool) {}diff --git a/vendor/candle-core/src/dummy_dtype.rs b/vendor/candle-core/src/dummy_dtype.rsnew file mode 100644index 0000000..5fdb096--- /dev/null+++ b/vendor/candle-core/src/dummy_dtype.rs@@ -0,0 +1,268 @@+//! Dummy data types for experimental/future float formats+//!+//! These are placeholder types for experimental floating-point formats+//! that are defined in the safetensors spec but not yet fully implemented.++use crate::{DType, Error, Result, WithDType};++/// 6-bit float with 2 exponent bits and 3 mantissa bits (MX6 format)+/// This is a dummy type.+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]+pub struct F6E2M3;++/// 6-bit float with 3 exponent bits and 2 mantissa bits (MX6 format)+/// This is a dummy type.+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]+pub struct F6E3M2;++/// 4-bit float (MX4 format)+/// This is a dummy type.+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]+pub struct F4;++/// 8-bit float with 8 exponent bits and 0 mantissa bits+/// This is a dummy type.+#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]+pub struct F8E8M0;++// Implement WithDType for dummy types+macro_rules! dummy_with_dtype {+ ($ty:ty, $dtype:ident) => {+ impl WithDType for $ty {+ const DTYPE: DType = DType::$dtype;++ fn from_f64(_v: f64) -> Self {+ panic!(+ "{} is a dummy type and cannot be constructed",+ stringify!($ty)+ )+ }++ fn to_f64(self) -> f64 {+ panic!(+ "{} is a dummy type and cannot be converted",+ stringify!($ty)+ )+ }++ fn to_scalar(self) -> crate::scalar::Scalar {+ panic!(+ "{} is a dummy type and cannot be converted to scalar",+ stringify!($ty)+ )+ }++ fn cpu_storage_ref(_data: &[Self]) -> crate::CpuStorageRef<'_> {+ panic!(+ "{} is a dummy type and does not support storage",+ stringify!($ty)+ )+ }++ fn to_cpu_storage_owned(_data: Vec<Self>) -> crate::CpuStorage {+ panic!(+ "{} is a dummy type and does not support storage",+ stringify!($ty)+ )+ }++ fn cpu_storage_data(_s: crate::CpuStorage) -> Result<Vec<Self>> {+ Err(Error::UnsupportedDTypeForOp(DType::$dtype, "cpu_storage_data").bt())+ }++ fn cpu_storage_as_slice(_s: &crate::CpuStorage) -> Result<&[Self]> {+ Err(Error::UnsupportedDTypeForOp(DType::$dtype, "cpu_storage_as_slice").bt())+ }+ }+ };+}++dummy_with_dtype!(F6E2M3, F6E2M3);+dummy_with_dtype!(F6E3M2, F6E3M2);+dummy_with_dtype!(F4, F4);+dummy_with_dtype!(F8E8M0, F8E8M0);++// Implement NumAssign traits for dummy types+macro_rules! dummy_num_assign {+ ($ty:ty) => {+ impl std::ops::AddAssign for $ty {+ fn add_assign(&mut self, _other: Self) {+ panic!(+ "{} is a dummy type and does not support operations",+ stringify!($ty)+ )+ }+ }++ impl std::ops::SubAssign for $ty {+ fn sub_assign(&mut self, _other: Self) {+ panic!(+ "{} is a dummy type and does not support operations",+ stringify!($ty)+ )+ }+ }++ impl std::ops::MulAssign for $ty {+ fn mul_assign(&mut self, _other: Self) {+ panic!(+ "{} is a dummy type and does not support operations",+ stringify!($ty)+ )+ }+ }++ impl std::ops::DivAssign for $ty {+ fn div_assign(&mut self, _other: Self) {+ panic!(+ "{} is a dummy type and does not support operations",+ stringify!($ty)+ )+ }+ }++ impl std::ops::RemAssign for $ty {+ fn rem_assign(&mut self, _other: Self) {+ panic!(+ "{} is a dummy type and does not support operations",+ stringify!($ty)+ )+ }+ }++ impl std::ops::Add for $ty {+ type Output = Self;+ fn add(self, _other: Self) -> Self {+ panic!(+ "{} is a dummy type and does not support operations",+ stringify!($ty)+ )+ }+ }++ impl std::ops::Sub for $ty {+ type Output = Self;+ fn sub(self, _other: Self) -> Self {+ panic!(+ "{} is a dummy type and does not support operations",+ stringify!($ty)+ )+ }+ }++ impl std::ops::Mul for $ty {+ type Output = Self;+ fn mul(self, _other: Self) -> Self {+ panic!(+ "{} is a dummy type and does not support operations",+ stringify!($ty)+ )+ }+ }++ impl std::ops::Div for $ty {+ type Output = Self;+ fn div(self, _other: Self) -> Self {+ panic!(+ "{} is a dummy type and does not support operations",+ stringify!($ty)+ )+ }+ }++ impl std::ops::Rem for $ty {+ type Output = Self;+ fn rem(self, _other: Self) -> Self {+ panic!(+ "{} is a dummy type and does not support operations",+ stringify!($ty)+ )+ }+ }++ impl num_traits::Zero for $ty {+ fn zero() -> Self {+ panic!(+ "{} is a dummy type and does not support operations",+ stringify!($ty)+ )+ }++ fn is_zero(&self) -> bool {+ panic!(+ "{} is a dummy type and does not support operations",+ stringify!($ty)+ )+ }+ }++ impl num_traits::One for $ty {+ fn one() -> Self {+ panic!(+ "{} is a dummy type and does not support operations",+ stringify!($ty)+ )+ }+ }++ impl num_traits::Num for $ty {+ type FromStrRadixErr = std::num::ParseFloatError;++ fn from_str_radix(+ _str: &str,+ _radix: u32,+ ) -> std::result::Result<Self, Self::FromStrRadixErr> {+ panic!(+ "{} is a dummy type and does not support parsing",+ stringify!($ty)+ )+ }+ }++ impl crate::cpu::kernels::VecOps for $ty {+ fn min(self, _other: Self) -> Self {+ panic!(+ "{} is a dummy type and does not support operations",+ stringify!($ty)+ )+ }++ fn max(self, _other: Self) -> Self {+ panic!(+ "{} is a dummy type and does not support operations",+ stringify!($ty)+ )+ }+ }+ };+}++dummy_num_assign!(F6E2M3);+dummy_num_assign!(F6E3M2);+dummy_num_assign!(F4);+dummy_num_assign!(F8E8M0);++// Display implementations+impl std::fmt::Display for F6E2M3 {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ write!(f, "F6E2M3")+ }+}++impl std::fmt::Display for F6E3M2 {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ write!(f, "F6E3M2")+ }+}++impl std::fmt::Display for F4 {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ write!(f, "F4")+ }+}++impl std::fmt::Display for F8E8M0 {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ write!(f, "F8E8M0")+ }+}diff --git a/vendor/candle-core/src/dummy_metal_backend.rs b/vendor/candle-core/src/dummy_metal_backend.rsnew file mode 100644index 0000000..8c23b58--- /dev/null+++ b/vendor/candle-core/src/dummy_metal_backend.rs@@ -0,0 +1,280 @@+#![allow(dead_code)]+use crate::op::{BinaryOpT, CmpOp, ReduceOp, UnaryOpT};+use crate::{CpuStorage, DType, Error, Layout, Result, Shape};++#[derive(Debug, Clone)]+pub struct MetalDevice;++#[derive(Debug)]+pub struct MetalStorage;++#[derive(thiserror::Error, Debug)]+pub enum MetalError {+ #[error("{0}")]+ Message(String),+}++impl From<String> for MetalError {+ fn from(e: String) -> Self {+ MetalError::Message(e)+ }+}++macro_rules! fail {+ () => {+ unimplemented!("metal support has not been enabled, add `metal` feature to enable.")+ };+}++impl crate::backend::BackendStorage for MetalStorage {+ type Device = MetalDevice;++ fn try_clone(&self, _: &Layout) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn dtype(&self) -> DType {+ fail!()+ }++ fn device(&self) -> &Self::Device {+ fail!()+ }++ fn const_set(&mut self, _: crate::scalar::Scalar, _: &Layout) -> Result<()> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn to_cpu_storage(&self) -> Result<CpuStorage> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn affine(&self, _: &Layout, _: f64, _: f64) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn powf(&self, _: &Layout, _: f64) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn elu(&self, _: &Layout, _: f64) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn reduce_op(&self, _: ReduceOp, _: &Layout, _: &[usize]) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn cmp(&self, _: CmpOp, _: &Self, _: &Layout, _: &Layout) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn to_dtype(&self, _: &Layout, _: DType) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn unary_impl<B: UnaryOpT>(&self, _: &Layout) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn binary_impl<B: BinaryOpT>(&self, _: &Self, _: &Layout, _: &Layout) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn where_cond(&self, _: &Layout, _: &Self, _: &Layout, _: &Self, _: &Layout) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn conv1d(+ &self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: &crate::conv::ParamsConv1D,+ ) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn conv_transpose1d(+ &self,+ _l: &Layout,+ _kernel: &Self,+ _kernel_l: &Layout,+ _params: &crate::conv::ParamsConvTranspose1D,+ ) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn conv2d(+ &self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: &crate::conv::ParamsConv2D,+ ) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn conv_transpose2d(+ &self,+ _l: &Layout,+ _kernel: &Self,+ _kernel_l: &Layout,+ _params: &crate::conv::ParamsConvTranspose2D,+ ) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn index_select(&self, _: &Self, _: &Layout, _: &Layout, _: usize) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }+ fn gather(&self, _: &Layout, _: &Self, _: &Layout, _: usize) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn scatter_set(+ &mut self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: usize,+ ) -> Result<()> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn scatter_add_set(+ &mut self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: usize,+ ) -> Result<()> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn index_add(+ &self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: &Self,+ _: &Layout,+ _: usize,+ ) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn matmul(+ &self,+ _: &Self,+ _: (usize, usize, usize, usize),+ _: &Layout,+ _: &Layout,+ ) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn copy_strided_src(&self, _: &mut Self, _: usize, _: &Layout) -> Result<()> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn copy2d(+ &self,+ _: &mut Self,+ _: usize,+ _: usize,+ _: usize,+ _: usize,+ _: usize,+ _: usize,+ ) -> Result<()> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn avg_pool2d(&self, _: &Layout, _: (usize, usize), _: (usize, usize)) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn max_pool2d(&self, _: &Layout, _: (usize, usize), _: (usize, usize)) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn upsample_nearest1d(&self, _: &Layout, _: usize) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn upsample_nearest2d(&self, _: &Layout, _: usize, _: usize) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn upsample_bilinear2d(+ &self,+ _: &Layout,+ _: usize,+ _: usize,+ _: bool,+ _: Option<f64>,+ _: Option<f64>,+ ) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }+}++impl crate::backend::BackendDevice for MetalDevice {+ type Storage = MetalStorage;+ fn new(_: usize) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn set_seed(&self, _: u64) -> Result<()> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn get_current_seed(&self) -> Result<u64> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn location(&self) -> crate::DeviceLocation {+ fail!()+ }++ fn same_device(&self, _: &Self) -> bool {+ fail!()+ }++ fn zeros_impl(&self, _shape: &Shape, _dtype: DType) -> Result<Self::Storage> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ unsafe fn alloc_uninit(&self, _shape: &Shape, _dtype: DType) -> Result<Self::Storage> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn storage_from_slice<T: crate::WithDType>(&self, _: &[T]) -> Result<Self::Storage> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn storage_from_cpu_storage(&self, _: &CpuStorage) -> Result<Self::Storage> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn storage_from_cpu_storage_owned(&self, _: CpuStorage) -> Result<Self::Storage> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn rand_uniform(&self, _: &Shape, _: DType, _: f64, _: f64) -> Result<Self::Storage> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn rand_normal(&self, _: &Shape, _: DType, _: f64, _: f64) -> Result<Self::Storage> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ fn synchronize(&self) -> Result<()> {+ Ok(())+ }+}diff --git a/vendor/candle-core/src/error.rs b/vendor/candle-core/src/error.rsnew file mode 100644index 0000000..c5a1f95--- /dev/null+++ b/vendor/candle-core/src/error.rs@@ -0,0 +1,390 @@+//! Candle-specific Error and Result+use std::{convert::Infallible, fmt::Display};++use crate::{DType, DeviceLocation, Layout, MetalError, Shape};++#[derive(Debug, Clone)]+pub struct MatMulUnexpectedStriding {+ pub lhs_l: Layout,+ pub rhs_l: Layout,+ pub bmnk: (usize, usize, usize, usize),+ pub msg: &'static str,+}++impl std::fmt::Debug for Error {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ write!(f, "{self}")+ }+}++/// Main library error type.+#[derive(thiserror::Error)]+pub enum Error {+ // === DType Errors ===+ #[error("{msg}, expected: {expected:?}, got: {got:?}")]+ UnexpectedDType {+ msg: &'static str,+ expected: DType,+ got: DType,+ },++ #[error("dtype mismatch in {op}, lhs: {lhs:?}, rhs: {rhs:?}")]+ DTypeMismatchBinaryOp {+ lhs: DType,+ rhs: DType,+ op: &'static str,+ },++ #[error("unsupported dtype {0:?} for op {1}")]+ UnsupportedDTypeForOp(DType, &'static str),++ // === Dimension Index Errors ===+ #[error("{op}: dimension index {dim} out of range for shape {shape:?}")]+ DimOutOfRange {+ shape: Shape,+ dim: i32,+ op: &'static str,+ },++ #[error("{op}: duplicate dim index {dims:?} for shape {shape:?}")]+ DuplicateDimIndex {+ shape: Shape,+ dims: Vec<usize>,+ op: &'static str,+ },++ // === Shape Errors ===+ #[error("unexpected rank, expected: {expected}, got: {got} ({shape:?})")]+ UnexpectedNumberOfDims {+ expected: usize,+ got: usize,+ shape: Shape,+ },++ #[error("{msg}, expected: {expected:?}, got: {got:?}")]+ UnexpectedShape {+ msg: String,+ expected: Shape,+ got: Shape,+ },++ #[error(+ "Shape mismatch, got buffer of size {buffer_size} which is compatible with shape {shape:?}"+ )]+ ShapeMismatch { buffer_size: usize, shape: Shape },++ #[error("shape mismatch in {op}, lhs: {lhs:?}, rhs: {rhs:?}")]+ ShapeMismatchBinaryOp {+ lhs: Shape,+ rhs: Shape,+ op: &'static str,+ },++ #[error("shape mismatch in cat for dim {dim}, shape for arg 1: {first_shape:?} shape for arg {n}: {nth_shape:?}")]+ ShapeMismatchCat {+ dim: usize,+ first_shape: Shape,+ n: usize,+ nth_shape: Shape,+ },++ #[error("Cannot divide tensor of shape {shape:?} equally along dim {dim} into {n_parts}")]+ ShapeMismatchSplit {+ shape: Shape,+ dim: usize,+ n_parts: usize,+ },++ #[error("{op} can only be performed on a single dimension")]+ OnlySingleDimension { op: &'static str, dims: Vec<usize> },++ #[error("empty tensor for {op}")]+ EmptyTensor { op: &'static str },++ // === Device Errors ===+ #[error("device mismatch in {op}, lhs: {lhs:?}, rhs: {rhs:?}")]+ DeviceMismatchBinaryOp {+ lhs: DeviceLocation,+ rhs: DeviceLocation,+ op: &'static str,+ },++ // === Op Specific Errors ===+ #[error("narrow invalid args {msg}: {shape:?}, dim: {dim}, start: {start}, len:{len}")]+ NarrowInvalidArgs {+ shape: Shape,+ dim: usize,+ start: usize,+ len: usize,+ msg: &'static str,+ },++ #[error("conv1d invalid args {msg}: inp: {inp_shape:?}, k: {k_shape:?}, pad: {padding}, stride: {stride}")]+ Conv1dInvalidArgs {+ inp_shape: Shape,+ k_shape: Shape,+ padding: usize,+ stride: usize,+ msg: &'static str,+ },++ #[error("{op} invalid index {index} with dim size {size}")]+ InvalidIndex {+ op: &'static str,+ index: usize,+ size: usize,+ },++ #[error("cannot broadcast {src_shape:?} to {dst_shape:?}")]+ BroadcastIncompatibleShapes { src_shape: Shape, dst_shape: Shape },++ #[error("cannot set variable {msg}")]+ CannotSetVar { msg: &'static str },++ // Box indirection to avoid large variant.+ #[error("{0:?}")]+ MatMulUnexpectedStriding(Box<MatMulUnexpectedStriding>),++ #[error("{op} only supports contiguous tensors")]+ RequiresContiguous { op: &'static str },++ #[error("{op} expects at least one tensor")]+ OpRequiresAtLeastOneTensor { op: &'static str },++ #[error("{op} expects at least two tensors")]+ OpRequiresAtLeastTwoTensors { op: &'static str },++ #[error("backward is not supported for {op}")]+ BackwardNotSupported { op: &'static str },++ // === Other Errors ===+ #[error("the candle crate has not been built with cuda support")]+ NotCompiledWithCudaSupport,++ #[error("the candle crate has not been built with metal support")]+ NotCompiledWithMetalSupport,++ #[error("cannot find tensor {path}")]+ CannotFindTensor { path: String },++ // === Wrapped Errors ===+ #[error(transparent)]+ Cuda(Box<dyn std::error::Error + Send + Sync>),++ #[error("Metal error {0}")]+ Metal(#[from] MetalError),++ #[error(transparent)]+ TryFromIntError(#[from] core::num::TryFromIntError),++ #[error("npy/npz error {0}")]+ Npy(String),++ /// Zip file format error.+ #[error(transparent)]+ Zip(#[from] zip::result::ZipError),++ /// Integer parse error.+ #[error(transparent)]+ ParseInt(#[from] std::num::ParseIntError),++ /// Utf8 parse error.+ #[error(transparent)]+ FromUtf8(#[from] std::string::FromUtf8Error),++ /// I/O error.+ #[error(transparent)]+ Io(#[from] std::io::Error),++ /// SafeTensor error.+ #[error(transparent)]+ SafeTensor(#[from] safetensors::SafeTensorError),++ #[error("unsupported safetensor dtype {0:?}")]+ UnsupportedSafeTensorDtype(safetensors::Dtype),++ /// Arbitrary errors wrapping.+ #[error("{0}")]+ Wrapped(Box<dyn std::fmt::Display + Send + Sync>),++ /// Arbitrary errors wrapping with context.+ #[error("{wrapped:?}\n{context:?}")]+ WrappedContext {+ wrapped: Box<dyn std::error::Error + Send + Sync>,+ context: String,+ },++ #[error("{context}\n{inner}")]+ Context {+ inner: Box<Self>,+ context: Box<dyn std::fmt::Display + Send + Sync>,+ },++ /// Adding path information to an error.+ #[error("path: {path:?} {inner}")]+ WithPath {+ inner: Box<Self>,+ path: std::path::PathBuf,+ },++ #[error("{inner}\n{backtrace}")]+ WithBacktrace {+ inner: Box<Self>,+ backtrace: Box<std::backtrace::Backtrace>,+ },++ /// User generated error message, typically created via `bail!`.+ #[error("{0}")]+ Msg(String),++ #[error("unwrap none")]+ UnwrapNone,+}++pub type Result<T> = std::result::Result<T, Error>;++impl Error {+ pub fn wrap(err: impl std::fmt::Display + Send + Sync + 'static) -> Self {+ Self::Wrapped(Box::new(err)).bt()+ }++ pub fn msg(err: impl std::fmt::Display) -> Self {+ Self::Msg(err.to_string()).bt()+ }++ pub fn debug(err: impl std::fmt::Debug) -> Self {+ Self::Msg(format!("{err:?}")).bt()+ }++ pub fn bt(self) -> Self {+ let backtrace = std::backtrace::Backtrace::capture();+ match backtrace.status() {+ std::backtrace::BacktraceStatus::Disabled+ | std::backtrace::BacktraceStatus::Unsupported => self,+ _ => Self::WithBacktrace {+ inner: Box::new(self),+ backtrace: Box::new(backtrace),+ },+ }+ }++ pub fn with_path<P: AsRef<std::path::Path>>(self, p: P) -> Self {+ Self::WithPath {+ inner: Box::new(self),+ path: p.as_ref().to_path_buf(),+ }+ }++ pub fn context(self, c: impl std::fmt::Display + Send + Sync + 'static) -> Self {+ Self::Context {+ inner: Box::new(self),+ context: Box::new(c),+ }+ }+}++#[macro_export]+macro_rules! bail {+ ($msg:literal $(,)?) => {+ return Err($crate::Error::Msg(format!($msg).into()).bt())+ };+ ($err:expr $(,)?) => {+ return Err($crate::Error::Msg(format!($err).into()).bt())+ };+ ($fmt:expr, $($arg:tt)*) => {+ return Err($crate::Error::Msg(format!($fmt, $($arg)*).into()).bt())+ };+}++pub fn zip<T, U>(r1: Result<T>, r2: Result<U>) -> Result<(T, U)> {+ match (r1, r2) {+ (Ok(r1), Ok(r2)) => Ok((r1, r2)),+ (Err(e), _) => Err(e),+ (_, Err(e)) => Err(e),+ }+}++pub(crate) mod private {+ pub trait Sealed {}++ impl<T, E> Sealed for std::result::Result<T, E> where E: std::error::Error {}+ impl<T> Sealed for Option<T> {}+}++/// Attach more context to an error.+///+/// Inspired by [`anyhow::Context`].+pub trait Context<T, E>: private::Sealed {+ /// Wrap the error value with additional context.+ fn context<C>(self, context: C) -> std::result::Result<T, Error>+ where+ C: Display + Send + Sync + 'static;++ /// Wrap the error value with additional context that is evaluated lazily+ /// only once an error does occur.+ fn with_context<C, F>(self, f: F) -> std::result::Result<T, Error>+ where+ C: Display + Send + Sync + 'static,+ F: FnOnce() -> C;+}++impl<T, E> Context<T, E> for std::result::Result<T, E>+where+ E: std::error::Error + Send + Sync + 'static,+{+ fn context<C>(self, context: C) -> std::result::Result<T, Error>+ where+ C: Display + Send + Sync + 'static,+ {+ // Not using map_err to save 2 useless frames off the captured backtrace+ // in ext_context.+ match self {+ Ok(ok) => Ok(ok),+ Err(error) => Err(Error::WrappedContext {+ wrapped: Box::new(error),+ context: context.to_string(),+ }+ .bt()),+ }+ }++ fn with_context<C, F>(self, context: F) -> std::result::Result<T, Error>+ where+ C: Display + Send + Sync + 'static,+ F: FnOnce() -> C,+ {+ match self {+ Ok(ok) => Ok(ok),+ Err(error) => Err(Error::WrappedContext {+ wrapped: Box::new(error),+ context: context().to_string(),+ }+ .bt()),+ }+ }+}++impl<T> Context<T, Infallible> for Option<T> {+ fn context<C>(self, context: C) -> std::result::Result<T, Error>+ where+ C: Display + Send + Sync + 'static,+ {+ // Not using ok_or_else to save 2 useless frames off the captured+ // backtrace.+ match self {+ Some(ok) => Ok(ok),+ None => Err(Error::msg(context).bt()),+ }+ }++ fn with_context<C, F>(self, context: F) -> std::result::Result<T, Error>+ where+ C: Display + Send + Sync + 'static,+ F: FnOnce() -> C,+ {+ match self {+ Some(v) => Ok(v),+ None => Err(Error::UnwrapNone.context(context()).bt()),+ }+ }+}diff --git a/vendor/candle-core/src/indexer.rs b/vendor/candle-core/src/indexer.rsnew file mode 100644index 0000000..d6cd6de--- /dev/null+++ b/vendor/candle-core/src/indexer.rs@@ -0,0 +1,257 @@+use crate::{Error, Tensor};+use std::ops::{+ Bound, Range, RangeBounds, RangeFrom, RangeFull, RangeInclusive, RangeTo, RangeToInclusive,+};++impl Tensor {+ /// Intended to be use by the trait `.i()`+ ///+ /// ```+ /// # use candle_core::{Tensor, DType, Device, IndexOp};+ /// let a = Tensor::zeros((2, 3), DType::F32, &Device::Cpu)?;+ ///+ /// let c = a.i(0..1)?;+ /// assert_eq!(c.shape().dims(), &[1, 3]);+ ///+ /// let c = a.i(0)?;+ /// assert_eq!(c.shape().dims(), &[3]);+ ///+ /// let c = a.i((.., ..2) )?;+ /// assert_eq!(c.shape().dims(), &[2, 2]);+ ///+ /// let c = a.i((.., ..=2))?;+ /// assert_eq!(c.shape().dims(), &[2, 3]);+ ///+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ fn index(&self, indexers: &[TensorIndexer]) -> Result<Self, Error> {+ let mut x = self.clone();+ let dims = self.shape().dims();+ let mut current_dim = 0;+ for (i, indexer) in indexers.iter().enumerate() {+ x = match indexer {+ TensorIndexer::Select(n) => x.narrow(current_dim, *n, 1)?.squeeze(current_dim)?,+ TensorIndexer::Narrow(left_bound, right_bound) => {+ let start = match left_bound {+ Bound::Included(n) => *n,+ Bound::Excluded(n) => *n + 1,+ Bound::Unbounded => 0,+ };+ let stop = match right_bound {+ Bound::Included(n) => *n + 1,+ Bound::Excluded(n) => *n,+ Bound::Unbounded => dims[i],+ };+ let out = x.narrow(current_dim, start, stop.saturating_sub(start))?;+ current_dim += 1;+ out+ }+ TensorIndexer::IndexSelect(indexes) => {+ if indexes.rank() != 1 {+ crate::bail!("multi-dimensional tensor indexing is not supported")+ }+ let out = x.index_select(&indexes.to_device(x.device())?, current_dim)?;+ current_dim += 1;+ out+ }+ TensorIndexer::Err(e) => crate::bail!("indexing error {e:?}"),+ };+ }+ Ok(x)+ }+}++#[derive(Debug)]+/// Generic structure used to index a slice of the tensor+pub enum TensorIndexer {+ /// This selects the elements for which an index has some specific value.+ Select(usize),+ /// This is a regular slice, purely indexing a chunk of the tensor+ Narrow(Bound<usize>, Bound<usize>),+ /// Indexing via a 1d tensor+ IndexSelect(Tensor),+ Err(Error),+}++impl From<usize> for TensorIndexer {+ fn from(index: usize) -> Self {+ TensorIndexer::Select(index)+ }+}++impl From<&[u32]> for TensorIndexer {+ fn from(index: &[u32]) -> Self {+ match Tensor::new(index, &crate::Device::Cpu) {+ Ok(tensor) => TensorIndexer::IndexSelect(tensor),+ Err(e) => TensorIndexer::Err(e),+ }+ }+}++impl From<Vec<u32>> for TensorIndexer {+ fn from(index: Vec<u32>) -> Self {+ let len = index.len();+ match Tensor::from_vec(index, len, &crate::Device::Cpu) {+ Ok(tensor) => TensorIndexer::IndexSelect(tensor),+ Err(e) => TensorIndexer::Err(e),+ }+ }+}++impl From<&Tensor> for TensorIndexer {+ fn from(tensor: &Tensor) -> Self {+ TensorIndexer::IndexSelect(tensor.clone())+ }+}++trait RB: RangeBounds<usize> {}+impl RB for Range<usize> {}+impl RB for RangeFrom<usize> {}+impl RB for RangeFull {}+impl RB for RangeInclusive<usize> {}+impl RB for RangeTo<usize> {}+impl RB for RangeToInclusive<usize> {}++impl<T: RB> From<T> for TensorIndexer {+ fn from(range: T) -> Self {+ use std::ops::Bound::*;+ let start = match range.start_bound() {+ Included(idx) => Included(*idx),+ Excluded(idx) => Excluded(*idx),+ Unbounded => Unbounded,+ };+ let end = match range.end_bound() {+ Included(idx) => Included(*idx),+ Excluded(idx) => Excluded(*idx),+ Unbounded => Unbounded,+ };+ TensorIndexer::Narrow(start, end)+ }+}++/// Trait used to implement multiple signatures for ease of use of the slicing+/// of a tensor+pub trait IndexOp<T> {+ /// Returns a slicing iterator which are the chunks of data necessary to+ /// reconstruct the desired tensor.+ fn i(&self, index: T) -> Result<Tensor, Error>;+}++impl<T> IndexOp<T> for Tensor+where+ T: Into<TensorIndexer>,+{+ ///```rust+ /// use candle_core::{Tensor, DType, Device, IndexOp};+ /// let a = Tensor::new(&[+ /// [0., 1.],+ /// [2., 3.],+ /// [4., 5.]+ /// ], &Device::Cpu)?;+ ///+ /// let b = a.i(0)?;+ /// assert_eq!(b.shape().dims(), &[2]);+ /// assert_eq!(b.to_vec1::<f64>()?, &[0., 1.]);+ ///+ /// let c = a.i(..2)?;+ /// assert_eq!(c.shape().dims(), &[2, 2]);+ /// assert_eq!(c.to_vec2::<f64>()?, &[+ /// [0., 1.],+ /// [2., 3.]+ /// ]);+ ///+ /// let d = a.i(1..)?;+ /// assert_eq!(d.shape().dims(), &[2, 2]);+ /// assert_eq!(d.to_vec2::<f64>()?, &[+ /// [2., 3.],+ /// [4., 5.]+ /// ]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ fn i(&self, index: T) -> Result<Tensor, Error> {+ self.index(&[index.into()])+ }+}++impl<A> IndexOp<(A,)> for Tensor+where+ A: Into<TensorIndexer>,+{+ ///```rust+ /// use candle_core::{Tensor, DType, Device, IndexOp};+ /// let a = Tensor::new(&[+ /// [0f32, 1.],+ /// [2. , 3.],+ /// [4. , 5.]+ /// ], &Device::Cpu)?;+ ///+ /// let b = a.i((0,))?;+ /// assert_eq!(b.shape().dims(), &[2]);+ /// assert_eq!(b.to_vec1::<f32>()?, &[0., 1.]);+ ///+ /// let c = a.i((..2,))?;+ /// assert_eq!(c.shape().dims(), &[2, 2]);+ /// assert_eq!(c.to_vec2::<f32>()?, &[+ /// [0., 1.],+ /// [2., 3.]+ /// ]);+ ///+ /// let d = a.i((1..,))?;+ /// assert_eq!(d.shape().dims(), &[2, 2]);+ /// assert_eq!(d.to_vec2::<f32>()?, &[+ /// [2., 3.],+ /// [4., 5.]+ /// ]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ fn i(&self, (a,): (A,)) -> Result<Tensor, Error> {+ self.index(&[a.into()])+ }+}+#[allow(non_snake_case)]+impl<A, B> IndexOp<(A, B)> for Tensor+where+ A: Into<TensorIndexer>,+ B: Into<TensorIndexer>,+{+ ///```rust+ /// use candle_core::{Tensor, DType, Device, IndexOp};+ /// let a = Tensor::new(&[[0f32, 1., 2.], [3., 4., 5.], [6., 7., 8.]], &Device::Cpu)?;+ ///+ /// let b = a.i((1, 0))?;+ /// assert_eq!(b.to_vec0::<f32>()?, 3.);+ ///+ /// let c = a.i((..2, 1))?;+ /// assert_eq!(c.shape().dims(), &[2]);+ /// assert_eq!(c.to_vec1::<f32>()?, &[1., 4.]);+ ///+ /// let d = a.i((2.., ..))?;+ /// assert_eq!(d.shape().dims(), &[1, 3]);+ /// assert_eq!(d.to_vec2::<f32>()?, &[[6., 7., 8.]]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ fn i(&self, (a, b): (A, B)) -> Result<Tensor, Error> {+ self.index(&[a.into(), b.into()])+ }+}++macro_rules! index_op_tuple {+ ($doc:tt, $($t:ident),+) => {+ #[allow(non_snake_case)]+ impl<$($t),*> IndexOp<($($t,)*)> for Tensor+ where+ $($t: Into<TensorIndexer>,)*+ {+ #[doc=$doc]+ fn i(&self, ($($t,)*): ($($t,)*)) -> Result<Tensor, Error> {+ self.index(&[$($t.into(),)*])+ }+ }+ };+}++index_op_tuple!("see [TensorIndex#method.i]", A, B, C);+index_op_tuple!("see [TensorIndex#method.i]", A, B, C, D);+index_op_tuple!("see [TensorIndex#method.i]", A, B, C, D, E);+index_op_tuple!("see [TensorIndex#method.i]", A, B, C, D, E, F);+index_op_tuple!("see [TensorIndex#method.i]", A, B, C, D, E, F, G);diff --git a/vendor/candle-core/src/layout.rs b/vendor/candle-core/src/layout.rsnew file mode 100644index 0000000..fddc8bf--- /dev/null+++ b/vendor/candle-core/src/layout.rs@@ -0,0 +1,403 @@+//! Tensor Layouts including contiguous or sparse strides++use crate::{Error, Result, Shape};++#[derive(Debug, PartialEq, Eq, Clone)]+pub struct Layout {+ shape: Shape,+ // The strides are given in number of elements and not in bytes.+ stride: Vec<usize>,+ start_offset: usize,+}++impl Layout {+ pub fn new(shape: Shape, stride: Vec<usize>, start_offset: usize) -> Self {+ Self {+ shape,+ stride,+ start_offset,+ }+ }++ pub fn contiguous_with_offset<S: Into<Shape>>(shape: S, start_offset: usize) -> Self {+ let shape = shape.into();+ let stride = shape.stride_contiguous();+ Self {+ shape,+ stride,+ start_offset,+ }+ }++ pub fn contiguous<S: Into<Shape>>(shape: S) -> Self {+ Self::contiguous_with_offset(shape, 0)+ }++ pub fn dims(&self) -> &[usize] {+ self.shape.dims()+ }++ /// The dimension size for a specified dimension index.+ pub fn dim<D: crate::shape::Dim>(&self, dim: D) -> Result<usize> {+ let dim = dim.to_index(&self.shape, "dim")?;+ Ok(self.dims()[dim])+ }++ pub fn shape(&self) -> &Shape {+ &self.shape+ }++ pub fn stride(&self) -> &[usize] {+ &self.stride+ }++ pub fn start_offset(&self) -> usize {+ self.start_offset+ }++ /// Returns outer stride along `dim` if valid.+ ///+ /// Two conditions must hold:+ /// 1. Inner dims `[dim..]` has standard contiguous strides.+ /// 2. Outer dims `[..dim]` are contiguous among themselves, i.e.+ /// `stride[k] == dims[k+1] * stride[k+1]` for `k` in `0..dim-1`.+ ///+ /// When the tensor is fully contiguous this returns `Some(dims[dim..].product())`.+ pub(crate) fn outer_stride_for_dim(&self, dim: usize) -> Option<usize> {+ let dims = self.dims();+ let strides = self.stride();++ // 1. Inner `dims[dim..]` must have contiguous strides.+ let mut expected = 1usize;+ for i in (dim..dims.len()).rev() {+ if strides[i] != expected {+ return None;+ }+ expected *= dims[i];+ }++ if dim == 0 {+ // No outer dims.+ // `expected = dims[dim..].product()`+ return Some(expected);+ }++ // 2. Outer `dims[0..dim]` must be internally contiguous.+ let outer_stride = strides[dim - 1];+ let mut expected_outer = outer_stride;+ for k in (0..dim - 1).rev() {+ expected_outer *= dims[k + 1];+ if strides[k] != expected_outer {+ return None;+ }+ }++ Some(outer_stride)+ }++ /// Checks if more than one logical index lands on the same cell (or when we can't prove it does not)+ pub fn has_internal_overlap(&self) -> bool {+ !self.range().is_some_and(|f| f.injective)+ }++ /// Returns range of cells this layout can reach, and wether it reaches all of them.+ /// Returns `None` on arithmetic overflow.+ fn range(&self) -> Option<LayoutRange> {+ // Filter out dims <= 1 as their strides are irrelevant.+ let mut axes: Vec<(usize, usize)> = self+ .dims()+ .iter()+ .zip(self.stride())+ .filter(|(&d, _)| d > 1)+ .map(|(&d, &s)| (d, s))+ .collect();+ axes.sort_unstable_by_key(|&(_, s)| s);++ let mut span = 0usize;+ let mut injective = true;+ let mut dense = true;++ for (d, s) in axes {+ if s <= span {+ // This dim can land on a cell another dim already reaches.+ injective = false;+ dense = false;+ } else if s - span != 1 {+ // Has a gap+ dense = false;+ }+ span = span.checked_add((d - 1).checked_mul(s)?)?;+ }++ let lo = self.start_offset();+ Some(LayoutRange {+ lo,+ hi: lo.checked_add(span)?,+ injective,+ dense,+ })+ }++ /// Relation between this layout and another+ pub fn relation(&self, other: &Self) -> LayoutRelation {+ if self == other {+ return LayoutRelation::Identical;+ }++ if self.shape().elem_count() == 0 || other.shape().elem_count() == 0 {+ return LayoutRelation::Disjoint;+ }++ // Extract [`LayoutRange`] from layout.+ let (a, b) = match (self.range(), other.range()) {+ (Some(a), Some(b)) => (a, b),+ _ => return LayoutRelation::Unknown, // address arithmetic overflowed+ };++ if a.separated_from(&b) {+ return LayoutRelation::Disjoint;+ }++ if a.densely_contains(&b) || b.densely_contains(&a) {+ return LayoutRelation::Overlapping;+ }++ // We end up here when layouts ranges overlap and neither is dense, such as disjoint+ // column slices. Figuring out these cases is a bounded integer feasibility problem+ // over the strides. This is NP-hard in general. Cheap at common tensor ranks.+ // If we want to move more cases out of unknown into disjoint/overlapping it can be done using the+ // same approach as numpy: https://github.com/numpy/numpy/blob/main/numpy/_core/src/common/mem_overlap.c+ LayoutRelation::Unknown+ }++ /// Returns the appropriate start and stop offset if the data is stored in a C+ /// contiguous (aka row major) way.+ pub fn contiguous_offsets(&self) -> Option<(usize, usize)> {+ if self.is_contiguous() {+ let start_o = self.start_offset;+ Some((start_o, start_o + self.shape.elem_count()))+ } else {+ None+ }+ }++ /// Returns true if the data is stored in a C contiguous (aka row major) way.+ /// Note that this does not implies that the start offset is 0 or that there are no extra+ /// elements at the end of the storage.+ pub fn is_contiguous(&self) -> bool {+ self.shape.is_contiguous(&self.stride)+ }++ /// Returns true if the data is stored in a Fortran contiguous (aka column major) way.+ pub fn is_fortran_contiguous(&self) -> bool {+ self.shape.is_fortran_contiguous(&self.stride)+ }++ pub fn is_scalar(&self) -> bool {+ let dims = self.dims();+ dims.is_empty() || dims.iter().all(|d| *d == 1)+ }++ /// Returns true if the data is actually a scalar during broadcast+ pub fn is_scalar_broadcast(&self) -> bool {+ self.stride().iter().all(|s| *s == 0)+ }++ pub fn is_scalar_like(&self) -> bool {+ self.is_scalar() || self.is_scalar_broadcast()+ }++ pub fn narrow(&self, dim: usize, start: usize, len: usize) -> Result<Self> {+ let dims = self.shape().dims();+ if dim >= dims.len() {+ Err(Error::DimOutOfRange {+ shape: self.shape().clone(),+ dim: dim as i32,+ op: "narrow",+ }+ .bt())?+ }+ if start + len > dims[dim] {+ Err(Error::NarrowInvalidArgs {+ shape: self.shape.clone(),+ dim,+ start,+ len,+ msg: "start + len > dim_len",+ }+ .bt())?+ }+ let mut dims = dims.to_vec();+ dims[dim] = len;+ Ok(Self {+ shape: Shape::from(dims),+ stride: self.stride.clone(),+ start_offset: self.start_offset + self.stride[dim] * start,+ })+ }++ pub fn transpose(&self, dim1: usize, dim2: usize) -> Result<Self> {+ let rank = self.shape.rank();+ if rank <= dim1 || rank <= dim2 {+ Err(Error::UnexpectedNumberOfDims {+ expected: usize::max(dim1, dim2),+ got: rank,+ shape: self.shape().clone(),+ }+ .bt())?+ }+ let mut stride = self.stride().to_vec();+ let mut dims = self.shape().dims().to_vec();+ dims.swap(dim1, dim2);+ stride.swap(dim1, dim2);+ Ok(Self {+ shape: Shape::from(dims),+ stride,+ start_offset: self.start_offset,+ })+ }++ pub fn permute(&self, idxs: &[usize]) -> Result<Self> {+ let is_permutation =+ idxs.len() == self.shape.rank() && (0..idxs.len()).all(|i| idxs.contains(&i));+ if !is_permutation {+ crate::bail!(+ "dimension mismatch in permute, tensor {:?}, dims: {:?}",+ self.dims(),+ idxs+ )+ }+ let stride = self.stride();+ let dims = self.shape().dims();+ let mut perm_stride = stride.to_vec();+ let mut perm_dims = dims.to_vec();+ for (i, &idx) in idxs.iter().enumerate() {+ perm_stride[i] = stride[idx];+ perm_dims[i] = dims[idx];+ }+ Ok(Self {+ shape: Shape::from(perm_dims),+ stride: perm_stride,+ start_offset: self.start_offset,+ })+ }++ pub fn broadcast_as<S: Into<Shape>>(&self, shape: S) -> Result<Self> {+ let shape = shape.into();+ if shape.rank() < self.shape().rank() {+ return Err(Error::BroadcastIncompatibleShapes {+ src_shape: self.shape().clone(),+ dst_shape: shape,+ }+ .bt());+ }+ let added_dims = shape.rank() - self.shape().rank();+ let mut stride = vec![0; added_dims];+ for (&dst_dim, (&src_dim, &src_stride)) in shape.dims()[added_dims..]+ .iter()+ .zip(self.dims().iter().zip(self.stride()))+ {+ let s = if dst_dim == src_dim {+ src_stride+ } else if src_dim != 1 {+ return Err(Error::BroadcastIncompatibleShapes {+ src_shape: self.shape().clone(),+ dst_shape: shape,+ }+ .bt());+ } else {+ 0+ };+ stride.push(s)+ }+ Ok(Self {+ shape,+ stride,+ start_offset: self.start_offset,+ })+ }++ pub(crate) fn strided_index(&self) -> crate::StridedIndex<'_> {+ crate::StridedIndex::from_layout(self)+ }++ pub(crate) fn strided_blocks(&self) -> crate::StridedBlocks<'_> {+ let mut block_len = 1usize;+ let mut contiguous_dims = 0usize; // Counted from the right.+ for (&stride, &dim) in self.stride().iter().zip(self.dims().iter()).rev() {+ // Size-1 dimensions are trivially contiguous regardless of their stride.+ if dim == 1 {+ contiguous_dims += 1;+ continue;+ }+ if stride != block_len {+ break;+ }+ block_len *= dim;+ contiguous_dims += 1;+ }+ let index_dims = self.dims().len() - contiguous_dims;+ match index_dims {+ 0 => crate::StridedBlocks::SingleBlock {+ start_offset: self.start_offset,+ len: block_len,+ },+ 1 => crate::StridedBlocks::UniformBlocks {+ start_offset: self.start_offset,+ block_len,+ count: self.dims()[0],+ src_stride: self.stride[0],+ },+ _ => {+ let block_start_index = crate::StridedIndex::new(+ &self.dims()[..index_dims],+ &self.stride[..index_dims],+ self.start_offset,+ );+ crate::StridedBlocks::MultipleBlocks {+ block_start_index,+ block_len,+ }+ }+ }+ }+}++/// Describes the range of cells a [`Layout`] can reach within its allocation,+/// and whether it reaches all of them.+struct LayoutRange {+ /// Lowest reachable cell. Equal to layout `start_offset`.+ lo: usize,+ /// Highest reachable cell (inclusive).+ hi: usize,+ /// Proven to map distinct logical indices to distinct cells.+ injective: bool,+ /// Indicates that layout occupies every cell in `lo..=hi`.+ /// Does not necessarily mean that layout is contiguous.+ dense: bool,+}++impl LayoutRange {+ /// Bounding intervals cannot meet, which means these layouts are seperate.+ fn separated_from(&self, other: &Self) -> bool {+ self.hi < other.lo || other.hi < self.lo+ }++ /// If a dense layout contains another's `lo` there is overlap.+ fn densely_contains(&self, other: &Self) -> bool {+ self.dense && other.lo >= self.lo && other.lo <= self.hi+ }+}++/// How two layouts over the same allocation relate.+#[derive(Copy, Clone, PartialEq, Eq, Debug)]+pub enum LayoutRelation {+ /// Simple assignment. x[i] += x[i]+ Identical,+ /// Completely distinct layouts.+ Disjoint,+ /// Any kind of overlap.+ Overlapping,+ /// Could not prove relation.+ Unknown,+}diff --git a/vendor/candle-core/src/lib.rs b/vendor/candle-core/src/lib.rsnew file mode 100644index 0000000..7237510--- /dev/null+++ b/vendor/candle-core/src/lib.rs@@ -0,0 +1,179 @@+//! ML framework for Rust+//!+//! ```rust+//! use candle_core::{Tensor, DType, Device};+//! # use candle_core::Error;+//! # fn main() -> Result<(), Error>{+//!+//! let a = Tensor::arange(0f32, 6f32, &Device::Cpu)?.reshape((2, 3))?;+//! let b = Tensor::arange(0f32, 12f32, &Device::Cpu)?.reshape((3, 4))?;+//! let c = a.matmul(&b)?;+//!+//! # Ok(())}+//! ```+//!+//! ## Features+//!+//! - Simple syntax (looks and feels like PyTorch)+//! - CPU and Cuda backends (and M1 support)+//! - Enable serverless (CPU) small and fast deployments+//! - Model training+//! - Distributed computing (NCCL).+//! - Models out of the box (Llama, Whisper, Falcon, ...)+//!+//! ## FAQ+//!+//! - Why Candle?+//!+//! Candle stems from the need to reduce binary size in order to *enable serverless*+//! possible by making the whole engine smaller than PyTorch very large library volume+//!+//! And simply *removing Python* from production workloads.+//! Python can really add overhead in more complex workflows and the [GIL](https://www.backblaze.com/blog/the-python-gil-past-present-and-future/) is a notorious source of headaches.+//!+//! Rust is cool, and a lot of the HF ecosystem already has Rust crates [safetensors](https://github.com/huggingface/safetensors) and [tokenizers](https://github.com/huggingface/tokenizers)+//!+//! ## Other Crates+//!+//! Candle consists of a number of crates. This crate holds core the common data structures but you may wish+//! to look at the docs for the other crates which can be found here:+//!+//! - [candle-core](https://docs.rs/candle-core/). Core Datastructures and DataTypes.+//! - [candle-nn](https://docs.rs/candle-nn/). Building blocks for Neural Nets.+//! - [candle-datasets](https://docs.rs/candle-datasets/). Rust access to commonly used Datasets like MNIST.+//! - [candle-examples](https://docs.rs/candle-examples/). Examples of Candle in Use.+//! - [candle-onnx](https://docs.rs/candle-onnx/). Loading and using ONNX models.+//! - [candle-pyo3](https://docs.rs/candle-pyo3/). Access to Candle from Python.+//! - [candle-transformers](https://docs.rs/candle-transformers/). Candle implementation of many published transformer models.+//!++#[cfg(feature = "accelerate")]+mod accelerate;+pub mod backend;+pub mod backprop;+pub mod conv;+mod convert;+pub mod cpu;+pub mod cpu_backend;+#[cfg(feature = "cuda")]+pub mod cuda_backend;+mod custom_op;+mod device;+pub mod display;+mod dtype;+pub mod dummy_cuda_backend;+pub mod dummy_dtype;+mod dummy_metal_backend;+pub mod error;+mod indexer;+pub mod layout;+#[cfg(feature = "metal")]+pub mod metal_backend;+#[cfg(feature = "mkl")]+mod mkl;+pub mod nditer;+pub mod npy;+pub mod op;+pub mod pickle;+pub mod quantized;+pub mod safetensors;+pub mod scalar;+pub mod shape;+mod sort;+mod storage;+pub mod streaming;+mod strided_index;+mod tensor;+mod tensor_cat;+pub mod test_utils;+pub mod utils;+mod variable;++#[cfg(feature = "cudnn")]+pub use cuda_backend::cudnn;+#[cfg(feature = "cutile")]+pub use cuda_backend::cutile;++pub use cpu_backend::{CpuStorage, CpuStorageRef};+pub use custom_op::{CustomOp1, CustomOp2, CustomOp3, InplaceOp1, InplaceOp2, InplaceOp3};+pub use device::{Device, DeviceLocation, NdArray};+pub use dtype::{DType, DTypeParseError, FloatDType, IntDType, WithDType};+pub use dummy_dtype::{F4, F6E2M3, F6E3M2, F8E8M0};+pub use error::{Context, Error, Result};+pub use indexer::{IndexOp, TensorIndexer};+pub use layout::Layout;+pub use nditer::NdIter;+pub use shape::{Shape, D};+pub use storage::{Storage, StorageMutRef, StorageRef};+pub use streaming::{StreamTensor, StreamingBinOp, StreamingModule};+pub use strided_index::{StridedBlocks, StridedIndex};+pub use tensor::{Tensor, TensorId};+pub use variable::Var;++#[cfg(feature = "cuda")]+pub use cuda_backend as cuda;++#[cfg(not(feature = "cuda"))]+pub use dummy_cuda_backend as cuda;++pub use cuda::{CudaDevice, CudaStorage};++#[cfg(feature = "metal")]+pub use metal_backend::{MetalDevice, MetalError, MetalStorage};++#[cfg(not(feature = "metal"))]+pub use dummy_metal_backend::{MetalDevice, MetalError, MetalStorage};++#[cfg(feature = "mkl")]+extern crate intel_mkl_src;++#[cfg(feature = "accelerate")]+extern crate accelerate_src;++pub trait ToUsize2 {+ fn to_usize2(self) -> (usize, usize);+}++impl ToUsize2 for usize {+ fn to_usize2(self) -> (usize, usize) {+ (self, self)+ }+}++impl ToUsize2 for (usize, usize) {+ fn to_usize2(self) -> (usize, usize) {+ self+ }+}++/// Defining a module with forward method using a single argument.+pub trait Module {+ fn forward(&self, xs: &Tensor) -> Result<Tensor>;+}++impl<T: Fn(&Tensor) -> Result<Tensor>> Module for T {+ fn forward(&self, xs: &Tensor) -> Result<Tensor> {+ self(xs)+ }+}++impl<M: Module> Module for Option<&M> {+ fn forward(&self, xs: &Tensor) -> Result<Tensor> {+ match self {+ None => Ok(xs.clone()),+ Some(m) => m.forward(xs),+ }+ }+}++/// A single forward method using a single tensor argument and a flag to+/// separate the training and evaluation behaviors.+pub trait ModuleT {+ fn forward_t(&self, xs: &Tensor, train: bool) -> Result<Tensor>;+}++impl<M: Module> ModuleT for M {+ fn forward_t(&self, xs: &Tensor, _train: bool) -> Result<Tensor> {+ self.forward(xs)+ }+}diff --git a/vendor/candle-core/src/metal_backend/device.rs b/vendor/candle-core/src/metal_backend/device.rsnew file mode 100644index 0000000..3d7b946--- /dev/null+++ b/vendor/candle-core/src/metal_backend/device.rs@@ -0,0 +1,497 @@+use crate::{DType, Result};++use candle_metal_kernels::{+ metal::{+ BlitCommandsGuard, Buffer, BufferMap, Commands, CommandsGuard, Device, MTLResourceOptions,+ ResidencySet,+ },+ Kernels,+};+use objc2_foundation::NSURL;+use objc2_metal::{MTLCaptureDescriptor, MTLCaptureDestination, MTLCaptureManager};+use std::path::Path;+use std::sync::{Arc, Mutex, RwLock};++use super::MetalError;++/// Unique identifier for metal devices.+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]+pub struct DeviceId(usize);++impl DeviceId {+ pub(crate) fn new() -> Self {+ // https://users.rust-lang.org/t/idiomatic-rust-way-to-generate-unique-id/33805+ use std::sync::atomic;+ static COUNTER: atomic::AtomicUsize = atomic::AtomicUsize::new(1);+ Self(COUNTER.fetch_add(1, atomic::Ordering::Relaxed))+ }+}++#[derive(Clone)]+pub struct MetalDevice {+ /// Unique identifier, the registryID is not sufficient as it identifies the GPU rather than+ /// the device itself.+ pub(crate) id: DeviceId,++ /// Raw metal device: <https://developer.apple.com/documentation/metal/mtldevice?language=objc>+ pub(crate) device: Device,++ pub(crate) commands: Arc<Commands>,++ /// Simple allocator struct.+ /// The buffers are stored in size buckets since ML tends to use similar shapes over and over.+ /// We store the buffers in [`Arc`] because it's much faster than Obj-c internal ref counting+ /// (could be linked to FFI communication overhead).+ ///+ /// Whenever a buffer has a strong_count==1, we can reuse it, it means it was dropped in the+ /// graph calculation, and only we the allocator kept a reference to it, therefore it's free+ /// to be reused. However, in order for this to work, we need to guarantee the order of+ /// operation, so that this buffer is not being used by another kernel at the same time.+ /// Arc is the CPU reference count, it doesn't mean anything on the GPU side of things.+ ///+ /// Whenever we actually allocate a new buffer, we make a full sweep to clean up unused buffers+ /// (strong_count = 1).+ pub(crate) buffers: Arc<RwLock<BufferMap>>,++ /// Same as `buffers` but uses `PRIVATE_RESOURCE_OPTIONS` (StorageModePrivate on macOS).+ /// Intermediate compute buffers don't need CPU access so Private avoids coherency overhead.+ pub(crate) private_buffers: Arc<RwLock<BufferMap>>,++ /// Simple keeper struct to keep track of the already compiled kernels so we can reuse them.+ /// Heavily used by [`candle_metal_kernels`]+ pub(crate) kernels: Arc<Kernels>,+ /// Seed for random number generation.+ pub(crate) seed: Arc<Mutex<Buffer>>,+ /// Last seed value set on this device.+ pub(crate) seed_value: Arc<RwLock<u64>>,+ /// Residency set registered on the command queue.+ pub(crate) residency_set: Arc<ResidencySet>,+}++// Resource options used for creating buffers. Shared storage mode allows both CPU and GPU to access the buffer.+pub const RESOURCE_OPTIONS: MTLResourceOptions = objc2_metal::MTLResourceOptions(+ MTLResourceOptions::StorageModeShared.0 | MTLResourceOptions::HazardTrackingModeUntracked.0,+);+// Resource options used for `new_private_buffer`. This uses `private` where supported.+#[cfg(target_os = "ios")]+pub const PRIVATE_RESOURCE_OPTIONS: MTLResourceOptions = RESOURCE_OPTIONS;+#[cfg(not(target_os = "ios"))]+pub const PRIVATE_RESOURCE_OPTIONS: MTLResourceOptions = objc2_metal::MTLResourceOptions(+ MTLResourceOptions::StorageModePrivate.0 | MTLResourceOptions::HazardTrackingModeUntracked.0,+);++impl std::fmt::Debug for MetalDevice {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ write!(f, "MetalDevice({:?})", self.id)+ }+}++impl std::ops::Deref for MetalDevice {+ type Target = Device;++ fn deref(&self) -> &Self::Target {+ &self.device+ }+}++impl MetalDevice {+ pub fn id(&self) -> DeviceId {+ self.id+ }++ pub fn metal_device(&self) -> &Device {+ &self.device+ }++ /// Coachwhip lab: what the buffer pool holds, as (bytes pooled, bytes in use, buffers pooled).+ /// A pooled buffer is in use while anything but the pool holds it (strong count above 1).+ pub fn pool_stats(&self) -> (usize, usize, usize) {+ let (mut pooled, mut in_use, mut count) = (0usize, 0usize, 0usize);+ for map in [&self.buffers, &self.private_buffers] {+ if let Ok(buffers) = map.read() {+ for (size, subs) in buffers.iter() {+ for s in subs {+ pooled += size;+ count += 1;+ if Arc::strong_count(s) > 1 {+ in_use += size;+ }+ }+ }+ }+ }+ (pooled, in_use, count)+ }++ fn drop_unused_buffers(&self) -> Result<()> {+ let mut buffers = self.buffers.write().map_err(MetalError::from)?;+ for subbuffers in buffers.values_mut() {+ subbuffers.retain(|s| {+ if Arc::strong_count(s) == 1 {+ self.residency_set.remove(s);+ false+ } else {+ true+ }+ });+ }+ let mut private_buffers = self.private_buffers.write().map_err(MetalError::from)?;+ for subbuffers in private_buffers.values_mut() {+ subbuffers.retain(|s| {+ if Arc::strong_count(s) == 1 {+ self.residency_set.remove(s);+ false+ } else {+ true+ }+ });+ }+ Ok(())+ }++ pub fn command_encoder<'a>(&'a self) -> Result<CommandsGuard<'a>> {+ let command_encoder = self.commands.command_encoder().map_err(MetalError::from)?;+ Ok(command_encoder)+ }++ pub fn blit_command_encoder(&self) -> Result<BlitCommandsGuard<'_>> {+ let command_encoder = self+ .commands+ .blit_command_encoder()+ .map_err(MetalError::from)?;+ Ok(command_encoder)+ }++ pub fn wait_until_completed(&self) -> Result<()> {+ self.commands+ .wait_until_completed()+ .map_err(MetalError::from)?;++ self.drop_unused_buffers()?;+ Ok(())+ }++ /// Commit and wait on the buffer holding the caller's work; safe for concurrent CPU readbacks.+ pub fn flush_and_wait_current(&self) -> Result<()> {+ self.commands+ .flush_and_wait_current()+ .map_err(MetalError::from)?;++ self.drop_unused_buffers()?;+ Ok(())+ }++ pub fn kernels(&self) -> &Kernels {+ &self.kernels+ }++ pub fn device(&self) -> &Device {+ &self.device+ }++ /// Registers buffers in the device's residency set, keeping them+ /// permanently GPU-resident instead of paying per-command-buffer residency+ /// bookkeeping. Useful for buffers candle did not allocate, e.g.+ /// `newBufferWithBytesNoCopy` views over an mmap'd weights file. No-op on+ /// systems without residency-set support.+ pub fn register_buffers<'a>(&self, bufs: impl IntoIterator<Item = &'a Buffer>) {+ self.residency_set.insert_batch(bufs);+ }++ /// Unregisters buffers previously passed to `register_buffers`, releasing+ /// the set's retain so they can be deallocated. Only unregister buffers+ /// you registered yourself, after GPU work referencing them has completed.+ pub fn unregister_buffers<'a>(&self, bufs: impl IntoIterator<Item = &'a Buffer>) {+ self.residency_set.remove_batch(bufs);+ }++ /// Returns a builder for buffer allocation. See `BufferBuilder`.+ pub fn new_buffer_builder(&self) -> BufferBuilder<'_> {+ BufferBuilder::new(self)+ }++ /// Creates a new buffer (not necessarily zeroed).+ ///+ /// Uses StorageModePrivate on macOS for faster GPU access (no CPU coherency overhead).+ /// Falls back to StorageModeShared on iOS where Private is not always available.+ pub fn new_buffer(+ &self,+ element_count: usize,+ dtype: DType,+ _name: &str,+ ) -> Result<Arc<Buffer>> {+ let size = element_count * dtype.size_in_bytes();+ let mut buffers = self.private_buffers.write().map_err(MetalError::from)?;+ if let Some(b) = find_available_buffer(size, &buffers) {+ return Ok(b.clone());+ }+ let size = buf_size(size);+ let subbuffers = buffers.entry(size).or_insert(vec![]);++ let new_buffer = self+ .device+ .new_buffer(size, PRIVATE_RESOURCE_OPTIONS)+ .map_err(MetalError::from)?;+ let new_buffer = Arc::new(new_buffer);+ self.residency_set.insert(&new_buffer);+ subbuffers.push(new_buffer.clone());+ Ok(new_buffer)+ }++ /// Creates a new private buffer (not necessarily zeroed).+ ///+ /// This is intentionally not in the Metal buffer pool to allow the efficient implementation of persistent buffers.+ pub fn new_private_buffer(+ &self,+ element_count: usize,+ dtype: DType,+ _name: &str,+ ) -> Result<Arc<Buffer>> {+ let size = element_count * dtype.size_in_bytes();+ let buffer = self+ .device+ .new_buffer(size, PRIVATE_RESOURCE_OPTIONS)+ .map_err(MetalError::from)?;+ let buffer = Arc::new(buffer);+ self.residency_set.insert(&buffer);+ Ok(buffer)+ }++ /// Creates a new buffer from data.+ ///+ /// Does not require synchronization, as [newBufferWithBytes](https://developer.apple.com/documentation/metal/mtldevice/1433429-newbufferwithbytes)+ /// allocates the buffer and copies over the existing data before returning the MTLBuffer.+ pub fn new_buffer_with_data<T>(&self, data: &[T]) -> Result<Arc<Buffer>> {+ let size = core::mem::size_of_val(data);+ let new_buffer = self+ .device+ .new_buffer_with_data(data.as_ptr().cast(), size, RESOURCE_OPTIONS)+ .map_err(MetalError::from)?;+ let mut buffers = self.buffers.write().map_err(MetalError::from)?;++ let subbuffers = buffers.entry(size).or_insert(vec![]);++ let new_buffer = Arc::new(new_buffer);+ self.residency_set.insert(&new_buffer);+ subbuffers.push(new_buffer.clone());+ Ok(new_buffer)+ }++ pub fn allocate_zeros(&self, size_in_bytes: usize) -> Result<Arc<Buffer>> {+ let buffer = self.allocate_buffer(size_in_bytes)?;+ let mut blit = self.blit_command_encoder()?;+ blit.set_label("zeros");+ blit.fill_buffer(&buffer, (0, buffer.length()), 0);+ /*+ // Alternative impl+ if size_in_bytes > 0 {+ let encoder = self.command_encoder()?;+ call_const_fill(+ &self.device,+ &encoder,+ &self.kernels,+ "fill_u8",+ size_in_bytes,+ &buffer,+ 0u8,+ )+ .map_err(crate::Error::wrap)?;+ }+ */+ Ok(buffer)+ }++ /// The critical allocator algorithm+ pub fn allocate_buffer(&self, size: usize) -> Result<Arc<Buffer>> {+ let mut buffers = self.buffers.write().map_err(MetalError::from)?;+ if let Some(b) = find_available_buffer(size, &buffers) {+ // Cloning also ensures we increment the strong count+ return Ok(b.clone());+ }+ let size = buf_size(size);+ let subbuffers = buffers.entry(size).or_insert(vec![]);++ let new_buffer = self+ .device+ .new_buffer(size, RESOURCE_OPTIONS)+ .map_err(MetalError::from)?;+ let new_buffer = Arc::new(new_buffer);+ self.residency_set.insert(&new_buffer);+ subbuffers.push(new_buffer.clone());+ Ok(new_buffer)+ }++ /// Create a metal GPU capture trace on [`path`].+ pub fn capture<P: AsRef<Path>>(&self, path: P) -> Result<()> {+ let capture = unsafe { MTLCaptureManager::sharedCaptureManager() };+ let descriptor = MTLCaptureDescriptor::new();+ descriptor.setDestination(MTLCaptureDestination::GPUTraceDocument);+ descriptor.set_capture_device(self.device().as_ref());+ // The [set_output_url] call requires an absolute path so we convert it if needed.+ if path.as_ref().is_absolute() {+ let url = NSURL::from_file_path(path);+ descriptor.setOutputURL(url.as_deref());+ } else {+ let path = std::env::current_dir()?.join(path);+ let url = NSURL::from_file_path(path);+ descriptor.setOutputURL(url.as_deref());+ }++ capture+ .startCaptureWithDescriptor_error(&descriptor)+ .map_err(|e| MetalError::from(e.to_string()))?;+ Ok(())+ }+}++fn buf_size(size: usize) -> usize {+ size.next_power_of_two()+}++/// Applies the [`BufferBuilder`] label, clearing any stale label on a reused pooled buffer.+#[cfg(feature = "metal-debug-labels")]+#[inline]+fn buffer_label(buffer: &Buffer, label: Option<&str>) {+ buffer.set_label(label.unwrap_or("unlabeled"));+}+#[cfg(not(feature = "metal-debug-labels"))]+#[inline]+fn buffer_label(_buffer: &Buffer, _label: Option<&str>) {}++type DataUpload<'a> = Box<dyn FnOnce(&MetalDevice) -> Result<Arc<Buffer>> + 'a>;++enum BufferInit<'a> {+ Typed { elem_count: usize, dtype: DType },+ Size(usize),+ Zeros(usize),+ Data(DataUpload<'a>),+}++/// Builder for `MTLBuffer` allocations; pool reuse handled by [`MetalDevice`].+pub struct BufferBuilder<'a> {+ device: &'a MetalDevice,+ label: Option<&'a str>,+}++/// [`BufferBuilder`] with an init kind set; `build()` lives here.+pub struct ReadyBufferBuilder<'a> {+ device: &'a MetalDevice,+ init: BufferInit<'a>,+ label: Option<&'a str>,+}++impl<'a> BufferBuilder<'a> {+ fn new(device: &'a MetalDevice) -> Self {+ Self {+ device,+ label: None,+ }+ }++ /// Allocate elem_count * dtype size bytes, uninitialized, private storage.+ pub fn with_size_for(self, elem_count: usize, dtype: DType) -> ReadyBufferBuilder<'a> {+ self.ready(BufferInit::Typed { elem_count, dtype })+ }++ /// Allocate size bytes, uninitialized, shared storage.+ pub fn with_size(self, size: usize) -> ReadyBufferBuilder<'a> {+ self.ready(BufferInit::Size(size))+ }++ /// Allocate size bytes, zero-filled, shared storage. Pool rounding may make+ /// the allocation larger than size; the extra bytes are also zeroed.+ pub fn with_zeros(self, size: usize) -> ReadyBufferBuilder<'a> {+ self.ready(BufferInit::Zeros(size))+ }++ /// Allocate a shared buffer initialized from data. Always allocates; does not+ /// reuse the pool.+ pub fn with_data<T>(self, data: &'a [T]) -> ReadyBufferBuilder<'a> {+ self.ready(BufferInit::Data(Box::new(move |device| {+ device.new_buffer_with_data(data)+ })))+ }++ pub fn with_label(mut self, label: &'a str) -> Self {+ self.label = Some(label);+ self+ }++ #[inline]+ fn ready(self, init: BufferInit<'a>) -> ReadyBufferBuilder<'a> {+ ReadyBufferBuilder {+ device: self.device,+ init,+ label: self.label,+ }+ }+}++impl<'a> ReadyBufferBuilder<'a> {+ pub fn with_label(mut self, label: &'a str) -> Self {+ self.label = Some(label);+ self+ }++ pub fn build(self) -> Result<Arc<Buffer>> {+ let buffer = match self.init {+ BufferInit::Typed { elem_count, dtype } => {+ self.device.new_buffer(elem_count, dtype, "")?+ }+ BufferInit::Size(size) => self.device.allocate_buffer(size)?,+ BufferInit::Zeros(size) => self.device.allocate_zeros(size)?,+ BufferInit::Data(upload) => upload(self.device)?,+ };+ buffer_label(&buffer, self.label);+ Ok(buffer)+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn test_buf_size_exact_powers_of_two() {+ assert_eq!(buf_size(1), 1);+ assert_eq!(buf_size(2), 2);+ assert_eq!(buf_size(4), 4);+ assert_eq!(buf_size(8), 8);+ assert_eq!(buf_size(16), 16);+ assert_eq!(buf_size(1024), 1024);+ }++ #[test]+ fn test_buf_size_rounds_up() {+ assert_eq!(buf_size(3), 4);+ assert_eq!(buf_size(5), 8);+ assert_eq!(buf_size(6), 8);+ assert_eq!(buf_size(7), 8);+ assert_eq!(buf_size(9), 16);+ assert_eq!(buf_size(1000), 1024);+ assert_eq!(buf_size(1025), 2048);+ }++ #[test]+ fn test_buf_size_bf16_f16_scalar() {+ // BF16 and F16 are 2 bytes per element. A scalar tensor requests+ // a 2-byte buffer. This must not be rounded down to 1.+ assert_eq!(buf_size(2), 2);+ }+}++fn find_available_buffer(size: usize, buffers: &BufferMap) -> Option<Arc<Buffer>> {+ let mut best_buffer: Option<&Arc<Buffer>> = None;+ let mut best_buffer_size = usize::MAX;+ for (buffer_size, subbuffers) in buffers.iter() {+ if buffer_size >= &size && buffer_size < &best_buffer_size {+ for sub in subbuffers {+ if Arc::strong_count(sub) == 1 {+ best_buffer = Some(sub);+ best_buffer_size = *buffer_size;+ }+ }+ }+ }+ best_buffer.cloned()+}diff --git a/vendor/candle-core/src/metal_backend/mod.rs b/vendor/candle-core/src/metal_backend/mod.rsnew file mode 100644index 0000000..9aba03a--- /dev/null+++ b/vendor/candle-core/src/metal_backend/mod.rs@@ -0,0 +1,2377 @@+//! Implementation of Backend traits for Metal+//!+use crate::backend::{BackendDevice, BackendStorage};+use crate::conv::{ParamsConv1D, ParamsConv2D, ParamsConvTranspose1D, ParamsConvTranspose2D};+use crate::op::{BinaryOpT, CmpOp, ReduceOp, UnaryOpT};+use crate::{CpuStorage, CpuStorageRef, DType, Error, Layout, Result, Shape};+use candle_metal_kernels::{+ metal::{Buffer, Commands, Device, ResidencySet},+ BufferOffset, CallConvTranspose2dCfg, Kernels, RESOURCE_OPTIONS,+};+use objc2_foundation::NSRange;+#[cfg(feature = "metal-debug-labels")]+use objc2_foundation::NSString;+#[cfg(feature = "metal-debug-labels")]+use objc2_metal::MTLCommandQueue;+use std::collections::HashMap;+use std::ffi::c_void;+use std::sync::{Arc, Mutex, PoisonError, RwLock, TryLockError};++mod device;+pub use device::{DeviceId, MetalDevice};++pub fn buffer_o<'a>(buffer: &'a Buffer, l: &Layout, dtype: DType) -> BufferOffset<'a> {+ BufferOffset {+ buffer,+ offset_in_bytes: l.start_offset() * dtype.size_in_bytes(),+ }+}+/// Simple way to catch lock error without+/// depending on T+#[derive(thiserror::Error, Debug)]+pub enum LockError {+ #[error("{0}")]+ Poisoned(String),+ #[error("Would block")]+ WouldBlock,+}++impl<T> From<TryLockError<T>> for MetalError {+ fn from(value: TryLockError<T>) -> Self {+ match value {+ TryLockError::Poisoned(p) => MetalError::LockError(LockError::Poisoned(p.to_string())),+ TryLockError::WouldBlock => MetalError::LockError(LockError::WouldBlock),+ }+ }+}++impl<T> From<PoisonError<T>> for MetalError {+ fn from(p: PoisonError<T>) -> Self {+ MetalError::LockError(LockError::Poisoned(p.to_string()))+ }+}++/// Metal related errors+#[derive(thiserror::Error, Debug)]+pub enum MetalError {+ #[error("{0}")]+ Message(String),+ #[error(transparent)]+ KernelError(#[from] candle_metal_kernels::MetalKernelError),+ #[error("{0:?}")]+ LockError(LockError),+ #[error("{msg}, expected: {expected:?}, got: {got:?}")]+ UnexpectedDType {+ msg: &'static str,+ expected: DType,+ got: DType,+ },+}++impl From<String> for MetalError {+ fn from(e: String) -> Self {+ MetalError::Message(e)+ }+}++#[derive(Debug, Clone)]+pub struct MetalStorage {+ /// The actual buffer containing the data.+ buffer: Arc<Buffer>,+ /// a reference to the device owning this buffer+ device: MetalDevice,+ /// The count of allocated elements in the buffer+ count: usize,+ /// The dtype is kept since buffers are untyped.+ dtype: DType,+}++impl BackendStorage for MetalStorage {+ type Device = MetalDevice;++ fn try_clone(&self, _: &Layout) -> Result<Self> {+ Ok(self.clone())+ }++ fn dtype(&self) -> DType {+ self.dtype+ }++ fn device(&self) -> &Self::Device {+ &self.device+ }++ fn to_cpu_storage(&self) -> Result<CpuStorage> {+ match self.dtype {+ DType::U8 => Ok(CpuStorage::U8(self.to_cpu()?)),+ DType::U32 => Ok(CpuStorage::U32(self.to_cpu()?)),+ DType::I16 => Ok(CpuStorage::I16(self.to_cpu()?)),+ DType::I32 => Ok(CpuStorage::I32(self.to_cpu()?)),+ DType::I64 => Ok(CpuStorage::I64(self.to_cpu()?)),+ DType::F16 => Ok(CpuStorage::F16(self.to_cpu()?)),+ DType::BF16 => Ok(CpuStorage::BF16(self.to_cpu()?)),+ DType::F32 => Ok(CpuStorage::F32(self.to_cpu()?)),+ DType::F64 => Ok(CpuStorage::F64(self.to_cpu()?)),+ DType::F8E4M3 => Ok(CpuStorage::F8E4M3(self.to_cpu()?)),+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ Err(crate::Error::UnsupportedDTypeForOp(self.dtype, "to_cpu_storage").bt())+ }+ }+ }++ fn affine(&self, layout: &Layout, mul: f64, add: f64) -> Result<Self> {+ let device = self.device().clone();++ let shape = layout.shape();+ let el = shape.elem_count();+ let dtype = self.dtype;++ let buffer = device+ .new_buffer_builder()+ .with_size_for(el, self.dtype)+ .with_label("affine")+ .build()?;+ let encoder = self.device.command_encoder()?;+ let src = buffer_o(&self.buffer, layout, dtype);+ if layout.is_contiguous() {+ let name = match self.dtype {+ DType::F32 => "affine_f32",+ DType::F16 => "affine_f16",+ DType::BF16 => "affine_bf16",+ DType::U8 => "affine_u8",+ DType::U32 => "affine_u32",+ DType::I64 => "affine_i64",+ dtype => crate::bail!("Metal contiguous affine {dtype:?} not implemented"),+ };+ candle_metal_kernels::call_affine(+ &device.device,+ &encoder,+ &device.kernels,+ name,+ self.dtype.size_in_bytes(),+ el,+ src,+ &buffer,+ mul as f32,+ add as f32,+ )+ .map_err(MetalError::from)?;+ } else {+ let name = match self.dtype {+ DType::F32 => "affine_f32_strided",+ DType::F16 => "affine_f16_strided",+ DType::BF16 => "affine_bf16_strided",+ DType::U8 => "affine_u8_strided",+ DType::U32 => "affine_u32_strided",+ DType::I64 => "affine_i64_strided",+ dtype => crate::bail!("Metal strided affine {dtype:?} not implemented"),+ };+ candle_metal_kernels::call_affine_strided(+ &device.device,+ &encoder,+ &device.kernels,+ name,+ layout.dims(),+ src,+ layout.stride(),+ &buffer,+ mul as f32,+ add as f32,+ )+ .map_err(MetalError::from)?;+ }+ Ok(Self::new(buffer, device.clone(), el, dtype))+ }++ fn powf(&self, layout: &Layout, pow: f64) -> Result<Self> {+ let device = self.device().clone();++ let shape = layout.shape();+ let el = shape.elem_count();+ let dtype = self.dtype;++ let buffer = device+ .new_buffer_builder()+ .with_size_for(el, self.dtype)+ .with_label("powf")+ .build()?;+ let encoder = self.device.command_encoder()?;+ let src = buffer_o(&self.buffer, layout, dtype);+ if layout.is_contiguous() {+ let name = match self.dtype {+ DType::F32 => "powf_f32",+ DType::F16 => "powf_f16",+ DType::BF16 => "powf_bf16",+ dtype => crate::bail!("Metal contiguous powf {dtype:?} not implemented"),+ };+ candle_metal_kernels::call_powf(+ &device.device,+ &encoder,+ &device.kernels,+ name,+ self.dtype.size_in_bytes(),+ el,+ src,+ &buffer,+ pow as f32,+ )+ .map_err(MetalError::from)?;+ } else {+ let name = match self.dtype {+ DType::F32 => "powf_f32_strided",+ DType::F16 => "powf_f16_strided",+ DType::BF16 => "powf_bf16_strided",+ dtype => crate::bail!("Metal strided powf {dtype:?} not implemented"),+ };+ candle_metal_kernels::call_powf_strided(+ &device.device,+ &encoder,+ &device.kernels,+ name,+ layout.dims(),+ src,+ layout.stride(),+ &buffer,+ pow as f32,+ )+ .map_err(MetalError::from)?;+ }+ Ok(Self::new(buffer, device.clone(), el, dtype))+ }++ fn elu(&self, layout: &Layout, alpha: f64) -> Result<Self> {+ let device = self.device().clone();++ let shape = layout.shape();+ let el = shape.elem_count();+ let dtype = self.dtype;++ let buffer = device+ .new_buffer_builder()+ .with_size_for(el, self.dtype)+ .with_label("elu")+ .build()?;+ let encoder = self.device.command_encoder()?;+ let src = buffer_o(&self.buffer, layout, self.dtype);+ if layout.is_contiguous() {+ let name = match self.dtype {+ DType::F32 => "elu_f32",+ DType::F16 => "elu_f16",+ DType::BF16 => "elu_bf16",+ dtype => crate::bail!("Metal contiguous elu {dtype:?} not implemented"),+ };+ candle_metal_kernels::call_elu(+ &device.device,+ &encoder,+ &device.kernels,+ name,+ self.dtype.size_in_bytes(),+ el,+ src,+ &buffer,+ alpha as f32,+ )+ .map_err(MetalError::from)?;+ } else {+ let name = match self.dtype {+ DType::F32 => "elu_f32_strided",+ DType::F16 => "elu_f16_strided",+ DType::BF16 => "elu_bf16_strided",+ dtype => crate::bail!("Metal strided elu {dtype:?} not implemented"),+ };+ candle_metal_kernels::call_elu_strided(+ &device.device,+ &encoder,+ &device.kernels,+ name,+ layout.dims(),+ src,+ layout.stride(),+ &buffer,+ alpha as f32,+ )+ .map_err(MetalError::from)?;+ }+ Ok(Self::new(buffer, device.clone(), el, dtype))+ }++ fn reduce_op(&self, op: ReduceOp, layout: &Layout, sum_dims: &[usize]) -> Result<Self> {+ let device = self.device.clone();++ let src_stride = layout.stride();+ let src_dims = layout.shape().dims();+ // Source dims and strides with the sum dims at the end.+ let mut dims = vec![];+ let mut stride = vec![];+ let mut dst_el: usize = 1;+ for (dim_idx, &d) in src_dims.iter().enumerate() {+ if !sum_dims.contains(&dim_idx) {+ dst_el *= d;+ dims.push(d);+ stride.push(src_stride[dim_idx]);+ }+ }++ for &dim_idx in sum_dims.iter() {+ dims.push(src_dims[dim_idx]);+ stride.push(src_stride[dim_idx]);+ }++ let reduction_shape = Shape::from(dims.clone());++ if layout.is_contiguous() && reduction_shape.is_contiguous(&stride) {+ let (name, check_empty, return_index) = match (op, self.dtype) {+ (ReduceOp::Sum, DType::F32) => ("fast_sum_f32", false, false),+ (ReduceOp::Min, DType::F32) => ("fast_min_f32", true, false),+ (ReduceOp::Max, DType::F32) => ("fast_max_f32", true, false),+ (ReduceOp::ArgMin, DType::F32) => ("fast_argmin_f32", true, true),+ (ReduceOp::ArgMax, DType::F32) => ("fast_argmax_f32", true, true),+ (ReduceOp::Sum, DType::U32) => ("fast_sum_u32", false, false),+ (ReduceOp::Min, DType::U32) => ("fast_min_u32", true, false),+ (ReduceOp::Max, DType::U32) => ("fast_max_u32", true, false),+ (ReduceOp::ArgMin, DType::U32) => ("fast_argmin_u32", true, true),+ (ReduceOp::ArgMax, DType::U32) => ("fast_argmax_u32", true, true),+ (ReduceOp::Sum, DType::F16) => ("fast_sum_f16", false, false),+ (ReduceOp::Min, DType::F16) => ("fast_min_f16", true, false),+ (ReduceOp::Max, DType::F16) => ("fast_max_f16", true, false),+ (ReduceOp::ArgMin, DType::F16) => ("fast_argmin_f16", true, true),+ (ReduceOp::ArgMax, DType::F16) => ("fast_argmax_f16", true, true),+ (ReduceOp::Sum, DType::BF16) => ("fast_sum_bf16", false, false),+ (ReduceOp::Min, DType::BF16) => ("fast_min_bf16", true, false),+ (ReduceOp::Max, DType::BF16) => ("fast_max_bf16", true, false),+ (ReduceOp::ArgMin, DType::BF16) => ("fast_argmin_bf16", true, true),+ (ReduceOp::ArgMax, DType::BF16) => ("fast_argmax_bf16", true, true),+ (ReduceOp::Sum, DType::I64) => ("fast_sum_i64", false, false),+ (ReduceOp::Min, DType::I64) => ("fast_min_i64", true, false),+ (ReduceOp::Max, DType::I64) => ("fast_max_i64", true, false),+ (ReduceOp::ArgMin, DType::I64) => ("fast_argmin_i64", true, true),+ (ReduceOp::ArgMax, DType::I64) => ("fast_argmax_i64", true, true),+ (ReduceOp::Sum, DType::U8) => ("fast_sum_u8", false, false),+ (ReduceOp::Min, DType::U8) => ("fast_min_u8", true, false),+ (ReduceOp::Max, DType::U8) => ("fast_max_u8", true, false),+ (ReduceOp::ArgMin, DType::U8) => ("fast_argmin_u8", true, true),+ (ReduceOp::ArgMax, DType::U8) => ("fast_argmax_u8", true, true),+ (k, dtype) => {+ crate::bail!("Metal contiguous reduce op {k:?} {dtype:?} not implemented")+ }+ };+ if check_empty && layout.shape().elem_count() == 0 {+ Err(crate::Error::EmptyTensor { op: "reduce" }.bt())?+ }+ let dtype = if return_index { DType::U32 } else { self.dtype };+ let buffer = device+ .new_buffer_builder()+ .with_size_for(dst_el, dtype)+ .with_label("reduce")+ .build()?;+ let encoder = self.device.command_encoder()?;+ let src = buffer_o(&self.buffer, layout, self.dtype);+ candle_metal_kernels::call_reduce_contiguous(+ &device.device,+ &encoder,+ &device.kernels,+ name,+ src_dims,+ dst_el,+ src,+ &buffer,+ )+ .map_err(MetalError::from)?;++ return Ok(Self::new(buffer, device, dst_el, dtype));+ }++ let (name, check_empty, return_index) = match (op, self.dtype) {+ (ReduceOp::Sum, DType::F32) => ("fast_sum_f32_strided", false, false),+ (ReduceOp::Min, DType::F32) => ("fast_min_f32_strided", true, false),+ (ReduceOp::Max, DType::F32) => ("fast_max_f32_strided", true, false),+ (ReduceOp::ArgMin, DType::F32) => ("fast_argmin_f32_strided", true, true),+ (ReduceOp::ArgMax, DType::F32) => ("fast_argmax_f32_strided", true, true),+ (ReduceOp::Sum, DType::U32) => ("fast_sum_u32_strided", false, false),+ (ReduceOp::Min, DType::U32) => ("fast_min_u32_strided", true, false),+ (ReduceOp::Max, DType::U32) => ("fast_max_u32_strided", true, false),+ (ReduceOp::ArgMin, DType::U32) => ("fast_argmin_u32_strided", true, true),+ (ReduceOp::ArgMax, DType::U32) => ("fast_argmax_u32_strided", true, true),+ (ReduceOp::Sum, DType::F16) => ("fast_sum_f16_strided", false, false),+ (ReduceOp::Min, DType::F16) => ("fast_min_f16_strided", true, false),+ (ReduceOp::Max, DType::F16) => ("fast_max_f16_strided", true, false),+ (ReduceOp::ArgMin, DType::F16) => ("fast_argmin_f16_strided", true, true),+ (ReduceOp::ArgMax, DType::F16) => ("fast_argmax_f16_strided", true, true),+ (ReduceOp::Sum, DType::BF16) => ("fast_sum_bf16_strided", false, false),+ (ReduceOp::Min, DType::BF16) => ("fast_min_bf16_strided", true, false),+ (ReduceOp::Max, DType::BF16) => ("fast_max_bf16_strided", true, false),+ (ReduceOp::ArgMin, DType::BF16) => ("fast_argmin_bf16_strided", true, true),+ (ReduceOp::ArgMax, DType::BF16) => ("fast_argmax_bf16_strided", true, true),+ (ReduceOp::Sum, DType::I64) => ("fast_sum_i64_strided", false, false),+ (ReduceOp::Min, DType::I64) => ("fast_min_i64_strided", true, false),+ (ReduceOp::Max, DType::I64) => ("fast_max_i64_strided", true, false),+ (ReduceOp::ArgMin, DType::I64) => ("fast_argmin_i64_strided", true, true),+ (ReduceOp::ArgMax, DType::I64) => ("fast_argmax_i64_strided", true, true),+ (ReduceOp::Sum, DType::U8) => ("fast_sum_u8_strided", false, false),+ (ReduceOp::Min, DType::U8) => ("fast_min_u8_strided", true, false),+ (ReduceOp::Max, DType::U8) => ("fast_max_u8_strided", true, false),+ (ReduceOp::ArgMin, DType::U8) => ("fast_argmin_u8_strided", true, true),+ (ReduceOp::ArgMax, DType::U8) => ("fast_argmax_u8_strided", true, true),+ (k, dtype) => crate::bail!("Metal strided reduce op {k:?} {dtype:?} not implemented"),+ };+ if check_empty && layout.shape().elem_count() == 0 {+ Err(crate::Error::EmptyTensor { op: "reduce" }.bt())?+ }+ let dtype = if return_index { DType::U32 } else { self.dtype };+ let buffer = device+ .new_buffer_builder()+ .with_size_for(dst_el, dtype)+ .with_label("reduce")+ .build()?;+ let encoder = self.device.command_encoder()?;+ let src = buffer_o(&self.buffer, layout, self.dtype);+ candle_metal_kernels::call_reduce_strided(+ &device.device,+ &encoder,+ &device.kernels,+ name,+ &dims,+ &stride,+ dst_el,+ src,+ &buffer,+ )+ .map_err(MetalError::from)?;++ Ok(Self::new(buffer, device, dst_el, dtype))+ }++ fn cmp(&self, op: CmpOp, rhs: &Self, lhs_l: &Layout, rhs_l: &Layout) -> Result<Self> {+ let name = match op {+ CmpOp::Eq => "eq",+ CmpOp::Ne => "ne",+ CmpOp::Le => "le",+ CmpOp::Ge => "ge",+ CmpOp::Lt => "lt",+ CmpOp::Gt => "gt",+ };+ self.binary(name, rhs, lhs_l, rhs_l)+ }++ fn const_set(&mut self, s: crate::scalar::Scalar, l: &Layout) -> Result<()> {+ use crate::scalar::Scalar;+ fn set<S: crate::WithDType + candle_metal_kernels::utils::EncoderParam>(+ self_: &mut MetalStorage,+ s: S,+ l: &Layout,+ ) -> Result<()> {+ let device = self_.device();+ let dtype = self_.dtype;+ let shape = l.shape();+ let el_count = shape.elem_count();+ let encoder = device.command_encoder()?;+ let dst = buffer_o(&self_.buffer, l, self_.dtype);++ if l.is_contiguous() {+ use candle_metal_kernels::unary::contiguous;+ let kernel_name = match dtype {+ DType::F16 => contiguous::const_set::HALF,+ DType::BF16 => contiguous::const_set::BFLOAT,+ DType::F32 => contiguous::const_set::FLOAT,+ DType::I64 => contiguous::const_set::I64,+ DType::U32 => contiguous::const_set::U32,+ DType::U8 => contiguous::const_set::U8,+ DType::F8E4M3 => crate::bail!("unsupported const-set f8e4m3"),+ DType::F64 => crate::bail!("unsupported const-set f64"),+ DType::F4+ | DType::F6E2M3+ | DType::F6E3M2+ | DType::F8E8M0+ | DType::I16+ | DType::I32 => {+ return Err(Error::UnsupportedDTypeForOp(dtype, "const-set").bt())+ }+ };+ candle_metal_kernels::call_const_set_contiguous(+ &device.device,+ &encoder,+ &device.kernels,+ kernel_name,+ dtype.size_in_bytes(),+ el_count,+ s,+ dst,+ )+ .map_err(MetalError::from)?;+ } else {+ use candle_metal_kernels::unary::strided;+ let kernel_name = match dtype {+ DType::F16 => strided::const_set::HALF,+ DType::BF16 => strided::const_set::BFLOAT,+ DType::F32 => strided::const_set::FLOAT,+ DType::I64 => strided::const_set::I64,+ DType::U32 => strided::const_set::U32,+ DType::U8 => strided::const_set::U8,+ DType::F8E4M3 => crate::bail!("unsupported const-set f8e4m3"),+ DType::F64 => crate::bail!("unsupported const-set f64"),+ DType::F4+ | DType::F6E2M3+ | DType::F6E3M2+ | DType::F8E8M0+ | DType::I16+ | DType::I32 => {+ return Err(Error::UnsupportedDTypeForOp(dtype, "const-set").bt())+ }+ };+ candle_metal_kernels::call_const_set_strided(+ &device.device,+ &encoder,+ &device.kernels,+ kernel_name,+ l.dims(),+ s,+ l.stride(),+ dst,+ )+ .map_err(MetalError::from)?;+ }+ Ok(())+ }+ match (self.dtype, s) {+ (DType::U8, Scalar::U8(s)) => set(self, s, l),+ (DType::U32, Scalar::U32(s)) => set(self, s, l),+ (DType::I64, Scalar::I64(s)) => set(self, s, l),+ (DType::F16, Scalar::F16(s)) => set(self, s, l),+ (DType::BF16, Scalar::BF16(s)) => set(self, s, l),+ (DType::F32, Scalar::F32(s)) => set(self, s, l),+ (DType::F64, Scalar::F64(s)) => set(self, s, l),+ _ => crate::bail!("dtype mismatch, expected {:?}, got {:?}", self.dtype, s),+ }+ }++ fn to_dtype(&self, layout: &Layout, dtype: DType) -> Result<Self> {+ let device = self.device();+ let shape = layout.shape();+ let el_count = shape.elem_count();+ let buffer = device+ .new_buffer_builder()+ .with_size_for(el_count, dtype)+ .with_label("to_dtype")+ .build()?;+ let encoder = device.command_encoder()?;+ let src = buffer_o(&self.buffer, layout, self.dtype);+ if layout.is_contiguous() {+ let kernel_name = match (self.dtype, dtype) {+ (DType::U32, DType::BF16) => "cast_u32_bf16",+ (DType::U32, DType::F16) => "cast_u32_f16",+ (DType::U32, DType::F32) => "cast_u32_f32",+ (DType::U32, DType::I64) => "cast_u32_i64",+ (DType::U32, DType::U8) => "cast_u32_u8",++ (DType::U8, DType::BF16) => "cast_u8_bf16",+ (DType::U8, DType::F16) => "cast_u8_f16",+ (DType::U8, DType::F32) => "cast_u8_f32",+ (DType::U8, DType::I64) => "cast_u8_i64",+ (DType::U8, DType::U32) => "cast_u8_u32",++ (DType::F32, DType::BF16) => "cast_f32_bf16",+ (DType::F32, DType::F16) => "cast_f32_f16",+ (DType::F32, DType::I64) => "cast_f32_i64",+ (DType::F32, DType::U32) => "cast_f32_u32",+ (DType::F32, DType::U8) => "cast_f32_u8",++ (DType::I64, DType::BF16) => "cast_i64_bf16",+ (DType::I64, DType::F16) => "cast_i64_f16",+ (DType::I64, DType::F32) => "cast_i64_f32",+ (DType::I64, DType::U32) => "cast_i64_u32",+ (DType::I64, DType::U8) => "cast_i64_u8",++ (DType::F16, DType::BF16) => "cast_f16_bf16",+ (DType::F16, DType::F32) => "cast_f16_f32",+ (DType::F16, DType::I64) => "cast_f16_i64",+ (DType::F16, DType::U32) => "cast_f16_u32",+ (DType::F16, DType::U8) => "cast_f16_u8",++ (DType::BF16, DType::F16) => "cast_bf16_f16",+ (DType::BF16, DType::F32) => "cast_bf16_f32",+ (DType::BF16, DType::I64) => "cast_bf16_i64",+ (DType::BF16, DType::U32) => "cast_bf16_u32",+ (DType::BF16, DType::U8) => "cast_bf16_u8",++ (left, right) => {+ crate::bail!("Metal contiguous to_dtype {left:?} {right:?} not implemented")+ }+ };+ candle_metal_kernels::call_cast_contiguous(+ &device.device,+ &encoder,+ &device.kernels,+ kernel_name,+ self.dtype.size_in_bytes(),+ el_count,+ src,+ &buffer,+ )+ .map_err(MetalError::from)?;+ } else {+ let kernel_name = match (self.dtype, dtype) {+ (DType::BF16, DType::F16) => "cast_bf16_f16_strided",+ (DType::BF16, DType::F32) => "cast_bf16_f32_strided",+ (DType::BF16, DType::I64) => "cast_bf16_i64_strided",+ (DType::BF16, DType::U32) => "cast_bf16_u32_strided",+ (DType::BF16, DType::U8) => "cast_bf16_u8_strided",++ (DType::F16, DType::BF16) => "cast_f16_bf16_strided",+ (DType::F16, DType::F32) => "cast_f16_f32_strided",+ (DType::F16, DType::I64) => "cast_f16_i64_strided",+ (DType::F16, DType::U32) => "cast_f16_u32_strided",+ (DType::F16, DType::U8) => "cast_f16_u8_strided",++ (DType::F32, DType::BF16) => "cast_f32_bf16_strided",+ (DType::F32, DType::F16) => "cast_f32_f16_strided",+ (DType::F32, DType::I64) => "cast_f32_i64_strided",+ (DType::F32, DType::U32) => "cast_f32_u32_strided",+ (DType::F32, DType::U8) => "cast_f32_u8_strided",++ (DType::I64, DType::F32) => "cast_i64_f32_strided",+ (DType::I64, DType::BF16) => "cast_i64_bf16_strided",+ (DType::I64, DType::F16) => "cast_i64_f16_strided",+ (DType::I64, DType::U32) => "cast_i64_u32_strided",+ (DType::I64, DType::U8) => "cast_i64_u8_strided",++ (DType::U32, DType::BF16) => "cast_u32_bf16_strided",+ (DType::U32, DType::F16) => "cast_u32_f16_strided",+ (DType::U32, DType::F32) => "cast_u32_f32_strided",+ (DType::U32, DType::I64) => "cast_u32_i64_strided",+ (DType::U32, DType::U8) => "cast_u32_u8_strided",++ (DType::U8, DType::BF16) => "cast_u8_bf16_strided",+ (DType::U8, DType::F16) => "cast_u8_f16_strided",+ (DType::U8, DType::F32) => "cast_u8_f32_strided",+ (DType::U8, DType::I64) => "cast_u8_i64_strided",+ (DType::U8, DType::U32) => "cast_u8_u32_strided",++ (left, right) => {+ crate::bail!("Metal strided to_dtype {left:?} {right:?} not implemented")+ }+ };+ candle_metal_kernels::call_cast_strided(+ &device.device,+ &encoder,+ &device.kernels,+ kernel_name,+ layout.dims(),+ src,+ layout.stride(),+ &buffer,+ )+ .map_err(MetalError::from)?;+ }+ Ok(Self::new(buffer, device.clone(), el_count, dtype))+ }++ fn unary_impl<B: UnaryOpT>(&self, layout: &Layout) -> Result<Self> {+ let device = self.device();+ let dtype = self.dtype;+ let shape = layout.shape();+ let el_count = shape.elem_count();+ let buffer = device+ .new_buffer_builder()+ .with_size_for(el_count, dtype)+ .with_label(B::KERNEL)+ .build()?;+ let encoder = device.command_encoder()?;+ let src = buffer_o(&self.buffer, layout, self.dtype);++ if layout.is_contiguous() {+ use candle_metal_kernels::unary::contiguous;+ let kernel_name = match (B::KERNEL, dtype) {+ ("uabs", DType::F16) => contiguous::abs::HALF,+ ("uabs", DType::F32) => contiguous::abs::FLOAT,+ ("uabs", DType::BF16) => contiguous::abs::BFLOAT,+ ("uceil", DType::F16) => contiguous::ceil::HALF,+ ("uceil", DType::F32) => contiguous::ceil::FLOAT,+ ("uceil", DType::BF16) => contiguous::ceil::BFLOAT,+ ("ucos", DType::F16) => contiguous::cos::HALF,+ ("ucos", DType::F32) => contiguous::cos::FLOAT,+ ("ucos", DType::BF16) => contiguous::cos::BFLOAT,+ ("uerf", DType::F16) => contiguous::erf::HALF,+ ("uerf", DType::F32) => contiguous::erf::FLOAT,+ ("uerf", DType::BF16) => contiguous::erf::BFLOAT,+ ("uexp", DType::F16) => contiguous::exp::HALF,+ ("uexp", DType::F32) => contiguous::exp::FLOAT,+ ("uexp", DType::BF16) => contiguous::exp::BFLOAT,+ ("ufloor", DType::F16) => contiguous::floor::HALF,+ ("ufloor", DType::F32) => contiguous::floor::FLOAT,+ ("ufloor", DType::BF16) => contiguous::floor::BFLOAT,+ ("ugelu_erf", DType::F16) => contiguous::gelu_erf::HALF,+ ("ugelu_erf", DType::F32) => contiguous::gelu_erf::FLOAT,+ ("ugelu_erf", DType::BF16) => contiguous::gelu_erf::BFLOAT,+ ("ugelu", DType::F16) => contiguous::gelu::HALF,+ ("ugelu", DType::F32) => contiguous::gelu::FLOAT,+ ("ugelu", DType::BF16) => contiguous::gelu::BFLOAT,+ ("ulog", DType::F16) => contiguous::log::HALF,+ ("ulog", DType::F32) => contiguous::log::FLOAT,+ ("ulog", DType::BF16) => contiguous::log::BFLOAT,+ ("uneg", DType::F16) => contiguous::neg::HALF,+ ("uneg", DType::F32) => contiguous::neg::FLOAT,+ ("uneg", DType::BF16) => contiguous::neg::BFLOAT,+ ("urecip", DType::F16) => contiguous::recip::HALF,+ ("urecip", DType::F32) => contiguous::recip::FLOAT,+ ("urecip", DType::BF16) => contiguous::recip::BFLOAT,+ ("urelu", DType::F16) => contiguous::relu::HALF,+ ("urelu", DType::F32) => contiguous::relu::FLOAT,+ ("urelu", DType::BF16) => contiguous::relu::BFLOAT,+ ("uround", DType::F16) => contiguous::round::HALF,+ ("uround", DType::F32) => contiguous::round::FLOAT,+ ("uround", DType::BF16) => contiguous::round::BFLOAT,+ ("usilu", DType::F16) => contiguous::silu::HALF,+ ("usilu", DType::F32) => contiguous::silu::FLOAT,+ ("usilu", DType::BF16) => contiguous::silu::BFLOAT,+ ("usin", DType::F16) => contiguous::sin::HALF,+ ("usin", DType::F32) => contiguous::sin::FLOAT,+ ("usin", DType::BF16) => contiguous::sin::BFLOAT,+ ("usqr", DType::F16) => contiguous::sqr::HALF,+ ("usqr", DType::F32) => contiguous::sqr::FLOAT,+ ("usqr", DType::BF16) => contiguous::sqr::BFLOAT,+ ("usqrt", DType::F16) => contiguous::sqrt::HALF,+ ("usqrt", DType::F32) => contiguous::sqrt::FLOAT,+ ("usqrt", DType::BF16) => contiguous::sqrt::BFLOAT,+ ("utanh", DType::F16) => contiguous::tanh::HALF,+ ("utanh", DType::F32) => contiguous::tanh::FLOAT,+ ("utanh", DType::BF16) => contiguous::tanh::BFLOAT,+ ("usign", DType::F16) => contiguous::sign::HALF,+ ("usign", DType::F32) => contiguous::sign::FLOAT,+ ("usign", DType::BF16) => contiguous::sign::BFLOAT,+ ("usign", DType::I64) => contiguous::sign::I64,+ (name, dtype) => {+ crate::bail!("Metal contiguous unary {name} {dtype:?} not implemented")+ }+ };++ candle_metal_kernels::call_unary_contiguous(+ &device.device,+ &encoder,+ &device.kernels,+ kernel_name,+ dtype.size_in_bytes(),+ el_count,+ src,+ &buffer,+ )+ .map_err(MetalError::from)?;+ } else {+ use candle_metal_kernels::unary::strided;+ let kernel_name = match (B::KERNEL, dtype) {+ ("ucos", DType::F32) => strided::cos::FLOAT,+ ("usin", DType::F32) => strided::sin::FLOAT,+ ("usqr", DType::F32) => strided::sqr::FLOAT,+ ("usqrt", DType::F32) => strided::sqrt::FLOAT,+ ("uneg", DType::F32) => strided::neg::FLOAT,+ ("uexp", DType::F32) => strided::exp::FLOAT,+ ("ulog", DType::F32) => strided::log::FLOAT,+ ("ugelu", DType::F32) => strided::gelu::FLOAT,+ ("ugelu_erf", DType::F32) => strided::gelu_erf::FLOAT,+ ("uerf", DType::F32) => strided::erf::FLOAT,+ ("usilu", DType::F32) => strided::silu::FLOAT,+ ("uabs", DType::F32) => strided::abs::FLOAT,+ ("uceil", DType::F32) => strided::ceil::FLOAT,+ ("ufloor", DType::F32) => strided::floor::FLOAT,+ ("urelu", DType::F32) => strided::relu::FLOAT,+ ("uround", DType::F32) => strided::round::FLOAT,+ ("utanh", DType::F32) => strided::tanh::FLOAT,++ ("ucos", DType::F16) => strided::cos::HALF,+ ("usin", DType::F16) => strided::sin::HALF,+ ("usqr", DType::F16) => strided::sqr::HALF,+ ("usqrt", DType::F16) => strided::sqrt::HALF,+ ("uneg", DType::F16) => strided::neg::HALF,+ ("uexp", DType::F16) => strided::exp::HALF,+ ("ulog", DType::F16) => strided::log::HALF,+ ("ugelu", DType::F16) => strided::gelu::HALF,+ ("ugelu_erf", DType::F16) => strided::gelu_erf::HALF,+ ("uerf", DType::F16) => strided::erf::HALF,+ ("usilu", DType::F16) => strided::silu::HALF,+ ("uabs", DType::F16) => strided::abs::HALF,+ ("uceil", DType::F16) => strided::ceil::HALF,+ ("ufloor", DType::F16) => strided::floor::HALF,+ ("urelu", DType::F16) => strided::relu::HALF,+ ("uround", DType::F16) => strided::round::HALF,+ ("utanh", DType::F16) => strided::tanh::HALF,++ ("ucos", DType::BF16) => strided::cos::BFLOAT,+ ("usin", DType::BF16) => strided::sin::BFLOAT,+ ("usqr", DType::BF16) => strided::sqr::BFLOAT,+ ("usqrt", DType::BF16) => strided::sqrt::BFLOAT,+ ("uneg", DType::BF16) => strided::neg::BFLOAT,+ ("uexp", DType::BF16) => strided::exp::BFLOAT,+ ("ulog", DType::BF16) => strided::log::BFLOAT,+ ("ugelu", DType::BF16) => strided::gelu::BFLOAT,+ ("ugelu_erf", DType::BF16) => strided::gelu_erf::BFLOAT,+ ("uerf", DType::BF16) => strided::erf::BFLOAT,+ ("usilu", DType::BF16) => strided::silu::BFLOAT,+ ("uabs", DType::BF16) => strided::abs::BFLOAT,+ ("uceil", DType::BF16) => strided::ceil::BFLOAT,+ ("ufloor", DType::BF16) => strided::floor::BFLOAT,+ ("urelu", DType::BF16) => strided::relu::BFLOAT,+ ("uround", DType::BF16) => strided::round::BFLOAT,+ ("utanh", DType::BF16) => strided::tanh::BFLOAT,++ (name, dtype) => {+ crate::bail!("Metal strided unary {name} {dtype:?} not implemented")+ }+ };+ let dst = BufferOffset::zero_offset(&buffer);+ candle_metal_kernels::call_unary_strided(+ &device.device,+ &encoder,+ &device.kernels,+ kernel_name,+ layout.dims(),+ src,+ layout.stride(),+ dst,+ )+ .map_err(MetalError::from)?;+ }++ Ok(Self::new(buffer, device.clone(), el_count, dtype))+ }++ fn binary_impl<B: BinaryOpT>(+ &self,+ rhs: &Self,+ lhs_l: &Layout,+ rhs_l: &Layout,+ ) -> Result<Self> {+ self.binary(B::KERNEL, rhs, lhs_l, rhs_l)+ }++ fn where_cond(+ &self,+ layout: &Layout,+ t: &Self,+ t_l: &Layout,+ f: &Self,+ f_l: &Layout,+ ) -> Result<Self> {+ let device = self.device.clone();+ let shape = t_l.shape();+ let dims = shape.dims();+ let el = shape.elem_count();+ let dtype = t.dtype;+ let buffer = self+ .device+ .new_buffer_builder()+ .with_size_for(el, dtype)+ .with_label("where")+ .build()?;+ let encoder = self.device.command_encoder()?;+ if t.dtype() != f.dtype() {+ crate::bail!(+ "Invalid where: different dtypes for values {:?} != {:?}",+ t.dtype(),+ f.dtype()+ );+ }+ let name = match (self.dtype, t.dtype()) {+ (DType::U8, DType::F32) => "where_u8_f32",+ (DType::U32, DType::F32) => "where_u32_f32",+ (DType::U8, DType::BF16) => "where_u8_bf16",+ (DType::U8, DType::F16) => "where_u8_f16",+ (DType::U8, DType::I64) => "where_u8_i64",+ (DType::U8, DType::U32) => "where_u8_u32",+ (DType::U8, DType::U8) => "where_u8_u8",+ (left, right) => crate::bail!("Metal where_cond {left:?} {right:?} not implemented"),+ };+ let src = buffer_o(&self.buffer, layout, self.dtype);+ let t = buffer_o(&t.buffer, t_l, t.dtype);+ let f = buffer_o(&f.buffer, f_l, f.dtype);+ candle_metal_kernels::call_where_cond(+ &device.device,+ &encoder,+ &device.kernels,+ name,+ dtype.size_in_bytes(),+ dims,+ src,+ layout.stride(),+ layout.is_contiguous(),+ t,+ t_l.stride(),+ t_l.is_contiguous(),+ f,+ f_l.stride(),+ f_l.is_contiguous(),+ &buffer,+ )+ .map_err(MetalError::from)?;+ Ok(Self::new(buffer, device, el, dtype))+ }++ fn conv1d(+ &self,+ layout: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &ParamsConv1D,+ ) -> Result<Self> {+ let device = self.device().clone();+ let shape = layout.shape();+ let dims = shape.dims();+ let strides = layout.stride();++ let stride = params.stride;+ let dilation = params.dilation;+ let padding = params.padding;+ let k_size = params.k_size;+ let l_out = (dims[2] + 2 * padding - dilation * (k_size - 1) - 1) / stride + 1;+ let dst_el = dims[0] * l_out * dims[1] * k_size;+ let dst = self+ .device+ .new_buffer_builder()+ .with_size_for(dst_el, self.dtype)+ .with_label("conv1d_im2col")+ .build()?;+ let encoder = self.device.command_encoder()?;+ let name = match self.dtype {+ DType::F32 => "im2col1d_f32",+ DType::F16 => "im2col1d_f16",+ DType::BF16 => "im2col1d_bf16",+ DType::U8 => "im2col1d_u8",+ DType::U32 => "im2col1d_u32",+ dtype => crate::bail!("Metal conv1d {dtype:?} not implemented"),+ };+ let src = buffer_o(&self.buffer, layout, self.dtype);+ candle_metal_kernels::call_im2col1d_strided(+ &self.device.device,+ &encoder,+ &self.device.kernels,+ name,+ layout.shape().dims(),+ strides,+ (k_size, stride, padding, dilation),+ src,+ &dst,+ )+ .map_err(MetalError::from)?;+ drop(encoder);+ let col = Self {+ buffer: dst,+ device,+ count: dst_el,+ dtype: self.dtype,+ };+ let l_out = params.l_out();+ let b = params.b_size;+ let n = params.c_out;+ let k = params.k_size * params.c_in;+ let m = l_out;+ let col_l = Layout::contiguous((b, m, k));+ let res = if kernel_l.is_contiguous() {+ let kernel_l = Layout::contiguous_with_offset((1, n, k), kernel_l.start_offset())+ .transpose(1, 2)?+ .broadcast_as((b, k, n))?;+ col.matmul(kernel, (b, m, n, k), &col_l, &kernel_l)?+ } else {+ // Make the kernel contiguous if not already the case.+ let mut kernel_c = self.device().zeros_impl(kernel_l.shape(), kernel.dtype())?;+ kernel.copy_strided_src(&mut kernel_c, 0, kernel_l)?;+ let kernel_l = Layout::contiguous_with_offset((1, n, k), kernel_l.start_offset())+ .transpose(1, 2)?+ .broadcast_as((b, k, n))?;+ col.matmul(kernel, (b, m, n, k), &col_l, &kernel_l)?+ };+ let res_l = Layout::contiguous((b, l_out, n)).transpose(1, 2)?;+ let mut res_t = self.device().zeros_impl(res_l.shape(), res.dtype())?;+ res.copy_strided_src(&mut res_t, 0, &res_l)?;+ Ok(res_t)+ }++ fn conv_transpose1d(+ &self,+ layout: &Layout,+ k: &Self,+ k_layout: &Layout,+ params: &ParamsConvTranspose1D,+ ) -> Result<Self> {+ const USE_COL2IM_CONV1D_TR: bool = true;++ let can_use_col2im = k_layout.is_contiguous()+ && params.dilation == 1+ && params.padding == 0+ && params.output_padding == 0;+ let l_out = params.l_out();+ let dst_el = params.c_out * l_out * params.b_size;++ let buffer = if USE_COL2IM_CONV1D_TR && can_use_col2im {+ let (b_size, c_in, l_in) = layout.shape().dims3()?;+ let (c_in2, c_out, k_size) = k_layout.shape().dims3()?;+ if c_in != c_in2 {+ crate::bail!(+ "convtr1d: shape mismatch on c_in {:?} {:?}",+ layout.shape(),+ k_layout.shape()+ )+ }+ let buffer = self+ .device+ .new_buffer_builder()+ .with_size_for(dst_el, self.dtype)+ .with_label("conv_transpose1d")+ .build()?;++ let name = match self.dtype {+ DType::F32 => "col2im1d_f32",+ DType::F16 => "col2im1d_f16",+ DType::BF16 => "col2im1d_bf16",+ DType::U32 => "col2im1d_u32",+ DType::U8 => "col2im1d_u8",+ dtype => crate::bail!("metal col2im1d {dtype:?} not implemented"),+ };+ let col = {+ // This merges the last two dimensions of the kernel together.+ let kernel_l_mm = Layout::new(+ (b_size, c_in, k_size * c_out).into(),+ vec![0, k_size * c_out, 1],+ k_layout.start_offset(),+ );+ self.matmul(+ k,+ (b_size, l_in, c_out * k_size, c_in),+ &layout.transpose(1, 2)?,+ &kernel_l_mm,+ )?+ };+ // It is important for the command encoder to be obtained *after* the matmul+ // kernel has run, otherwise we might use a command-buffer that has been committed+ // already resulting in the following error.+ // _status < MTLCommandBufferStatusCommitted >+ // -[IOGPUMetalCommandBuffer setCurrentCommandEncoder:]+ let encoder = self.device.command_encoder()?;+ candle_metal_kernels::call_col2im1d(+ &self.device.device,+ &encoder,+ &self.device.kernels,+ name,+ &[b_size, l_in, c_out, k_size],+ params.k_size,+ params.stride,+ BufferOffset::zero_offset(&col.buffer),+ &buffer,+ )+ .map_err(MetalError::from)?;+ buffer+ } else {+ let buffer = self+ .device+ .new_buffer_builder()+ .with_size_for(dst_el, self.dtype)+ .with_label("conv_transpose1d")+ .build()?;++ let encoder = self.device.command_encoder()?;+ let name = match self.dtype {+ DType::F32 => "conv_transpose1d_f32",+ DType::F16 => "conv_transpose1d_f16",+ DType::BF16 => "conv_transpose1d_bf16",+ DType::U32 => "conv_transpose1d_u32",+ DType::U8 => "conv_transpose1d_u8",+ dtype => crate::bail!("Metal conv_transpose1d {dtype:?} not implemented"),+ };+ candle_metal_kernels::call_conv_transpose1d(+ &self.device.device,+ &encoder,+ &self.device.kernels,+ name,+ params.dilation,+ params.stride,+ params.padding,+ params.output_padding,+ params.c_out,+ l_out,+ params.b_size,+ layout.dims(),+ layout.stride(),+ k_layout.dims(),+ k_layout.stride(),+ &self.buffer,+ layout.start_offset() * self.dtype.size_in_bytes(),+ &k.buffer,+ k_layout.start_offset() * k.dtype.size_in_bytes(),+ &buffer,+ )+ .map_err(MetalError::from)?;+ buffer+ };+ Ok(Self::new(buffer, self.device.clone(), dst_el, self.dtype))+ }++ fn conv2d(+ &self,+ layout: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &ParamsConv2D,+ ) -> Result<Self> {+ let device = self.device().clone();+ let shape = layout.shape();+ let dims = shape.dims();++ let stride = params.stride;+ let dilation = params.dilation;+ let padding = params.padding;+ let h_k = params.k_h;+ let w_k = params.k_w;+ let h = dims[2];+ let w = dims[3];+ let h_out = (h + 2 * padding - dilation * (h_k - 1) - 1) / stride + 1;+ let w_out = (w + 2 * padding - dilation * (w_k - 1) - 1) / stride + 1;+ let dst_el = dims[0] * h_out * w_out * dims[1] * h_k * w_k;++ let dst = self+ .device+ .new_buffer_builder()+ .with_size_for(dst_el, self.dtype)+ .with_label("conv2d_im2col")+ .build()?;+ let encoder = self.device.command_encoder()?;+ let name = match self.dtype {+ DType::F32 => "im2col_f32",+ DType::F16 => "im2col_f16",+ DType::BF16 => "im2col_bf16",+ DType::U8 => "im2col_u8",+ DType::U32 => "im2col_u32",+ dtype => crate::bail!("Metal conv2d {dtype:?} not implemented"),+ };+ let src = buffer_o(&self.buffer, layout, self.dtype);+ candle_metal_kernels::call_im2col_strided(+ &self.device.device,+ &encoder,+ &self.device.kernels,+ name,+ layout.shape().dims(),+ layout.stride(),+ (h_k, w_k, stride, padding, dilation),+ src,+ &dst,+ )+ .map_err(MetalError::from)?;+ drop(encoder);+ let col = Self {+ buffer: dst,+ device,+ count: dst_el,+ dtype: self.dtype,+ };+ let h_out = params.out_h();+ let w_out = params.out_w();+ let b = params.b_size;+ let n = params.c_out;+ let k = params.k_h * params.k_w * params.c_in;+ let m = h_out * w_out;+ let col_l = Layout::contiguous((b, m, k));+ let res = if kernel_l.is_contiguous() {+ let kernel_l = Layout::contiguous_with_offset((1, n, k), kernel_l.start_offset())+ .transpose(1, 2)?+ .broadcast_as((b, k, n))?;+ col.matmul(kernel, (b, m, n, k), &col_l, &kernel_l)?+ } else {+ // Make the kernel contiguous if not already the case.+ let mut kernel_c = self.device().zeros_impl(kernel_l.shape(), kernel.dtype())?;+ kernel.copy_strided_src(&mut kernel_c, 0, kernel_l)?;+ let kernel_l = Layout::contiguous_with_offset((1, n, k), kernel_l.start_offset())+ .transpose(1, 2)?+ .broadcast_as((b, k, n))?;+ col.matmul(kernel, (b, m, n, k), &col_l, &kernel_l)?+ };+ let res_l = Layout::contiguous((b, h_out, w_out, n))+ .transpose(1, 2)?+ .transpose(1, 3)?;+ let mut res_t = self.device().zeros_impl(res_l.shape(), res.dtype())?;+ res.copy_strided_src(&mut res_t, 0, &res_l)?;+ Ok(res_t)+ }++ fn conv_transpose2d(+ &self,+ l: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &ParamsConvTranspose2D,+ ) -> Result<Self> {+ // Kernel shape: (c_in_k, c_out, h_k, w_k)+ // Input shape: (b_size, c_in, h_in, w_in)+ let (out_w, out_h) = (params.out_w(), params.out_h());+ let dst_el = params.c_out * out_w * out_h * params.b_size;++ let dims = l.dims();+ if dims.len() != 4 {+ crate::bail!("unexpected input shape for conv_transpose2d {dims:?}, expected 4")+ }++ let k_dims = kernel_l.dims();+ if k_dims.len() != 4 {+ crate::bail!("unexpected kernel shape for conv_transpose2d {k_dims:?}, expected 4")+ }++ let buffer = self+ .device+ .new_buffer_builder()+ .with_size_for(dst_el, self.dtype)+ .with_label("conv_transpose2d")+ .build()?;++ let encoder = self.device.command_encoder()?;++ let name = match self.dtype {+ DType::F32 => "conv_transpose2d_f32",+ DType::F16 => "conv_transpose2d_f16",+ DType::BF16 => "conv_transpose2d_bf16",+ dtype => crate::bail!("Metal conv_transpose2d {dtype:?} not implemented"),+ };++ candle_metal_kernels::call_conv_transpose2d(+ &self.device.device,+ &encoder,+ &self.device.kernels,+ name,+ CallConvTranspose2dCfg {+ dilation: params.dilation,+ stride: params.stride,+ padding: params.padding,+ output_padding: params.output_padding,+ c_out: params.c_out,+ out_h,+ out_w,+ b_size: params.b_size,+ input_dims: l.dims(),+ input_stride: l.stride(),+ kernel_dims: kernel_l.dims(),+ kernel_stride: kernel_l.stride(),+ input_offset: l.start_offset() * self.dtype.size_in_bytes(),+ kernel_offset: kernel_l.start_offset() * kernel.dtype.size_in_bytes(),+ },+ &self.buffer,+ &kernel.buffer,+ &buffer,+ )+ .map_err(MetalError::from)?;+ Ok(Self::new(buffer, self.device.clone(), dst_el, self.dtype))+ }++ fn avg_pool2d(+ &self,+ inp_l: &Layout,+ (w_k, h_k): (usize, usize),+ (w_stride, h_stride): (usize, usize),+ ) -> Result<Self> {+ let shape = inp_l.shape();+ let (b_size, channels, width, height) = shape.dims4()?;+ let strides = inp_l.stride();+ let name = match self.dtype {+ DType::F32 => "avg_pool2d_f32",+ DType::F16 => "avg_pool2d_f16",+ DType::BF16 => "avg_pool2d_bf16",+ DType::U8 => "avg_pool2d_u8",+ DType::U32 => "avg_pool2d_u32",+ dtype => crate::bail!("Metal avg_pool2d {dtype:?} not implemented"),+ };+ let out_w = (width - w_k) / w_stride + 1;+ let out_h = (height - h_k) / h_stride + 1;+ let dst_el = out_w * out_h * b_size * channels;+ let buffer = self+ .device+ .new_buffer_builder()+ .with_size_for(dst_el, self.dtype)+ .with_label("avg_pool2d")+ .build()?;+ let encoder = self.device.command_encoder()?;+ candle_metal_kernels::call_pool2d(+ &self.device.device,+ &encoder,+ &self.device.kernels,+ name,+ inp_l.dims(),+ strides,+ out_w,+ out_h,+ w_k,+ h_k,+ w_stride,+ h_stride,+ &self.buffer,+ &buffer,+ )+ .map_err(MetalError::from)?;+ Ok(Self::new(buffer, self.device.clone(), dst_el, self.dtype))+ }++ fn max_pool2d(+ &self,+ inp_l: &Layout,+ (w_k, h_k): (usize, usize),+ (w_stride, h_stride): (usize, usize),+ ) -> Result<Self> {+ let shape = inp_l.shape();+ let (b_size, channels, width, height) = shape.dims4()?;+ let strides = inp_l.stride();+ let name = match self.dtype {+ DType::F32 => "max_pool2d_f32",+ DType::F16 => "max_pool2d_f16",+ DType::BF16 => "max_pool2d_bf16",+ DType::U8 => "max_pool2d_u8",+ DType::U32 => "max_pool2d_u32",+ dtype => crate::bail!("Metal max_pool2d {dtype:?} not implemented"),+ };+ let out_w = (width - w_k) / w_stride + 1;+ let out_h = (height - h_k) / h_stride + 1;+ let dst_el = out_w * out_h * b_size * channels;+ let buffer = self+ .device+ .new_buffer_builder()+ .with_size_for(dst_el, self.dtype)+ .with_label("max_pool2d")+ .build()?;+ let encoder = self.device.command_encoder()?;+ candle_metal_kernels::call_pool2d(+ &self.device.device,+ &encoder,+ &self.device.kernels,+ name,+ inp_l.dims(),+ strides,+ out_w,+ out_h,+ w_k,+ h_k,+ w_stride,+ h_stride,+ &self.buffer,+ &buffer,+ )+ .map_err(MetalError::from)?;+ Ok(Self::new(buffer, self.device.clone(), dst_el, self.dtype))+ }++ fn upsample_nearest1d(&self, _: &Layout, _: usize) -> Result<Self> {+ crate::bail!("Metal upsample_nearest1d not implemented")+ }++ fn upsample_nearest2d(&self, inp_l: &Layout, out_w: usize, out_h: usize) -> Result<Self> {+ // let inp = &inp.slice(inp_l.start_offset()..);+ let shape = inp_l.shape();+ let dims = shape.dims();+ let strides = inp_l.stride();+ if dims.len() != 4 {+ crate::bail!("unexpected input shape for upsample {dims:?}")+ }+ let name = match self.dtype {+ DType::F32 => "upsample_nearest2d_f32",+ DType::F16 => "upsample_nearest2d_f16",+ DType::BF16 => "upsample_nearest2d_bf16",+ DType::U8 => "upsample_nearest2d_u8",+ DType::U32 => "upsample_nearest2d_u32",+ dtype => crate::bail!("Metal upsample_nearest2d {dtype:?} not implemented"),+ };++ let dst_el = out_w * out_h * dims[0] * dims[1];+ let buffer = self+ .device+ .new_buffer_builder()+ .with_size_for(dst_el, self.dtype)+ .with_label("upsample_nearest2d")+ .build()?;+ let encoder = self.device.command_encoder()?;+ let src = buffer_o(&self.buffer, inp_l, self.dtype);+ candle_metal_kernels::call_upsample_nearest_2d(+ &self.device.device,+ &encoder,+ &self.device.kernels,+ name,+ dims,+ strides,+ out_w,+ out_h,+ src,+ &buffer,+ )+ .map_err(MetalError::from)?;+ Ok(Self::new(buffer, self.device.clone(), dst_el, self.dtype))+ }++ fn upsample_bilinear2d(+ &self,+ inp_l: &Layout,+ out_h: usize,+ out_w: usize,+ align_corners: bool,+ scale_h: Option<f64>,+ scale_w: Option<f64>,+ ) -> Result<Self> {+ let shape = inp_l.shape();+ let dims = shape.dims();+ let strides = inp_l.stride();++ if dims.len() != 4 {+ crate::bail!("unexpected input shape for upsample_bilinear2d {dims:?}")+ }++ let name = match self.dtype {+ DType::F32 => "upsample_bilinear2d_f32",+ DType::F16 => "upsample_bilinear2d_f16",+ DType::BF16 => "upsample_bilinear2d_bf16",+ DType::U8 => "upsample_bilinear2d_u8",+ DType::U32 => "upsample_bilinear2d_u32",+ dtype => crate::bail!("Metal upsample_bilinear2d {dtype:?} not implemented"),+ };++ let dst_el = out_w * out_h * dims[0] * dims[1];+ let buffer = self+ .device+ .new_buffer_builder()+ .with_size_for(dst_el, self.dtype)+ .with_label("upsample_bilinear2d")+ .build()?;++ let encoder = self.device.command_encoder()?;++ let src = buffer_o(&self.buffer, inp_l, self.dtype);+ candle_metal_kernels::call_upsample_bilinear_2d(+ &self.device.device,+ &encoder,+ &self.device.kernels,+ name,+ dims,+ strides,+ out_w,+ out_h,+ align_corners,+ scale_h,+ scale_w,+ src,+ &buffer,+ )+ .map_err(MetalError::from)?;++ Ok(Self::new(buffer, self.device.clone(), dst_el, self.dtype))+ }++ fn gather(&self, src_l: &Layout, ids: &Self, ids_l: &Layout, dim: usize) -> Result<Self> {+ if !ids_l.is_contiguous() {+ return Err(crate::Error::RequiresContiguous { op: "gather" }.bt());+ };+ let ids_el = ids_l.dims()[dim];+ let dst_el = ids_l.shape().elem_count();+ let dtype = self.dtype;+ let device = self.device();+ let buffer = device+ .new_buffer_builder()+ .with_size_for(dst_el, dtype)+ .with_label("gather")+ .build()?;+ let name = match (ids.dtype, self.dtype) {+ (DType::U8, DType::U8) => "gather_u8_u8",+ (DType::U8, DType::F32) => "gather_u8_f32",+ (DType::U8, DType::F16) => "gather_u8_f16",+ (DType::U8, DType::BF16) => "gather_u8_bf16",+ (DType::U8, DType::U32) => "gather_u8_u32",+ (DType::U8, DType::I64) => "gather_u8_i64",+ (DType::U32, DType::F32) => "gather_u32_f32",+ (DType::U32, DType::F16) => "gather_u32_f16",+ (DType::U32, DType::BF16) => "gather_u32_bf16",+ (DType::U32, DType::U32) => "gather_u32_u32",+ (DType::U32, DType::I64) => "gather_u32_i64",+ (DType::I64, DType::F32) => "gather_i64_f32",+ (DType::I64, DType::F16) => "gather_i64_f16",+ (DType::I64, DType::BF16) => "gather_i64_bf16",+ (DType::I64, DType::U32) => "gather_i64_u32",+ (DType::I64, DType::I64) => "gather_i64_i64",+ (left, right) => crate::bail!("Metal gather {left:?} {right:?} not implemented"),+ };+ let encoder = self.device.command_encoder()?;+ let src = buffer_o(&self.buffer, src_l, dtype);+ let ids = buffer_o(&ids.buffer, ids_l, ids.dtype);+ candle_metal_kernels::call_gather(+ &device.device,+ &encoder,+ &self.device.kernels,+ name,+ src_l.dims(),+ ids_el,+ dim,+ src,+ ids,+ &buffer,+ )+ .map_err(MetalError::from)?;+ Ok(Self::new(buffer, device.clone(), dst_el, dtype))+ }++ fn scatter_set(+ &mut self,+ l: &Layout,+ ids: &Self,+ ids_l: &Layout,+ src: &Self,+ src_l: &Layout,+ dim: usize,+ ) -> Result<()> {+ if !l.is_contiguous() || !ids_l.is_contiguous() || !src_l.is_contiguous() {+ return Err(crate::Error::RequiresContiguous { op: "scatter" }.bt());+ };+ let name = match (ids.dtype, self.dtype) {+ (DType::U8, DType::F32) => "s_u8_f32",+ (DType::U8, DType::F16) => "s_u8_f16",+ (DType::U8, DType::BF16) => "s_u8_bf16",+ (DType::U32, DType::U32) => "s_u32_u32",+ (DType::U32, DType::F32) => "s_u32_f32",+ (DType::U32, DType::F16) => "s_u32_f16",+ (DType::U32, DType::BF16) => "s_u32_bf16",+ (DType::I64, DType::F32) => "s_i64_f32",+ (DType::I64, DType::F16) => "s_i64_f16",+ (DType::I64, DType::BF16) => "s_i64_bf16",+ _ => Err(MetalError::UnexpectedDType {+ msg: "scatter ids should be u8/u32/i64",+ expected: DType::U32,+ got: ids.dtype(),+ })?,+ };+ let encoder = self.device.command_encoder()?;+ let dst = buffer_o(&self.buffer, l, self.dtype);+ let src = buffer_o(&src.buffer, src_l, src.dtype);+ let ids = buffer_o(&ids.buffer, ids_l, ids.dtype);+ candle_metal_kernels::call_scatter(+ &self.device.device,+ &encoder,+ &self.device.kernels,+ name,+ src_l.dims(),+ l.dims(),+ dim,+ src,+ ids,+ dst,+ )+ .map_err(MetalError::from)?;+ Ok(())+ }++ fn scatter_add_set(+ &mut self,+ l: &Layout,+ ids: &Self,+ ids_l: &Layout,+ src: &Self,+ src_l: &Layout,+ dim: usize,+ ) -> Result<()> {+ if !l.is_contiguous() || !ids_l.is_contiguous() || !src_l.is_contiguous() {+ return Err(crate::Error::RequiresContiguous { op: "scatter-add" }.bt());+ };+ let name = match (ids.dtype, self.dtype) {+ (DType::U8, DType::F32) => "sa_u8_f32",+ (DType::U8, DType::F16) => "sa_u8_f16",+ (DType::U8, DType::BF16) => "sa_u8_bf16",+ (DType::U32, DType::U32) => "sa_u32_u32",+ (DType::U32, DType::F32) => "sa_u32_f32",+ (DType::U32, DType::F16) => "sa_u32_f16",+ (DType::U32, DType::BF16) => "sa_u32_bf16",+ (DType::I64, DType::F32) => "sa_i64_f32",+ (DType::I64, DType::F16) => "sa_i64_f16",+ (DType::I64, DType::BF16) => "sa_i64_bf16",+ _ => Err(MetalError::UnexpectedDType {+ msg: "scatter-add ids should be u8/u32/i64",+ expected: DType::U32,+ got: ids.dtype(),+ })?,+ };+ let encoder = self.device.command_encoder()?;+ let dst = buffer_o(&self.buffer, l, self.dtype);+ let src = buffer_o(&src.buffer, src_l, src.dtype);+ let ids = buffer_o(&ids.buffer, ids_l, ids.dtype);+ candle_metal_kernels::call_scatter(+ &self.device.device,+ &encoder,+ &self.device.kernels,+ name,+ src_l.dims(),+ l.dims(),+ dim,+ src,+ ids,+ dst,+ )+ .map_err(MetalError::from)?;+ Ok(())+ }++ fn index_select(&self, ids: &Self, src_l: &Layout, ids_l: &Layout, dim: usize) -> Result<Self> {+ if !ids_l.is_contiguous() {+ crate::bail!("Metal index_select requires contiguous ids")+ }+ let left_size: usize = src_l.dims()[..dim].iter().product();+ let right_size: usize = src_l.dims()[dim + 1..].iter().product();+ let ids_el = ids_l.shape().elem_count();+ let dst_el = ids_el * left_size * right_size;+ let dtype = self.dtype;+ let device = self.device();+ let buffer = device+ .new_buffer_builder()+ .with_size_for(dst_el, dtype)+ .with_label("index_select")+ .build()?;+ let name = match (ids.dtype, self.dtype) {+ (DType::U8, DType::U8) => "is_u8_u8",+ (DType::U8, DType::U32) => "is_u8_u32",+ (DType::U8, DType::I64) => "is_u8_i64",+ (DType::U8, DType::BF16) => "is_u8_bf16",+ (DType::U8, DType::F32) => "is_u8_f32",+ (DType::U8, DType::F16) => "is_u8_f16",++ (DType::U32, DType::U8) => "is_u32_u8",+ (DType::U32, DType::U32) => "is_u32_u32",+ (DType::U32, DType::I64) => "is_u32_i64",+ (DType::U32, DType::F32) => "is_u32_f32",+ (DType::U32, DType::F16) => "is_u32_f16",+ (DType::U32, DType::BF16) => "is_u32_bf16",++ (DType::I64, DType::U8) => "is_i64_u8",+ (DType::I64, DType::U32) => "is_i64_u32",+ (DType::I64, DType::I64) => "is_i64_i64",+ (DType::I64, DType::F32) => "is_i64_f32",+ (DType::I64, DType::F16) => "is_i64_f16",+ (DType::I64, DType::BF16) => "is_i64_bf16",++ (left, right) => {+ crate::bail!("Metal contiguous index_select {left:?} {right:?} not implemented")+ }+ };+ let encoder = self.device.command_encoder()?;+ let src = buffer_o(&self.buffer, src_l, dtype);+ let ids = buffer_o(&ids.buffer, ids_l, ids.dtype);+ candle_metal_kernels::call_index_select(+ &device.device,+ &encoder,+ &self.device.kernels,+ name,+ src_l.dims(),+ ids_el,+ dim,+ src_l.is_contiguous(),+ src_l.dims(),+ src_l.stride(),+ src,+ ids,+ &buffer,+ )+ .map_err(MetalError::from)?;+ Ok(Self::new(buffer, device.clone(), dst_el, dtype))+ }++ fn index_add(+ &self,+ l: &Layout,+ ids: &Self,+ ids_l: &Layout,+ src: &Self,+ src_l: &Layout,+ dim: usize,+ ) -> Result<Self> {+ let mut acc = self.device.zeros_impl(l.shape(), self.dtype())?;+ self.copy_strided_src(&mut acc, 0, l)?;+ if !ids_l.is_contiguous() || !src_l.is_contiguous() {+ return Err(crate::Error::RequiresContiguous { op: "index-add" }.bt());+ };+ let name = match (ids.dtype, self.dtype) {+ (DType::I64, DType::BF16) => "ia_i64_bf16",+ (DType::I64, DType::F16) => "ia_i64_f16",+ (DType::I64, DType::F32) => "ia_i64_f32",+ (DType::I64, DType::I64) => "ia_i64_i64",+ (DType::I64, DType::U32) => "ia_i64_u32",+ (DType::I64, DType::U8) => "ia_i64_u8",++ (DType::U32, DType::BF16) => "ia_u32_bf16",+ (DType::U32, DType::F16) => "ia_u32_f16",+ (DType::U32, DType::F32) => "ia_u32_f32",+ (DType::U32, DType::I64) => "ia_u32_i64",+ (DType::U32, DType::U32) => "ia_u32_u32",+ (DType::U32, DType::U8) => "ia_u32_u8",++ (DType::U8, DType::BF16) => "ia_u8_bf16",+ (DType::U8, DType::F16) => "ia_u8_f16",+ (DType::U8, DType::F32) => "ia_u8_f32",+ (DType::U8, DType::I64) => "ia_u8_i64",+ (DType::U8, DType::U32) => "ia_u8_u32",+ (DType::U8, DType::U8) => "ia_u8_u8",++ _ => Err(MetalError::UnexpectedDType {+ msg: "index-add ids should be u8/u32/i64",+ expected: DType::U32,+ got: ids.dtype(),+ })?,+ };+ let encoder = self.device.command_encoder()?;+ let src = buffer_o(&src.buffer, src_l, src.dtype);+ let ids = buffer_o(&ids.buffer, ids_l, ids.dtype);+ candle_metal_kernels::call_index_add(+ &self.device.device,+ &encoder,+ &self.device.kernels,+ name,+ src_l.dims(),+ l.dims(),+ ids_l.dims(),+ dim,+ src,+ ids,+ &acc.buffer,+ )+ .map_err(MetalError::from)?;+ Ok(acc)+ }++ fn matmul(+ &self,+ rhs: &Self,+ (b, m, n, k): (usize, usize, usize, usize),+ lhs_l: &Layout,+ rhs_l: &Layout,+ ) -> Result<Self> {+ let buffer = self+ .device+ .new_buffer_builder()+ .with_size_for(b * m * n, self.dtype)+ .with_label("matmul")+ .build()?;+ let encoder = self.device.command_encoder()?;+ let dtype = match self.dtype {+ DType::F32 => candle_metal_kernels::GemmDType::F32,+ DType::F16 => candle_metal_kernels::GemmDType::F16,+ DType::BF16 => candle_metal_kernels::GemmDType::BF16,+ dtype => {+ return Err(+ MetalError::Message(format!("mlx matmul doesn't support {dtype:?}")).into(),+ )+ }+ };+ candle_metal_kernels::call_mlx_gemm(+ &self.device.device,+ &encoder,+ &self.device.kernels,+ dtype,+ (b, m, n, k),+ lhs_l.stride(),+ lhs_l.start_offset() * self.dtype.size_in_bytes(),+ &self.buffer,+ rhs_l.stride(),+ rhs_l.start_offset() * rhs.dtype.size_in_bytes(),+ &rhs.buffer,+ &buffer,+ )+ .map_err(MetalError::from)?;++ Ok(Self::new(+ buffer,+ self.device.clone(),+ b * m * n,+ self.dtype(),+ ))+ }++ fn copy2d(+ &self,+ dst: &mut Self,+ d1: usize,+ d2: usize,+ src_s: usize,+ dst_s: usize,+ src_o: usize,+ dst_o: usize,+ ) -> Result<()> {+ if self.dtype() != dst.dtype() {+ crate::bail!(+ "copy2d with inconsistent dtypes {:?} {:?}",+ self.dtype(),+ dst.dtype()+ )+ }+ if src_s == d2 && dst_s == d2 {+ let mut blit = self.device.blit_command_encoder()?;+ blit.set_label("copy2d_contiguous");+ let src_offset = src_o * self.dtype.size_in_bytes();+ let length = d1 * d2 * self.dtype.size_in_bytes();+ let dst_offset = dst_o * dst.dtype().size_in_bytes();+ blit.copy_from_buffer(&self.buffer, src_offset, dst.buffer(), dst_offset, length);+ } else {+ let el_count = d1 * d2;+ if el_count == 0 {+ return Ok(());+ }+ let kernel_name = match self.dtype {+ DType::F32 => candle_metal_kernels::copy2d::FLOAT,+ DType::F16 => candle_metal_kernels::copy2d::HALF,+ DType::BF16 => candle_metal_kernels::copy2d::BFLOAT,+ DType::I64 => candle_metal_kernels::copy2d::I64,+ DType::I32 => candle_metal_kernels::copy2d::I32,+ DType::I16 => candle_metal_kernels::copy2d::I16,+ DType::U32 => candle_metal_kernels::copy2d::U32,+ DType::U8 => candle_metal_kernels::copy2d::U8,+ dtype => crate::bail!("Metal copy2d {dtype:?} not implemented"),+ };+ let encoder = self.device.command_encoder()?;+ candle_metal_kernels::call_copy2d(+ &self.device.device,+ &encoder,+ &self.device.kernels,+ kernel_name,+ &self.buffer,+ &dst.buffer,+ d1,+ d2,+ src_s,+ dst_s,+ src_o * self.dtype.size_in_bytes(),+ dst_o * self.dtype.size_in_bytes(),+ )+ .map_err(MetalError::from)?;+ }+ Ok(())+ }++ fn copy_strided_src(&self, dst: &mut Self, dst_offset: usize, src_l: &Layout) -> Result<()> {+ if src_l.is_contiguous() && self.dtype == dst.dtype() {+ let mut blit = self.device.blit_command_encoder()?;+ blit.set_label("copy_contiguous");+ let src_offset = src_l.start_offset() * self.dtype.size_in_bytes();+ let length = src_l.shape().elem_count() * self.dtype.size_in_bytes();+ let dst_offset = dst_offset * dst.dtype().size_in_bytes();+ blit.copy_from_buffer(&self.buffer, src_offset, dst.buffer(), dst_offset, length);+ } else {+ let src_shape = src_l.shape();+ let el_count = src_shape.elem_count();+ if el_count == 0 {+ return Ok(());+ }+ if self.dtype == dst.dtype() {+ let strides = src_l.stride();+ let ndim = strides.len();+ // Inner blocks are contiguous - eligible for copy2d+ if ndim == 1 || strides[ndim - 1] == 1 {+ if let crate::StridedBlocks::UniformBlocks {+ start_offset,+ block_len,+ count,+ src_stride,+ } = src_l.strided_blocks()+ {+ return self.copy2d(+ dst,+ count,+ block_len,+ src_stride,+ block_len,+ start_offset,+ dst_offset,+ );+ }+ }+ }+ let kernel_name = match self.dtype {+ DType::F32 => candle_metal_kernels::unary::strided::copy::FLOAT,+ DType::F16 => candle_metal_kernels::unary::strided::copy::HALF,+ DType::BF16 => candle_metal_kernels::unary::strided::copy::BFLOAT,+ DType::I64 => candle_metal_kernels::unary::strided::copy::I64,+ DType::U32 => candle_metal_kernels::unary::strided::copy::U32,+ DType::U8 => candle_metal_kernels::unary::strided::copy::U8,+ dtype => crate::bail!("Metal copy_strided {dtype:?} not implemented"),+ };+ let src = buffer_o(&self.buffer, src_l, self.dtype);+ let dst = BufferOffset {+ buffer: &dst.buffer,+ offset_in_bytes: dst_offset * dst.dtype.size_in_bytes(),+ };+ let encoder = self.device.command_encoder()?;+ candle_metal_kernels::call_unary_strided(+ &self.device.device,+ &encoder,+ &self.device.kernels,+ kernel_name,+ src_l.dims(),+ src,+ src_l.stride(),+ dst,+ )+ .map_err(MetalError::from)?;+ }+ Ok(())+ }+}++impl MetalStorage {+ pub fn new(buffer: Arc<Buffer>, device: MetalDevice, count: usize, dtype: DType) -> Self {+ Self {+ buffer,+ device,+ count,+ dtype,+ }+ }++ pub fn buffer(&self) -> &Buffer {+ &self.buffer+ }++ pub fn binary(+ &self,+ op: &'static str,+ rhs: &Self,+ lhs_l: &Layout,+ rhs_l: &Layout,+ ) -> Result<Self> {+ fn kernel_name(op: &'static str, dtype: &DType, suffix: &str) -> String {+ format!("{op}_{}{}", dtype.as_str(), suffix)+ }+ let device = self.device();+ let shape = lhs_l.shape();+ let el_count = shape.elem_count();+ let encoder = device.command_encoder()?;+ let lhs = buffer_o(&self.buffer, lhs_l, self.dtype);+ let rhs = buffer_o(&rhs.buffer, rhs_l, rhs.dtype);++ let dtype = match op {+ "eq" | "ne" | "le" | "lt" | "ge" | "gt" => DType::U8,+ _ => self.dtype,+ };+ let lhs_is_scalar = lhs_l.is_scalar_like();+ let rhs_is_scalar = rhs_l.is_scalar_like();+ let lhs_contiguous = lhs_l.is_contiguous();+ let rhs_contiguous = rhs_l.is_contiguous();++ let contiguous_kernel = kernel_name(op, &self.dtype, "");+ let kernel = match (lhs_is_scalar, rhs_is_scalar, lhs_contiguous, rhs_contiguous) {+ (true, true, _, _) => kernel_name(op, &self.dtype, "_scalar"),+ (true, false, _, true) => kernel_name(op, &self.dtype, "_sc"),+ (true, false, _, false) => kernel_name(op, &self.dtype, "_rss"),+ (false, true, true, _) => kernel_name(op, &self.dtype, "_cs"),+ (false, true, false, _) => kernel_name(op, &self.dtype, "_lss"),+ (false, false, true, true) => contiguous_kernel.clone(),+ (false, false, true, false) => kernel_name(op, &self.dtype, "_rstrided"),+ (false, false, false, true) => kernel_name(op, &self.dtype, "_lstrided"),+ (false, false, false, false) => kernel_name(op, &self.dtype, "_strided"),+ };++ let buffer = if kernel == contiguous_kernel {+ let buffer = device+ .new_buffer_builder()+ .with_size_for(el_count, dtype)+ .with_label(op)+ .build()?;+ candle_metal_kernels::call_binary_contiguous(+ &device.device,+ &encoder,+ &device.kernels,+ kernel,+ self.dtype.size_in_bytes(),+ el_count,+ lhs,+ rhs,+ &buffer,+ )+ .map_err(MetalError::from)?;+ buffer+ } else {+ let buffer = device+ .new_buffer_builder()+ .with_size_for(el_count, dtype)+ .with_label(op)+ .build()?;+ candle_metal_kernels::call_binary_strided(+ &device.device,+ &encoder,+ &device.kernels,+ kernel,+ self.dtype.size_in_bytes(),+ lhs_l.dims(),+ lhs,+ lhs_l.stride(),+ rhs,+ rhs_l.stride(),+ &buffer,+ )+ .map_err(MetalError::from)?;+ buffer+ };+ Ok(Self::new(buffer, device.clone(), el_count, dtype))+ }++ pub(crate) fn to_cpu<T: Clone>(&self) -> Result<Vec<T>> {+ let size = self.count * self.dtype.size_in_bytes();+ let buffer = self+ .device+ .new_buffer_builder()+ .with_size(size)+ .with_label("blit_to_cpu_dst")+ .build()?;+ {+ let mut blit = self.device.blit_command_encoder()?;+ blit.set_label("blit_to_cpu");+ blit.copy_from_buffer(&self.buffer, 0, &buffer, 0, size);+ }+ self.device.flush_and_wait_current()?;+ Ok(read_to_vec(&buffer, self.count))+ }+}++impl BackendDevice for MetalDevice {+ type Storage = MetalStorage;++ fn new(ordinal: usize) -> Result<Self> {+ let device = Device::all().swap_remove(ordinal);+ let command_queue = device.new_command_queue().map_err(MetalError::from)?;+ #[cfg(feature = "metal-debug-labels")]+ command_queue.setLabel(Some(&NSString::from_str("candle-metal")));+ let kernels = Arc::new(Kernels::new());+ let residency_set = Arc::new(ResidencySet::new(&device));+ let seed_buf = device+ .new_buffer_with_data(+ [299792458u64].as_ptr() as *const c_void,+ std::mem::size_of::<u64>(),+ RESOURCE_OPTIONS,+ )+ .map_err(MetalError::from)?;+ #[cfg(feature = "metal-debug-labels")]+ seed_buf.set_label("rng_seed");+ residency_set.insert(&seed_buf);+ let seed = Arc::new(Mutex::new(seed_buf));+ let commands = Commands::new(command_queue, &residency_set).map_err(MetalError::from)?;+ Ok(Self {+ id: DeviceId::new(),+ device,+ commands: Arc::new(commands),+ buffers: Arc::new(RwLock::new(HashMap::new())),+ private_buffers: Arc::new(RwLock::new(HashMap::new())),+ kernels,+ seed,+ seed_value: Arc::new(RwLock::new(299792458)),+ residency_set,+ })+ }++ fn location(&self) -> crate::DeviceLocation {+ crate::DeviceLocation::Metal {+ gpu_id: self.registry_id() as usize,+ }+ }++ fn same_device(&self, rhs: &Self) -> bool {+ self.id == rhs.id+ }++ unsafe fn alloc_uninit(&self, shape: &Shape, dtype: DType) -> Result<MetalStorage> {+ let buffer = self+ .new_buffer_builder()+ .with_size_for(shape.elem_count(), dtype)+ .with_label("alloc-uninit")+ .build()?;+ Ok(MetalStorage::new(+ buffer,+ self.clone(),+ shape.elem_count(),+ dtype,+ ))+ }++ fn zeros_impl(&self, shape: &Shape, dtype: DType) -> Result<MetalStorage> {+ let size = shape.elem_count() * dtype.size_in_bytes();+ let buffer = self+ .new_buffer_builder()+ .with_zeros(size)+ .with_label("zeros")+ .build()?;+ Ok(MetalStorage::new(+ buffer,+ self.clone(),+ shape.elem_count(),+ dtype,+ ))+ }++ fn storage_from_slice<T: crate::WithDType>(&self, s: &[T]) -> Result<Self::Storage> {+ let label = "storage_from_slice";+ let (count, buffer) = match T::cpu_storage_ref(s) {+ CpuStorageRef::U8(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorageRef::U32(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorageRef::I16(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorageRef::I32(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorageRef::I64(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorageRef::BF16(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorageRef::F16(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorageRef::F32(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorageRef::F64(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorageRef::F8E4M3(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorageRef::F6E2M3(_)+ | CpuStorageRef::F6E3M2(_)+ | CpuStorageRef::F4(_)+ | CpuStorageRef::F8E8M0(_) => {+ return Err(Error::UnsupportedDTypeForOp(T::DTYPE, "to_dtype").bt())+ }+ };+ Ok(Self::Storage::new(buffer?, self.clone(), count, T::DTYPE))+ }++ fn storage_from_cpu_storage(&self, storage: &CpuStorage) -> Result<Self::Storage> {+ let label = "storage_from_cpu_storage";+ let (count, buffer) = match storage {+ CpuStorage::U8(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorage::U32(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorage::I16(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorage::I32(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorage::I64(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorage::BF16(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorage::F16(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorage::F32(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorage::F64(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorage::F8E4M3(storage) => (+ storage.len(),+ self.new_buffer_builder()+ .with_data(storage)+ .with_label(label)+ .build(),+ ),+ CpuStorage::F6E2M3(_)+ | CpuStorage::F6E3M2(_)+ | CpuStorage::F4(_)+ | CpuStorage::F8E8M0(_) => {+ return Err(Error::UnsupportedDTypeForOp(storage.dtype(), "to_dtype").bt())+ }+ };+ Ok(Self::Storage::new(+ buffer?,+ self.clone(),+ count,+ storage.dtype(),+ ))+ }++ fn storage_from_cpu_storage_owned(&self, storage: CpuStorage) -> Result<Self::Storage> {+ self.storage_from_cpu_storage(&storage)+ }++ fn rand_uniform(+ &self,+ shape: &Shape,+ dtype: DType,+ min: f64,+ max: f64,+ ) -> Result<Self::Storage> {+ let name = match dtype {+ DType::F32 => "rand_uniform_f32",+ DType::F16 => "rand_uniform_f16",+ DType::BF16 => "rand_uniform_bf16",+ dtype => crate::bail!("rand_uniform not implemented for {dtype:?}"),+ };+ let buffer = self+ .new_buffer_builder()+ .with_size_for(shape.elem_count(), dtype)+ .with_label("rand_uniform")+ .build()?;+ let encoder = self.command_encoder()?;+ candle_metal_kernels::call_random_uniform(+ &self.device,+ &encoder,+ &self.kernels,+ name,+ min as f32,+ max as f32,+ shape.elem_count(),+ &self.seed.lock().unwrap(),+ &buffer,+ )+ .map_err(MetalError::from)?;++ Ok(Self::Storage::new(+ buffer,+ self.clone(),+ shape.elem_count(),+ dtype,+ ))+ }++ fn rand_normal(+ &self,+ shape: &Shape,+ dtype: DType,+ mean: f64,+ stddev: f64,+ ) -> Result<Self::Storage> {+ let name = match dtype {+ DType::F32 => "rand_normal_f32",+ DType::F16 => "rand_normal_f16",+ DType::BF16 => "rand_normal_bf16",+ dtype => crate::bail!("rand_uniform not implemented for {dtype:?}"),+ };+ let buffer = self+ .new_buffer_builder()+ .with_size_for(shape.elem_count(), dtype)+ .with_label("rand_normal")+ .build()?;+ let encoder = self.command_encoder()?;+ candle_metal_kernels::call_random_normal(+ &self.device,+ &encoder,+ &self.kernels,+ name,+ mean as f32,+ stddev as f32,+ shape.elem_count(),+ &self.seed.lock().unwrap(),+ &buffer,+ )+ .map_err(MetalError::from)?;++ Ok(Self::Storage::new(+ buffer,+ self.clone(),+ shape.elem_count(),+ dtype,+ ))+ }++ fn set_seed(&self, seed: u64) -> Result<()> {+ *self.seed_value.write().unwrap() = seed;++ let seed_buffer = self.seed.try_lock().map_err(MetalError::from)?;+ let contents = seed_buffer.data();+ unsafe {+ std::ptr::copy_nonoverlapping([seed].as_ptr(), contents as *mut u64, 1);+ }+ seed_buffer.did_modify_range(NSRange::new(0, 8));++ Ok(())+ }++ fn get_current_seed(&self) -> Result<u64> {+ Ok(*self.seed_value.read().unwrap())+ }++ fn synchronize(&self) -> Result<()> {+ self.wait_until_completed()+ }+}++fn read_to_vec<T: Clone>(buffer: &Buffer, n: usize) -> Vec<T> {+ let ptr = buffer.contents() as *const T;+ assert!(!ptr.is_null());+ let slice = unsafe { std::slice::from_raw_parts(ptr, n) };+ slice.to_vec()+}diff --git a/vendor/candle-core/src/mkl.rs b/vendor/candle-core/src/mkl.rsnew file mode 100644index 0000000..359add7--- /dev/null+++ b/vendor/candle-core/src/mkl.rs@@ -0,0 +1,418 @@+#![allow(dead_code)]+use libc::{c_char, c_double, c_float, c_int};++mod ffi {+ use super::*;+ extern "C" {+ pub fn vsTanh(n: c_int, a: *const c_float, y: *mut c_float);+ pub fn vdTanh(n: c_int, a: *const c_double, y: *mut c_double);+ pub fn vsExp(n: c_int, a: *const c_float, y: *mut c_float);+ pub fn vdExp(n: c_int, a: *const c_double, y: *mut c_double);+ pub fn vsLn(n: c_int, a: *const c_float, y: *mut c_float);+ pub fn vdLn(n: c_int, a: *const c_double, y: *mut c_double);+ pub fn vsSin(n: c_int, a: *const c_float, y: *mut c_float);+ pub fn vdSin(n: c_int, a: *const c_double, y: *mut c_double);+ pub fn vsCos(n: c_int, a: *const c_float, y: *mut c_float);+ pub fn vdCos(n: c_int, a: *const c_double, y: *mut c_double);+ pub fn vsSqrt(n: c_int, a: *const c_float, y: *mut c_float);+ pub fn vdSqrt(n: c_int, a: *const c_double, y: *mut c_double);++ pub fn vsAdd(n: c_int, a: *const c_float, b: *const c_float, y: *mut c_float);+ pub fn vdAdd(n: c_int, a: *const c_double, b: *const c_double, y: *mut c_double);+ pub fn vsSub(n: c_int, a: *const c_float, b: *const c_float, y: *mut c_float);+ pub fn vdSub(n: c_int, a: *const c_double, b: *const c_double, y: *mut c_double);+ pub fn vsMul(n: c_int, a: *const c_float, b: *const c_float, y: *mut c_float);+ pub fn vdMul(n: c_int, a: *const c_double, b: *const c_double, y: *mut c_double);+ pub fn vsDiv(n: c_int, a: *const c_float, b: *const c_float, y: *mut c_float);+ pub fn vdDiv(n: c_int, a: *const c_double, b: *const c_double, y: *mut c_double);+ pub fn vsFmax(n: c_int, a: *const c_float, b: *const c_float, y: *mut c_float);+ pub fn vdFmax(n: c_int, a: *const c_double, b: *const c_double, y: *mut c_double);+ pub fn vsFmin(n: c_int, a: *const c_float, b: *const c_float, y: *mut c_float);+ pub fn vdFmin(n: c_int, a: *const c_double, b: *const c_double, y: *mut c_double);++ pub fn sgemm_(+ transa: *const c_char,+ transb: *const c_char,+ m: *const c_int,+ n: *const c_int,+ k: *const c_int,+ alpha: *const c_float,+ a: *const c_float,+ lda: *const c_int,+ b: *const c_float,+ ldb: *const c_int,+ beta: *const c_float,+ c: *mut c_float,+ ldc: *const c_int,+ );+ pub fn dgemm_(+ transa: *const c_char,+ transb: *const c_char,+ m: *const c_int,+ n: *const c_int,+ k: *const c_int,+ alpha: *const c_double,+ a: *const c_double,+ lda: *const c_int,+ b: *const c_double,+ ldb: *const c_int,+ beta: *const c_double,+ c: *mut c_double,+ ldc: *const c_int,+ );+ pub fn hgemm_(+ transa: *const c_char,+ transb: *const c_char,+ m: *const c_int,+ n: *const c_int,+ k: *const c_int,+ alpha: *const half::f16,+ a: *const half::f16,+ lda: *const c_int,+ b: *const half::f16,+ ldb: *const c_int,+ beta: *const half::f16,+ c: *mut half::f16,+ ldc: *const c_int,+ );+ }+}++#[allow(clippy::too_many_arguments)]+#[inline]+pub unsafe fn sgemm(+ transa: u8,+ transb: u8,+ m: i32,+ n: i32,+ k: i32,+ alpha: f32,+ a: &[f32],+ lda: i32,+ b: &[f32],+ ldb: i32,+ beta: f32,+ c: &mut [f32],+ ldc: i32,+) {+ ffi::sgemm_(+ &(transa as c_char),+ &(transb as c_char),+ &m,+ &n,+ &k,+ &alpha,+ a.as_ptr(),+ &lda,+ b.as_ptr(),+ &ldb,+ &beta,+ c.as_mut_ptr(),+ &ldc,+ )+}++#[allow(clippy::too_many_arguments)]+#[inline]+pub unsafe fn dgemm(+ transa: u8,+ transb: u8,+ m: i32,+ n: i32,+ k: i32,+ alpha: f64,+ a: &[f64],+ lda: i32,+ b: &[f64],+ ldb: i32,+ beta: f64,+ c: &mut [f64],+ ldc: i32,+) {+ ffi::dgemm_(+ &(transa as c_char),+ &(transb as c_char),+ &m,+ &n,+ &k,+ &alpha,+ a.as_ptr(),+ &lda,+ b.as_ptr(),+ &ldb,+ &beta,+ c.as_mut_ptr(),+ &ldc,+ )+}++#[allow(clippy::too_many_arguments)]+#[inline]+pub unsafe fn hgemm(+ transa: u8,+ transb: u8,+ m: i32,+ n: i32,+ k: i32,+ alpha: half::f16,+ a: &[half::f16],+ lda: i32,+ b: &[half::f16],+ ldb: i32,+ beta: half::f16,+ c: &mut [half::f16],+ ldc: i32,+) {+ ffi::hgemm_(+ &(transa as c_char),+ &(transb as c_char),+ &m,+ &n,+ &k,+ &alpha,+ a.as_ptr(),+ &lda,+ b.as_ptr(),+ &ldb,+ &beta,+ c.as_mut_ptr(),+ &ldc,+ )+}++#[inline]+pub fn vs_exp(a: &[f32], y: &mut [f32]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vsExp(a_len as i32, a.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vd_exp(a: &[f64], y: &mut [f64]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vdExp(a_len as i32, a.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vs_ln(a: &[f32], y: &mut [f32]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vsLn(a_len as i32, a.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vd_ln(a: &[f64], y: &mut [f64]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vdLn(a_len as i32, a.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vs_sin(a: &[f32], y: &mut [f32]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vsSin(a_len as i32, a.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vd_sin(a: &[f64], y: &mut [f64]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vdSin(a_len as i32, a.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vs_cos(a: &[f32], y: &mut [f32]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vsCos(a_len as i32, a.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vd_cos(a: &[f64], y: &mut [f64]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vdCos(a_len as i32, a.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vs_sqrt(a: &[f32], y: &mut [f32]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vsSqrt(a_len as i32, a.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vd_sqrt(a: &[f64], y: &mut [f64]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vdSqrt(a_len as i32, a.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vs_sqr(a: &[f32], y: &mut [f32]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vsMul(a_len as i32, a.as_ptr(), a.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vd_sqr(a: &[f64], y: &mut [f64]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vdMul(a_len as i32, a.as_ptr(), a.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vs_tanh(a: &[f32], y: &mut [f32]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vsTanh(a_len as i32, a.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vd_tanh(a: &[f64], y: &mut [f64]) {+ let a_len = a.len();+ let y_len = y.len();+ if a_len != y_len {+ panic!("a and y have different lengths {a_len} <> {y_len}")+ }+ unsafe { ffi::vdTanh(a_len as i32, a.as_ptr(), y.as_mut_ptr()) }+}++// The vector functions from mkl can be performed in place by using the same array for input and+// output.+// https://www.intel.com/content/www/us/en/docs/onemkl/developer-reference-c/2023-2/vector-mathematical-functions.html+#[inline]+pub fn vs_tanh_inplace(y: &mut [f32]) {+ unsafe { ffi::vsTanh(y.len() as i32, y.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vd_tanh_inplace(y: &mut [f64]) {+ unsafe { ffi::vdTanh(y.len() as i32, y.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vs_exp_inplace(y: &mut [f32]) {+ unsafe { ffi::vsExp(y.len() as i32, y.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vd_exp_inplace(y: &mut [f64]) {+ unsafe { ffi::vdExp(y.len() as i32, y.as_ptr(), y.as_mut_ptr()) }+}++#[inline]+pub fn vs_gelu(vs: &[f32], ys: &mut [f32]) {+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = (2.0f32 / std::f32::consts::PI).sqrt() * v * (1.0 + 0.044715 * v * v)+ }+ vs_tanh_inplace(ys);+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = 0.5 * v * (1.0 + *y)+ }+}++#[inline]+pub fn vd_gelu(vs: &[f64], ys: &mut [f64]) {+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = (2.0f64 / std::f64::consts::PI).sqrt() * v * (1.0 + 0.044715 * v * v)+ }+ vd_tanh_inplace(ys);+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = 0.5 * v * (1.0 + *y)+ }+}++#[inline]+pub fn vs_silu(vs: &[f32], ys: &mut [f32]) {+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = -v+ }+ vs_exp_inplace(ys);+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = v / (1.0 + *y)+ }+}++#[inline]+pub fn vd_silu(vs: &[f64], ys: &mut [f64]) {+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = -v+ }+ vd_exp_inplace(ys);+ for (&v, y) in vs.iter().zip(ys.iter_mut()) {+ *y = v / (1.0 + *y)+ }+}++macro_rules! binary_op {+ ($fn_name:ident, $ty:ty, $mkl_name:ident) => {+ #[inline]+ pub fn $fn_name(a: &[$ty], b: &[$ty], y: &mut [$ty]) {+ let a_len = a.len();+ let b_len = b.len();+ let y_len = y.len();+ if a_len != y_len || b_len != y_len {+ panic!(+ "{} a,b,y len mismatch {a_len} {b_len} {y_len}",+ stringify!($fn_name)+ );+ }+ unsafe { ffi::$mkl_name(a_len as i32, a.as_ptr(), b.as_ptr(), y.as_mut_ptr()) }+ }+ };+}+binary_op!(vs_add, f32, vsAdd);+binary_op!(vd_add, f64, vdAdd);+binary_op!(vs_sub, f32, vsSub);+binary_op!(vd_sub, f64, vdSub);+binary_op!(vs_mul, f32, vsMul);+binary_op!(vd_mul, f64, vdMul);+binary_op!(vs_div, f32, vsDiv);+binary_op!(vd_div, f64, vdDiv);+binary_op!(vs_max, f32, vsFmax);+binary_op!(vd_max, f64, vdFmax);+binary_op!(vs_min, f32, vsFmin);+binary_op!(vd_min, f64, vdFmin);diff --git a/vendor/candle-core/src/nditer.rs b/vendor/candle-core/src/nditer.rsnew file mode 100644index 0000000..1dc0494--- /dev/null+++ b/vendor/candle-core/src/nditer.rs@@ -0,0 +1,363 @@+use std::iter::FusedIterator;++use crate::Layout;++/// Maximum tensor rank supported by [`NdIter`].+pub const MAX_DIMS: usize = 8;++/// Multi-dimensional iterator that walks `N` tensor layouts simultaneously.+///+/// Internally, adjacent dimensions are merged whenever their strides allow it,+/// reducing the number of outer iterations. Each call to [`Iterator::next`] yields+/// `[usize; N]`, one base offset per layout, for one inner slice of `inner_size`+/// elements. Callers iterate that slice using `inner_strides`:+///+/// ```ignore+/// let nd_iter = NdIter::new([lhs_l, rhs_l]);+/// let inner_size = nd_iter.inner_size;+/// let [inner_ls, inner_rs] = nd_iter.inner_strides;+/// for [lhs_off, rhs_off] in nd_iter {+/// for i in 0..inner_size {+/// let lhs = lhs_buf[lhs_off + i * inner_ls];+/// let rhs = rhs_buf[rhs_off + i * inner_rs];+/// ...+/// }+/// }+/// ```+///+/// Note: `inner_strides[n]` can be 1 contiguous, but also 0 (broadcast/scalar) or > 1 (non-contiguous),+/// so callers must not assume contiguous element access within a slice.+pub struct NdIter<const N: usize> {+ pub inner_size: usize,+ /// Per-layout element stride within each inner slice: `element_offset = base + i * stride`.+ pub inner_strides: [usize; N],++ outer_dims: [usize; MAX_DIMS],+ outer_strides: [[usize; MAX_DIMS]; N],+ outer_len: usize,++ start_offsets: [usize; N],+ offsets: [usize; N],+ coords: [usize; MAX_DIMS],+ remaining: usize,+}++impl<const N: usize> NdIter<N> {+ pub fn new(layouts: [&Layout; N]) -> NdIter<N> {+ let dims = layouts[0].dims();+ debug_assert!(+ dims.len() <= MAX_DIMS,+ "rank {} exceeds MAX_DIMS={}",+ dims.len(),+ MAX_DIMS+ );+ #[cfg(debug_assertions)]+ for l in &layouts {+ debug_assert_eq!(l.dims(), dims);+ }++ let rank = dims.len();+ let mut out_dims = [0usize; MAX_DIMS];+ let mut out_strides = [[0usize; MAX_DIMS]; N];+ let mut out_len;++ if rank == 0 {+ out_dims[0] = 1;+ out_len = 1;+ } else {+ out_dims[0] = dims[0];+ for n in 0..N {+ out_strides[n][0] = layouts[n].stride()[0];+ }+ out_len = 1;++ for (i, d) in dims.iter().enumerate().take(rank).skip(1) {+ let top = out_len - 1;+ let last_d = out_dims[top];++ let (can_merge, use_inner) = if last_d == 1 {+ (true, true)+ } else if *d == 1 {+ (true, false)+ } else {+ let can_merge =+ (0..N).all(|n| out_strides[n][top] == layouts[n].stride()[i] * d);+ (can_merge, true)+ };+ if can_merge {+ out_dims[top] = last_d * d;+ if use_inner {+ for n in 0..N {+ out_strides[n][top] = layouts[n].stride()[i];+ }+ }+ } else {+ out_dims[out_len] = *d;+ for n in 0..N {+ out_strides[n][out_len] = layouts[n].stride()[i];+ }+ out_len += 1;+ }+ }+ }++ // Update inner strides+ let inner_idx = out_len - 1;+ let inner_size = out_dims[inner_idx];+ let mut inner_strides = [0usize; N];+ for n in 0..N {+ inner_strides[n] = out_strides[n][inner_idx];+ }++ // Update outer dims+ let outer_len = inner_idx;+ let mut outer_dims = [0usize; MAX_DIMS];+ outer_dims[..outer_len].copy_from_slice(&out_dims[..outer_len]);++ // Update outer strides+ let mut outer_strides = [[0usize; MAX_DIMS]; N];+ for n in 0..N {+ outer_strides[n][..outer_len].copy_from_slice(&out_strides[n][..outer_len]);+ }++ // Update offsets+ let mut offsets = [0usize; N];+ for n in 0..N {+ offsets[n] = layouts[n].start_offset();+ }++ // Number of outer blocks (product of all dims except the innermost).+ let remaining = out_dims[..outer_len].iter().product();++ NdIter {+ inner_size,+ inner_strides,+ outer_dims,+ outer_strides,+ outer_len,+ start_offsets: offsets,+ offsets,+ coords: [0; MAX_DIMS],+ remaining,+ }+ }+}++impl<const N: usize> Iterator for NdIter<N> {+ type Item = [usize; N];++ #[inline]+ fn next(&mut self) -> Option<Self::Item> {+ if self.remaining == 0 {+ return None;+ }+ let item = self.offsets;+ self.remaining -= 1;++ for k in (0..self.outer_len).rev() {+ self.coords[k] += 1;+ for n in 0..N {+ self.offsets[n] += self.outer_strides[n][k];+ }+ if self.coords[k] < self.outer_dims[k] {+ break;+ }+ self.coords[k] = 0;+ for n in 0..N {+ self.offsets[n] -= self.outer_dims[k] * self.outer_strides[n][k];+ }+ }++ Some(item)+ }++ #[inline]+ fn nth(&mut self, n: usize) -> Option<Self::Item> {+ if n >= self.remaining {+ self.remaining = 0;+ return None;+ }++ let total = self.outer_dims[..self.outer_len].iter().product::<usize>();+ let mut position = total - self.remaining + n;+ self.offsets = self.start_offsets;+ self.coords = [0; MAX_DIMS];++ for k in (0..self.outer_len).rev() {+ let coord = position % self.outer_dims[k];+ position /= self.outer_dims[k];+ self.coords[k] = coord;+ for layout in 0..N {+ self.offsets[layout] += coord * self.outer_strides[layout][k];+ }+ }+ self.remaining -= n;+ self.next()+ }++ #[inline]+ fn size_hint(&self) -> (usize, Option<usize>) {+ (self.remaining, Some(self.remaining))+ }+}++impl<const N: usize> ExactSizeIterator for NdIter<N> {}+impl<const N: usize> FusedIterator for NdIter<N> {}++#[cfg(test)]+mod tests {+ use super::*;+ use crate::shape::Shape;++ fn layout(dims: &[usize], strides: &[usize]) -> Layout {+ Layout::new(Shape::from(dims.to_vec()), strides.to_vec(), 0)+ }++ #[test]+ fn rank0_scalar() {+ let l = Layout::contiguous(());+ let mut it = NdIter::new([&l, &l]);+ assert_eq!(it.inner_size, 1);+ assert_eq!(it.inner_strides, [0, 0]);+ assert_eq!(it.len(), 1);+ assert_eq!(it.next(), Some([0, 0]));+ assert_eq!(it.next(), None);+ }++ #[test]+ fn rank1_contiguous_single_block() {+ let l = Layout::contiguous(&[5]);+ let mut it = NdIter::new([&l, &l]);+ assert_eq!(it.inner_size, 5);+ assert_eq!(it.inner_strides, [1, 1]);+ assert_eq!(it.len(), 1);+ assert_eq!(it.next(), Some([0, 0]));+ assert_eq!(it.next(), None);+ }++ #[test]+ fn rank2_contiguous_merges_to_one_block() {+ // [3, 4] strides[4, 1] -> fully merged, inner_size=12+ let l = Layout::contiguous(&[3, 4]);+ let it = NdIter::new([&l, &l]);+ assert_eq!(it.inner_size, 12);+ assert_eq!(it.inner_strides, [1, 1]);+ assert_eq!(it.len(), 1);+ }++ #[test]+ fn rank3_contiguous_fully_merged() {+ let l = Layout::contiguous(&[2, 3, 4]);+ let it = NdIter::new([&l]);+ assert_eq!(it.inner_size, 24);+ assert_eq!(it.inner_strides, [1]);+ assert_eq!(it.len(), 1);+ }++ #[test]+ fn rank3_outer_gap_partial_merge() {+ // shape [2, 3, 4]+ // strides [24, 4, 1]+ // dims 1 + 2 merge (4 == 1 * 4), dim 0 stays+ let l = layout(&[2, 3, 4], &[24, 4, 1]);+ let it = NdIter::new([&l]);+ assert_eq!(it.inner_size, 12); // 3 * 4 merged+ assert_eq!(it.inner_strides, [1]);+ assert_eq!(it.len(), 2); // outer: dim 0 has size 2+ let offsets: Vec<_> = it.collect();+ assert_eq!(offsets, vec![[0], [24]]);+ }++ #[test]+ fn rank2_no_merge() {+ // shape [3, 4]+ // strides [1, 3]+ // stride[0] = 1 != stride[1] * dim[1] = 12 -> no merge+ let l = layout(&[3, 4], &[1, 3]);+ let it = NdIter::new([&l, &l]);+ assert_eq!(it.inner_size, 4);+ assert_eq!(it.inner_strides, [3, 3]);+ assert_eq!(it.len(), 3);+ let offsets: Vec<_> = it.collect();+ assert_eq!(offsets, vec![[0, 0], [1, 1], [2, 2]]);+ }++ #[test]+ fn broadcast_zeros_merge() {+ // shape [3, 4]+ // strides [0, 0]+ // 0 == 0 * 4 -> dims merge+ let l = layout(&[3, 4], &[0, 0]);+ let it = NdIter::new([&l, &l]);+ assert_eq!(it.inner_size, 12);+ assert_eq!(it.inner_strides, [0, 0]);+ assert_eq!(it.len(), 1);+ }++ #[test]+ fn mixed_contiguous_and_broadcast_merge() {+ // lhs [4, 1] contiguous+ // rhs [0,0] broadcast+ // both conditions pass -> merge+ let lhs = Layout::contiguous(&[3, 4]);+ let rhs = layout(&[3, 4], &[0, 0]);+ let it = NdIter::new([&lhs, &rhs]);+ assert_eq!(it.inner_size, 12);+ assert_eq!(it.inner_strides, [1, 0]);+ assert_eq!(it.len(), 1);+ }++ #[test]+ fn offsets_lhs_contiguous_rhs_strided() {+ // lhs row-major [2,3] strides [3,1]+ // rhs col-major [2,3] strides [1,2]+ // lhs can merge (3 == 1 * 3)+ // rhs can not (1 ≠ 2 * 3 = 6) -> no merge for the pair+ let lhs = Layout::contiguous(&[2, 3]);+ let rhs = layout(&[2, 3], &[1, 2]);+ let it = NdIter::new([&lhs, &rhs]);+ assert_eq!(it.inner_size, 3);+ assert_eq!(it.inner_strides, [1, 2]);+ assert_eq!(it.len(), 2);+ let offsets: Vec<_> = it.collect();+ assert_eq!(offsets, vec![[0, 0], [3, 1]]);+ }++ #[test]+ fn start_offset_reflected_in_first_iter() {+ let l = Layout::contiguous_with_offset(4, 7);+ let mut it = NdIter::new([&l]);+ assert_eq!(it.next(), Some([7]));+ assert_eq!(it.next(), None);+ }++ #[test]+ fn start_offset_advances_with_outer_dims() {+ // shape [2, 3]+ // strides [4, 1]+ // start_offset=10+ // strides can't merge (4 != 1 * 3)+ // outer blocks at 10 and 14 (10 + 1 * outer_stride)+ let l = Layout::new(Shape::from(vec![2, 3]), vec![4, 1], 10);+ let offsets: Vec<_> = NdIter::new([&l]).collect();+ assert_eq!(offsets, vec![[10], [14]]);+ }++ #[test]+ fn nth_jumps_to_outer_block() {+ let lhs = Layout::new(Shape::from(vec![2, 3, 4]), vec![40, 10, 1], 5);+ let rhs = Layout::new(Shape::from(vec![2, 3, 4]), vec![50, 12, 1], 7);+ let expected: Vec<_> = NdIter::new([&lhs, &rhs]).collect();++ for (n, &offsets) in expected.iter().enumerate() {+ assert_eq!(NdIter::new([&lhs, &rhs]).nth(n), Some(offsets));+ }+ assert_eq!(NdIter::new([&lhs, &rhs]).nth(expected.len()), None);++ let mut iter = NdIter::new([&lhs, &rhs]);+ assert_eq!(iter.next(), Some(expected[0]));+ assert_eq!(iter.nth(2), Some(expected[3]));+ assert_eq!(iter.next(), Some(expected[4]));+ assert_eq!(iter.len(), expected.len() - 5);+ }+}diff --git a/vendor/candle-core/src/npy.rs b/vendor/candle-core/src/npy.rsnew file mode 100644index 0000000..28721e8--- /dev/null+++ b/vendor/candle-core/src/npy.rs@@ -0,0 +1,475 @@+//! Numpy support for tensors.+//!+//! The spec for the npy format can be found in+//! [npy-format](https://docs.scipy.org/doc/numpy-1.14.2/neps/npy-format.html).+//! The functions from this module can be used to read tensors from npy/npz files+//! or write tensors to these files. A npy file contains a single tensor (unnamed)+//! whereas a npz file can contain multiple named tensors. npz files are also compressed.+//!+//! These two formats are easy to use in Python using the numpy library.+//!+//! ```python+//! import numpy as np+//! x = np.arange(10)+//!+//! # Write a npy file.+//! np.save("test.npy", x)+//!+//! # Read a value from the npy file.+//! x = np.load("test.npy")+//!+//! # Write multiple values to a npz file.+//! values = { "x": x, "x_plus_one": x + 1 }+//! np.savez("test.npz", **values)+//!+//! # Load multiple values from a npz file.+//! values = np.loadz("test.npz")+//! ```+use crate::{DType, Device, Error, Result, Shape, Tensor};+use byteorder::{LittleEndian, ReadBytesExt};+use half::{bf16, f16, slice::HalfFloatSliceExt};+use std::collections::HashMap;+use std::fs::File;+use std::io::{BufReader, Read, Write};+use std::path::Path;++const NPY_MAGIC_STRING: &[u8] = b"\x93NUMPY";+const NPY_SUFFIX: &str = ".npy";++fn read_header<R: Read>(reader: &mut R) -> Result<String> {+ let mut magic_string = vec![0u8; NPY_MAGIC_STRING.len()];+ reader.read_exact(&mut magic_string)?;+ if magic_string != NPY_MAGIC_STRING {+ return Err(Error::Npy("magic string mismatch".to_string()));+ }+ let mut version = [0u8; 2];+ reader.read_exact(&mut version)?;+ let header_len_len = match version[0] {+ 1 => 2,+ 2 => 4,+ otherwise => return Err(Error::Npy(format!("unsupported version {otherwise}"))),+ };+ let mut header_len = vec![0u8; header_len_len];+ reader.read_exact(&mut header_len)?;+ let header_len = header_len+ .iter()+ .rev()+ .fold(0_usize, |acc, &v| 256 * acc + v as usize);+ let mut header = vec![0u8; header_len];+ reader.read_exact(&mut header)?;+ Ok(String::from_utf8_lossy(&header).to_string())+}++#[derive(Debug, PartialEq)]+struct Header {+ descr: DType,+ fortran_order: bool,+ shape: Vec<usize>,+}++impl Header {+ fn shape(&self) -> Shape {+ Shape::from(self.shape.as_slice())+ }++ fn to_string(&self) -> Result<String> {+ let fortran_order = if self.fortran_order { "True" } else { "False" };+ let mut shape = self+ .shape+ .iter()+ .map(|x| x.to_string())+ .collect::<Vec<_>>()+ .join(",");+ let descr = match self.descr {+ DType::BF16 => Err(Error::Npy("bf16 is not supported".into()))?,+ DType::F16 => "f2",+ DType::F32 => "f4",+ DType::F64 => "f8",+ DType::I16 => "i2",+ DType::I32 => "i4",+ DType::I64 => "i8",+ DType::U32 => "u4",+ DType::U8 => "u1",+ DType::F8E4M3 => Err(Error::Npy("f8e4m3 is not supported".into()))?,+ DType::F6E2M3 => Err(Error::Npy("f6e2m3 is not supported".into()))?,+ DType::F6E3M2 => Err(Error::Npy("f6e3m2 is not supported".into()))?,+ DType::F4 => Err(Error::Npy("f4 is not supported".into()))?,+ DType::F8E8M0 => Err(Error::Npy("f8e8m0 is not supported".into()))?,+ };+ if !shape.is_empty() {+ shape.push(',')+ }+ Ok(format!(+ "{{'descr': '<{descr}', 'fortran_order': {fortran_order}, 'shape': ({shape}), }}"+ ))+ }++ // Hacky parser for the npy header, a typical example would be:+ // {'descr': '<f8', 'fortran_order': False, 'shape': (128,), }+ fn parse(header: &str) -> Result<Header> {+ let header =+ header.trim_matches(|c: char| c == '{' || c == '}' || c == ',' || c.is_whitespace());++ let mut parts: Vec<String> = vec![];+ let mut start_index = 0usize;+ let mut cnt_parenthesis = 0i64;+ for (index, c) in header.char_indices() {+ match c {+ '(' => cnt_parenthesis += 1,+ ')' => cnt_parenthesis -= 1,+ ',' if cnt_parenthesis == 0 => {+ parts.push(header[start_index..index].to_owned());+ start_index = index + 1;+ }+ _ => {}+ }+ }+ parts.push(header[start_index..].to_owned());+ let mut part_map: HashMap<String, String> = HashMap::new();+ for part in parts.iter() {+ let part = part.trim();+ if !part.is_empty() {+ match part.split(':').collect::<Vec<_>>().as_slice() {+ [key, value] => {+ let key = key.trim_matches(|c: char| c == '\'' || c.is_whitespace());+ let value = value.trim_matches(|c: char| c == '\'' || c.is_whitespace());+ let _ = part_map.insert(key.to_owned(), value.to_owned());+ }+ _ => return Err(Error::Npy(format!("unable to parse header {header}"))),+ }+ }+ }+ let fortran_order = match part_map.get("fortran_order") {+ None => false,+ Some(fortran_order) => match fortran_order.as_ref() {+ "False" => false,+ "True" => true,+ _ => return Err(Error::Npy(format!("unknown fortran_order {fortran_order}"))),+ },+ };+ let descr = match part_map.get("descr") {+ None => return Err(Error::Npy("no descr in header".to_string())),+ Some(descr) => {+ if descr.is_empty() {+ return Err(Error::Npy("empty descr".to_string()));+ }+ if descr.starts_with('>') {+ return Err(Error::Npy(format!("little-endian descr {descr}")));+ }+ // the only supported types in tensor are:+ // float64, float32, float16,+ // complex64, complex128,+ // int64, int32, int16, int8,+ // uint8, and bool.+ match descr.trim_matches(|c: char| c == '=' || c == '<' || c == '|') {+ "e" | "f2" => DType::F16,+ "f" | "f4" => DType::F32,+ "d" | "f8" => DType::F64,+ "i" | "i4" => DType::I32,+ "q" | "i8" => DType::I64,+ "h" | "i2" => DType::I16,+ // "b" | "i1" => DType::S8,+ "B" | "u1" => DType::U8,+ "I" | "u4" => DType::U32,+ "?" | "b1" => DType::U8,+ // "F" | "F4" => DType::C64,+ // "D" | "F8" => DType::C128,+ descr => return Err(Error::Npy(format!("unrecognized descr {descr}"))),+ }+ }+ };+ let shape = match part_map.get("shape") {+ None => return Err(Error::Npy("no shape in header".to_string())),+ Some(shape) => {+ let shape = shape.trim_matches(|c: char| c == '(' || c == ')' || c == ',');+ if shape.is_empty() {+ vec![]+ } else {+ shape+ .split(',')+ .map(|v| v.trim().parse::<usize>())+ .collect::<std::result::Result<Vec<_>, _>>()?+ }+ }+ };+ Ok(Header {+ descr,+ fortran_order,+ shape,+ })+ }+}++impl Tensor {+ // TODO: Add the possibility to read directly to a device?+ pub(crate) fn from_reader<R: std::io::Read>(+ shape: Shape,+ dtype: DType,+ reader: &mut R,+ ) -> Result<Self> {+ let elem_count = shape.elem_count();+ match dtype {+ DType::BF16 => {+ let mut data_t = vec![bf16::ZERO; elem_count];+ reader.read_u16_into::<LittleEndian>(data_t.reinterpret_cast_mut())?;+ Tensor::from_vec(data_t, shape, &Device::Cpu)+ }+ DType::F16 => {+ let mut data_t = vec![f16::ZERO; elem_count];+ reader.read_u16_into::<LittleEndian>(data_t.reinterpret_cast_mut())?;+ Tensor::from_vec(data_t, shape, &Device::Cpu)+ }+ DType::F32 => {+ let mut data_t = vec![0f32; elem_count];+ reader.read_f32_into::<LittleEndian>(&mut data_t)?;+ Tensor::from_vec(data_t, shape, &Device::Cpu)+ }+ DType::F64 => {+ let mut data_t = vec![0f64; elem_count];+ reader.read_f64_into::<LittleEndian>(&mut data_t)?;+ Tensor::from_vec(data_t, shape, &Device::Cpu)+ }+ DType::U8 => {+ let mut data_t = vec![0u8; elem_count];+ reader.read_exact(&mut data_t)?;+ Tensor::from_vec(data_t, shape, &Device::Cpu)+ }+ DType::U32 => {+ let mut data_t = vec![0u32; elem_count];+ reader.read_u32_into::<LittleEndian>(&mut data_t)?;+ Tensor::from_vec(data_t, shape, &Device::Cpu)+ }+ DType::I16 => {+ let mut data_t = vec![0i16; elem_count];+ reader.read_i16_into::<LittleEndian>(&mut data_t)?;+ Tensor::from_vec(data_t, shape, &Device::Cpu)+ }+ DType::I32 => {+ let mut data_t = vec![0i32; elem_count];+ reader.read_i32_into::<LittleEndian>(&mut data_t)?;+ Tensor::from_vec(data_t, shape, &Device::Cpu)+ }+ DType::I64 => {+ let mut data_t = vec![0i64; elem_count];+ reader.read_i64_into::<LittleEndian>(&mut data_t)?;+ Tensor::from_vec(data_t, shape, &Device::Cpu)+ }+ DType::F8E4M3 => {+ let mut data_t = vec![0u8; elem_count];+ reader.read_exact(&mut data_t)?;+ let data_f8: Vec<float8::F8E4M3> =+ data_t.into_iter().map(float8::F8E4M3::from_bits).collect();+ Tensor::from_vec(data_f8, shape, &Device::Cpu)+ }+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ Err(Error::UnsupportedDTypeForOp(dtype, "from_reader").bt())+ }+ }+ }++ /// Reads a npy file and return the stored multi-dimensional array as a tensor.+ pub fn read_npy<T: AsRef<Path>>(path: T) -> Result<Self> {+ let mut reader = File::open(path.as_ref())?;+ let header = read_header(&mut reader)?;+ let header = Header::parse(&header)?;+ if header.fortran_order {+ return Err(Error::Npy("fortran order not supported".to_string()));+ }+ Self::from_reader(header.shape(), header.descr, &mut reader)+ }++ /// Reads a npz file and returns the stored multi-dimensional arrays together with their names.+ pub fn read_npz<T: AsRef<Path>>(path: T) -> Result<Vec<(String, Self)>> {+ let zip_reader = BufReader::new(File::open(path.as_ref())?);+ let mut zip = zip::ZipArchive::new(zip_reader)?;+ let mut result = vec![];+ for i in 0..zip.len() {+ let mut reader = zip.by_index(i)?;+ let name = {+ let name = reader.name();+ name.strip_suffix(NPY_SUFFIX).unwrap_or(name).to_owned()+ };+ let header = read_header(&mut reader)?;+ let header = Header::parse(&header)?;+ if header.fortran_order {+ return Err(Error::Npy("fortran order not supported".to_string()));+ }+ let s = Self::from_reader(header.shape(), header.descr, &mut reader)?;+ result.push((name, s))+ }+ Ok(result)+ }++ /// Reads a npz file and returns the stored multi-dimensional arrays for some specified names.+ pub fn read_npz_by_name<T: AsRef<Path>>(path: T, names: &[&str]) -> Result<Vec<Self>> {+ let zip_reader = BufReader::new(File::open(path.as_ref())?);+ let mut zip = zip::ZipArchive::new(zip_reader)?;+ let mut result = vec![];+ for name in names.iter() {+ let mut reader = match zip.by_name(&format!("{name}{NPY_SUFFIX}")) {+ Ok(reader) => reader,+ Err(_) => Err(Error::Npy(format!(+ "no array for {name} in {:?}",+ path.as_ref()+ )))?,+ };+ let header = read_header(&mut reader)?;+ let header = Header::parse(&header)?;+ if header.fortran_order {+ return Err(Error::Npy("fortran order not supported".to_string()));+ }+ let s = Self::from_reader(header.shape(), header.descr, &mut reader)?;+ result.push(s)+ }+ Ok(result)+ }++ fn write<T: Write>(&self, f: &mut T) -> Result<()> {+ f.write_all(NPY_MAGIC_STRING)?;+ f.write_all(&[1u8, 0u8])?;+ let header = Header {+ descr: self.dtype(),+ fortran_order: false,+ shape: self.dims().to_vec(),+ };+ let mut header = header.to_string()?;+ let pad = 16 - (NPY_MAGIC_STRING.len() + 5 + header.len()) % 16;+ for _ in 0..pad % 16 {+ header.push(' ')+ }+ header.push('\n');+ f.write_all(&[(header.len() % 256) as u8, (header.len() / 256) as u8])?;+ f.write_all(header.as_bytes())?;+ self.write_bytes(f)+ }++ /// Writes a multi-dimensional array in the npy format.+ pub fn write_npy<T: AsRef<Path>>(&self, path: T) -> Result<()> {+ let mut f = File::create(path.as_ref())?;+ self.write(&mut f)+ }++ /// Writes multiple multi-dimensional arrays using the npz format.+ pub fn write_npz<S: AsRef<str>, T: AsRef<Tensor>, P: AsRef<Path>>(+ ts: &[(S, T)],+ path: P,+ ) -> Result<()> {+ let mut zip = zip::ZipWriter::new(File::create(path.as_ref())?);+ let options: zip::write::FileOptions<()> =+ zip::write::FileOptions::default().compression_method(zip::CompressionMethod::Stored);++ for (name, tensor) in ts.iter() {+ zip.start_file(format!("{}.npy", name.as_ref()), options)?;+ tensor.as_ref().write(&mut zip)?+ }+ Ok(())+ }+}++/// Lazy tensor loader.+pub struct NpzTensors {+ index_per_name: HashMap<String, usize>,+ path: std::path::PathBuf,+ // We do not store a zip reader as it needs mutable access to extract data. Instead we+ // re-create a zip reader for each tensor.+}++impl NpzTensors {+ pub fn new<T: AsRef<Path>>(path: T) -> Result<Self> {+ let path = path.as_ref().to_owned();+ let zip_reader = BufReader::new(File::open(&path)?);+ let mut zip = zip::ZipArchive::new(zip_reader)?;+ let mut index_per_name = HashMap::new();+ for i in 0..zip.len() {+ let file = zip.by_index(i)?;+ let name = {+ let name = file.name();+ name.strip_suffix(NPY_SUFFIX).unwrap_or(name).to_owned()+ };+ index_per_name.insert(name, i);+ }+ Ok(Self {+ index_per_name,+ path,+ })+ }++ pub fn names(&self) -> Vec<&String> {+ self.index_per_name.keys().collect()+ }++ /// This only returns the shape and dtype for a named tensor. Compared to `get`, this avoids+ /// reading the whole tensor data.+ pub fn get_shape_and_dtype(&self, name: &str) -> Result<(Shape, DType)> {+ let index = match self.index_per_name.get(name) {+ None => crate::bail!("cannot find tensor {name}"),+ Some(index) => *index,+ };+ let zip_reader = BufReader::new(File::open(&self.path)?);+ let mut zip = zip::ZipArchive::new(zip_reader)?;+ let mut reader = zip.by_index(index)?;+ let header = read_header(&mut reader)?;+ let header = Header::parse(&header)?;+ Ok((header.shape(), header.descr))+ }++ pub fn get(&self, name: &str) -> Result<Option<Tensor>> {+ let index = match self.index_per_name.get(name) {+ None => return Ok(None),+ Some(index) => *index,+ };+ // We hope that the file has not changed since first reading it.+ let zip_reader = BufReader::new(File::open(&self.path)?);+ let mut zip = zip::ZipArchive::new(zip_reader)?;+ let mut reader = zip.by_index(index)?;+ let header = read_header(&mut reader)?;+ let header = Header::parse(&header)?;+ if header.fortran_order {+ return Err(Error::Npy("fortran order not supported".to_string()));+ }+ let tensor = Tensor::from_reader(header.shape(), header.descr, &mut reader)?;+ Ok(Some(tensor))+ }+}++#[cfg(test)]+mod tests {+ use super::Header;++ #[test]+ fn parse() {+ let h = "{'descr': '<f8', 'fortran_order': False, 'shape': (128,), }";+ assert_eq!(+ Header::parse(h).unwrap(),+ Header {+ descr: crate::DType::F64,+ fortran_order: false,+ shape: vec![128]+ }+ );+ let h = "{'descr': '<f4', 'fortran_order': True, 'shape': (256,1,128), }";+ let h = Header::parse(h).unwrap();+ assert_eq!(+ h,+ Header {+ descr: crate::DType::F32,+ fortran_order: true,+ shape: vec![256, 1, 128]+ }+ );+ assert_eq!(+ h.to_string().unwrap(),+ "{'descr': '<f4', 'fortran_order': True, 'shape': (256,1,128,), }"+ );++ let h = Header {+ descr: crate::DType::U32,+ fortran_order: false,+ shape: vec![],+ };+ assert_eq!(+ h.to_string().unwrap(),+ "{'descr': '<u4', 'fortran_order': False, 'shape': (), }"+ );+ }+}diff --git a/vendor/candle-core/src/op.rs b/vendor/candle-core/src/op.rsnew file mode 100644index 0000000..513e94e--- /dev/null+++ b/vendor/candle-core/src/op.rs@@ -0,0 +1,1204 @@+//! Tensor Operation Enums and Traits+//!+#![allow(clippy::redundant_closure_call)]+use crate::Tensor;+use float8::F8E4M3 as f8e4m3;+use half::{bf16, f16};+use num_traits::float::Float;++#[derive(Clone, Copy, PartialEq, Eq)]+pub enum CmpOp {+ Eq,+ Ne,+ Le,+ Ge,+ Lt,+ Gt,+}++#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum ReduceOp {+ Sum,+ Min,+ Max,+ ArgMin,+ ArgMax,+}++impl ReduceOp {+ pub(crate) fn name(&self) -> &'static str {+ match self {+ Self::ArgMax => "argmax",+ Self::ArgMin => "argmin",+ Self::Min => "min",+ Self::Max => "max",+ Self::Sum => "sum",+ }+ }+}++// These ops return the same type as their input type.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum BinaryOp {+ Add,+ Mul,+ Sub,+ Div,+ Maximum,+ Minimum,+}++// Unary ops with no argument+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum UnaryOp {+ Exp,+ Log,+ Sin,+ Cos,+ Abs,+ Neg,+ Recip,+ Sqr,+ Sqrt,+ Gelu,+ GeluErf,+ Erf,+ Relu,+ Silu,+ Tanh,+ Floor,+ Ceil,+ Round,+ Sign,+}++#[derive(Clone)]+pub enum Op {+ Binary(Tensor, Tensor, BinaryOp),+ Unary(Tensor, UnaryOp),+ Cmp(Tensor, CmpOp),+ // The third argument is the reduced shape with `keepdim=true`.+ Reduce(Tensor, ReduceOp, Vec<usize>),+ Matmul(Tensor, Tensor),+ Gather(Tensor, Tensor, usize),+ Scatter(Tensor, Tensor, Tensor, usize),+ ScatterAdd(Tensor, Tensor, Tensor, usize),+ IndexSelect(Tensor, Tensor, usize),+ IndexAdd(Tensor, Tensor, Tensor, usize),+ WhereCond(Tensor, Tensor, Tensor),++ #[allow(dead_code)]+ Conv1D {+ arg: Tensor,+ kernel: Tensor,+ padding: usize,+ stride: usize,+ dilation: usize,+ },++ #[allow(dead_code)]+ ConvTranspose1D {+ arg: Tensor,+ kernel: Tensor,+ padding: usize,+ output_padding: usize,+ stride: usize,+ dilation: usize,+ },++ #[allow(dead_code)]+ Conv2D {+ arg: Tensor,+ kernel: Tensor,+ padding: usize,+ stride: usize,+ dilation: usize,+ },++ #[allow(dead_code)]+ ConvTranspose2D {+ arg: Tensor,+ kernel: Tensor,+ padding: usize,+ output_padding: usize,+ stride: usize,+ dilation: usize,+ },++ AvgPool2D {+ arg: Tensor,+ kernel_size: (usize, usize),+ stride: (usize, usize),+ },++ MaxPool2D {+ arg: Tensor,+ kernel_size: (usize, usize),+ stride: (usize, usize),+ },++ UpsampleNearest1D {+ arg: Tensor,+ target_size: usize,+ },+ UpsampleNearest2D {+ arg: Tensor,+ target_h: usize,+ target_w: usize,+ },+ UpsampleBilinear2D {+ arg: Tensor,+ target_h: usize,+ target_w: usize,+ align_corners: bool,+ },++ Cat(Vec<Tensor>, usize),++ #[allow(dead_code)] // add is currently unused.+ Affine {+ arg: Tensor,+ mul: f64,+ add: f64,+ },+ ToDType(Tensor),+ Copy(Tensor),+ Broadcast(Tensor),+ Narrow(Tensor, usize, usize, usize),+ SliceScatter0(Tensor, Tensor, usize),+ Reshape(Tensor),+ ToDevice(Tensor),+ Transpose(Tensor, usize, usize),+ Permute(Tensor, Vec<usize>),+ Elu(Tensor, f64),+ Powf(Tensor, f64),+ CustomOp1(+ Tensor,+ std::sync::Arc<Box<dyn crate::CustomOp1 + Send + Sync>>,+ ),+ CustomOp2(+ Tensor,+ Tensor,+ std::sync::Arc<Box<dyn crate::CustomOp2 + Send + Sync>>,+ ),+ CustomOp3(+ Tensor,+ Tensor,+ Tensor,+ std::sync::Arc<Box<dyn crate::CustomOp3 + Send + Sync>>,+ ),+}++pub trait UnaryOpT {+ const NAME: &'static str;+ const KERNEL: &'static str;+ const V: Self;+ fn bf16(v1: bf16) -> bf16;+ fn f16(v1: f16) -> f16;+ fn f32(v1: f32) -> f32;+ fn f64(v1: f64) -> f64;+ fn u8(v1: u8) -> u8;+ fn u32(v1: u32) -> u32;+ fn i16(v1: i16) -> i16;+ fn i32(v1: i32) -> i32;+ fn i64(v1: i64) -> i64;+ fn f8e4m3(v1: f8e4m3) -> f8e4m3;++ #[inline(always)]+ fn bf16_vec(xs: &[bf16], ys: &mut [bf16]) {+ xs.iter().zip(ys).for_each(|(&x, y)| *y = Self::bf16(x))+ }+ #[inline(always)]+ fn f16_vec(xs: &[f16], ys: &mut [f16]) {+ xs.iter().zip(ys).for_each(|(&x, y)| *y = Self::f16(x))+ }+ #[inline(always)]+ fn f32_vec(xs: &[f32], ys: &mut [f32]) {+ xs.iter().zip(ys).for_each(|(&x, y)| *y = Self::f32(x))+ }+ #[inline(always)]+ fn f64_vec(xs: &[f64], ys: &mut [f64]) {+ xs.iter().zip(ys).for_each(|(&x, y)| *y = Self::f64(x))+ }+}++pub trait BinaryOpT {+ const NAME: &'static str;+ const KERNEL: &'static str;+ const V: Self;+ fn bf16(v1: bf16, v2: bf16) -> bf16;+ fn f16(v1: f16, v2: f16) -> f16;+ fn f32(v1: f32, v2: f32) -> f32;+ fn f64(v1: f64, v2: f64) -> f64;+ fn u8(v1: u8, v2: u8) -> u8;+ fn u32(v1: u32, v2: u32) -> u32;+ fn i16(v1: i16, v2: i16) -> i16;+ fn i32(v1: i32, v2: i32) -> i32;+ fn i64(v1: i64, v2: i64) -> i64;+ fn f8e4m3(v1: f8e4m3, v2: f8e4m3) -> f8e4m3;++ #[inline(always)]+ fn bf16_vec(xs1: &[bf16], xs2: &[bf16], ys: &mut [bf16]) {+ xs1.iter()+ .zip(xs2)+ .zip(ys)+ .for_each(|((&a, &b), y)| *y = Self::bf16(a, b))+ }+ #[inline(always)]+ fn f16_vec(xs1: &[f16], xs2: &[f16], ys: &mut [f16]) {+ xs1.iter()+ .zip(xs2)+ .zip(ys)+ .for_each(|((&a, &b), y)| *y = Self::f16(a, b))+ }+ #[inline(always)]+ fn f32_vec(xs1: &[f32], xs2: &[f32], ys: &mut [f32]) {+ xs1.iter()+ .zip(xs2)+ .zip(ys)+ .for_each(|((&a, &b), y)| *y = Self::f32(a, b))+ }+ #[inline(always)]+ fn f64_vec(xs1: &[f64], xs2: &[f64], ys: &mut [f64]) {+ xs1.iter()+ .zip(xs2)+ .zip(ys)+ .for_each(|((&a, &b), y)| *y = Self::f64(a, b))+ }+ #[inline(always)]+ fn u8_vec(xs1: &[u8], xs2: &[u8], ys: &mut [u8]) {+ xs1.iter()+ .zip(xs2)+ .zip(ys)+ .for_each(|((&a, &b), y)| *y = Self::u8(a, b))+ }+ #[inline(always)]+ fn u32_vec(xs1: &[u32], xs2: &[u32], ys: &mut [u32]) {+ xs1.iter()+ .zip(xs2)+ .zip(ys)+ .for_each(|((&a, &b), y)| *y = Self::u32(a, b))+ }+ #[inline(always)]+ fn i16_vec(xs1: &[i16], xs2: &[i16], ys: &mut [i16]) {+ xs1.iter()+ .zip(xs2)+ .zip(ys)+ .for_each(|((&a, &b), y)| *y = Self::i16(a, b))+ }+ #[inline(always)]+ fn i32_vec(xs1: &[i32], xs2: &[i32], ys: &mut [i32]) {+ xs1.iter()+ .zip(xs2)+ .zip(ys)+ .for_each(|((&a, &b), y)| *y = Self::i32(a, b))+ }+ #[inline(always)]+ fn i64_vec(xs1: &[i64], xs2: &[i64], ys: &mut [i64]) {+ xs1.iter()+ .zip(xs2)+ .zip(ys)+ .for_each(|((&a, &b), y)| *y = Self::i64(a, b))+ }++ // Scalar-broadcast variants: ys[i] = f(xs[i], scalar).+ // Used by binary_map_vec for the (1,0) inner-stride branch where one tensor+ // is contiguous and the other broadcasts a single value.+ #[inline(always)]+ fn bf16_scalar_vec(scalar: bf16, xs: &[bf16], ys: &mut [bf16]) {+ xs.iter()+ .zip(ys)+ .for_each(|(&x, y)| *y = Self::bf16(x, scalar))+ }+ #[inline(always)]+ fn f16_scalar_vec(scalar: f16, xs: &[f16], ys: &mut [f16]) {+ xs.iter()+ .zip(ys)+ .for_each(|(&x, y)| *y = Self::f16(x, scalar))+ }+ #[inline(always)]+ fn f32_scalar_vec(scalar: f32, xs: &[f32], ys: &mut [f32]) {+ xs.iter()+ .zip(ys)+ .for_each(|(&x, y)| *y = Self::f32(x, scalar))+ }+ #[inline(always)]+ fn f64_scalar_vec(scalar: f64, xs: &[f64], ys: &mut [f64]) {+ xs.iter()+ .zip(ys)+ .for_each(|(&x, y)| *y = Self::f64(x, scalar))+ }+ #[inline(always)]+ fn u8_scalar_vec(scalar: u8, xs: &[u8], ys: &mut [u8]) {+ xs.iter()+ .zip(ys)+ .for_each(|(&x, y)| *y = Self::u8(x, scalar))+ }+ #[inline(always)]+ fn u32_scalar_vec(scalar: u32, xs: &[u32], ys: &mut [u32]) {+ xs.iter()+ .zip(ys)+ .for_each(|(&x, y)| *y = Self::u32(x, scalar))+ }+ #[inline(always)]+ fn i16_scalar_vec(scalar: i16, xs: &[i16], ys: &mut [i16]) {+ xs.iter()+ .zip(ys)+ .for_each(|(&x, y)| *y = Self::i16(x, scalar))+ }+ #[inline(always)]+ fn i32_scalar_vec(scalar: i32, xs: &[i32], ys: &mut [i32]) {+ xs.iter()+ .zip(ys)+ .for_each(|(&x, y)| *y = Self::i32(x, scalar))+ }+ #[inline(always)]+ fn i64_scalar_vec(scalar: i64, xs: &[i64], ys: &mut [i64]) {+ xs.iter()+ .zip(ys)+ .for_each(|(&x, y)| *y = Self::i64(x, scalar))+ }+}++pub struct Add;+pub struct Div;+pub struct Mul;+pub struct Sub;+pub struct Maximum;+pub struct Minimum;+pub struct Exp;+pub struct Log;+pub struct Sin;+pub struct Cos;+pub struct Abs;+pub struct Neg;+pub struct Recip;+pub struct Sqr;+pub struct Sqrt;+pub struct Gelu;+pub struct GeluErf;+pub struct Erf;+pub struct Relu;+pub struct Silu;+pub struct Tanh;+pub struct Floor;+pub struct Ceil;+pub struct Round;+pub struct Sign;++// `$name` is an ident; stringify! derives the NAME string and the KERNEL prefix automatically.+// The optional `$vec_op` names a method on `crate::cpu::kernels::VecOps` that overrides+// f32_vec/f64_vec with an optimised implementation (MKL / Accelerate / SIMD).+macro_rules! bin_op {+ ($op:ident, $name:ident, $e:expr $(, $vec_op:ident, $scalar_op:ident)?) => {+ impl BinaryOpT for $op {+ const NAME: &'static str = stringify!($name);+ const KERNEL: &'static str = concat!("b", stringify!($name));+ const V: Self = $op;+ #[inline(always)]+ fn bf16(v1: bf16, v2: bf16) -> bf16 { $e(v1, v2) }+ #[inline(always)]+ fn f16(v1: f16, v2: f16) -> f16 { $e(v1, v2) }+ #[inline(always)]+ fn f32(v1: f32, v2: f32) -> f32 { $e(v1, v2) }+ #[inline(always)]+ fn f64(v1: f64, v2: f64) -> f64 { $e(v1, v2) }+ #[inline(always)]+ fn u8(v1: u8, v2: u8) -> u8 { $e(v1, v2) }+ #[inline(always)]+ fn u32(v1: u32, v2: u32) -> u32 { $e(v1, v2) }+ #[inline(always)]+ fn i16(v1: i16, v2: i16) -> i16 { $e(v1, v2) }+ #[inline(always)]+ fn i32(v1: i32, v2: i32) -> i32 { $e(v1, v2) }+ #[inline(always)]+ fn i64(v1: i64, v2: i64) -> i64 { $e(v1, v2) }+ #[inline(always)]+ fn f8e4m3(v1: f8e4m3, v2: f8e4m3) -> f8e4m3 { $e(v1, v2) }+ $(+ #[inline(always)]+ fn f32_vec(lhs: &[f32], rhs: &[f32], res: &mut [f32]) {+ <f32 as crate::cpu::kernels::VecOps>::$vec_op(lhs, rhs, res)+ }+ #[inline(always)]+ fn f64_vec(lhs: &[f64], rhs: &[f64], res: &mut [f64]) {+ <f64 as crate::cpu::kernels::VecOps>::$vec_op(lhs, rhs, res)+ }+ #[inline(always)]+ fn bf16_vec(lhs: &[bf16], rhs: &[bf16], res: &mut [bf16]) {+ <bf16 as crate::cpu::kernels::VecOps>::$vec_op(lhs, rhs, res)+ }+ #[inline(always)]+ fn f16_vec(lhs: &[f16], rhs: &[f16], res: &mut [f16]) {+ <f16 as crate::cpu::kernels::VecOps>::$vec_op(lhs, rhs, res)+ }+ #[inline(always)]+ fn bf16_scalar_vec(scalar: bf16, xs: &[bf16], ys: &mut [bf16]) {+ <bf16 as crate::cpu::kernels::VecOps>::$scalar_op(scalar, xs, ys)+ }+ #[inline(always)]+ fn f16_scalar_vec(scalar: f16, xs: &[f16], ys: &mut [f16]) {+ <f16 as crate::cpu::kernels::VecOps>::$scalar_op(scalar, xs, ys)+ }+ #[inline(always)]+ fn f32_scalar_vec(scalar: f32, xs: &[f32], ys: &mut [f32]) {+ <f32 as crate::cpu::kernels::VecOps>::$scalar_op(scalar, xs, ys)+ }+ #[inline(always)]+ fn f64_scalar_vec(scalar: f64, xs: &[f64], ys: &mut [f64]) {+ <f64 as crate::cpu::kernels::VecOps>::$scalar_op(scalar, xs, ys)+ }+ )?+ }+ };+}++bin_op!(Add, add, |v1, v2| v1 + v2, vec_add, scalar_add);+bin_op!(Sub, sub, |v1, v2| v1 - v2);+bin_op!(Mul, mul, |v1, v2| v1 * v2, vec_mul, scalar_mul);+bin_op!(Div, div, |v1, v2| v1 / v2);+bin_op!(Minimum, minimum, |v1, v2| if v1 > v2 { v2 } else { v1 });+bin_op!(Maximum, maximum, |v1, v2| if v1 < v2 { v2 } else { v1 });++#[allow(clippy::redundant_closure_call)]+macro_rules! unary_op {+ ($op: ident, $name: literal, $a: ident, $e: expr) => {+ impl UnaryOpT for $op {+ const NAME: &'static str = $name;+ const KERNEL: &'static str = concat!("u", $name);+ const V: Self = $op;+ #[inline(always)]+ fn bf16($a: bf16) -> bf16 {+ $e+ }+ #[inline(always)]+ fn f16($a: f16) -> f16 {+ $e+ }+ #[inline(always)]+ fn f32($a: f32) -> f32 {+ $e+ }+ #[inline(always)]+ fn f64($a: f64) -> f64 {+ $e+ }+ #[inline(always)]+ fn u8(_: u8) -> u8 {+ todo!("no unary function for u8")+ }+ #[inline(always)]+ fn u32(_: u32) -> u32 {+ todo!("no unary function for u32")+ }+ #[inline(always)]+ fn i16(_: i16) -> i16 {+ todo!("no unary function for i16")+ }+ #[inline(always)]+ fn i32(_: i32) -> i32 {+ todo!("no unary function for i32")+ }+ #[inline(always)]+ fn i64(_: i64) -> i64 {+ todo!("no unary function for i64")+ }+ #[inline(always)]+ fn f8e4m3($a: f8e4m3) -> f8e4m3 {+ $e+ }+ }+ };++ ($op: ident, $name: literal, $a: ident, $e: expr, $f32_vec:ident, $f64_vec:ident) => {+ impl UnaryOpT for $op {+ const NAME: &'static str = $name;+ const KERNEL: &'static str = concat!("u", $name);+ const V: Self = $op;+ #[inline(always)]+ fn bf16($a: bf16) -> bf16 {+ $e+ }+ #[inline(always)]+ fn f16($a: f16) -> f16 {+ $e+ }+ #[inline(always)]+ fn f32($a: f32) -> f32 {+ $e+ }+ #[inline(always)]+ fn f64($a: f64) -> f64 {+ $e+ }+ #[inline(always)]+ fn u8(_: u8) -> u8 {+ todo!("no unary function for u8")+ }+ #[inline(always)]+ fn u32(_: u32) -> u32 {+ todo!("no unary function for u32")+ }+ #[inline(always)]+ fn i16(_: i16) -> i16 {+ todo!("no unary function for i16")+ }+ #[inline(always)]+ fn i32(_: i32) -> i32 {+ todo!("no unary function for i32")+ }+ #[inline(always)]+ fn i64(_: i64) -> i64 {+ todo!("no unary function for i64")+ }+ #[inline(always)]+ fn f8e4m3($a: f8e4m3) -> f8e4m3 {+ $e+ }++ #[cfg(feature = "mkl")]+ #[inline(always)]+ fn f32_vec(xs: &[f32], ys: &mut [f32]) {+ crate::mkl::$f32_vec(xs, ys)+ }+ #[cfg(feature = "mkl")]+ #[inline(always)]+ fn f64_vec(xs: &[f64], ys: &mut [f64]) {+ crate::mkl::$f64_vec(xs, ys)+ }++ #[cfg(feature = "accelerate")]+ #[inline(always)]+ fn f32_vec(xs: &[f32], ys: &mut [f32]) {+ crate::accelerate::$f32_vec(xs, ys)+ }+ #[cfg(feature = "accelerate")]+ #[inline(always)]+ fn f64_vec(xs: &[f64], ys: &mut [f64]) {+ crate::accelerate::$f64_vec(xs, ys)+ }+ }+ };+}++unary_op!(Exp, "exp", v, v.exp(), vs_exp, vd_exp);+unary_op!(Log, "log", v, v.ln(), vs_ln, vd_ln);+unary_op!(Sin, "sin", v, v.sin(), vs_sin, vd_sin);+unary_op!(Cos, "cos", v, v.cos(), vs_cos, vd_cos);+unary_op!(Tanh, "tanh", v, v.tanh(), vs_tanh, vd_tanh);+unary_op!(Neg, "neg", v, -v);+unary_op!(Recip, "recip", v, v.recip());+unary_op!(Sqr, "sqr", v, v * v, vs_sqr, vd_sqr);+unary_op!(Sqrt, "sqrt", v, v.sqrt(), vs_sqrt, vd_sqrt);++// Hardcode the value for sqrt(2/pi)+// https://github.com/huggingface/candle/issues/1982+#[allow(clippy::excessive_precision)]+const SQRT_TWO_OVER_PI_F32: f32 = 0.79788456080286535587989211986876373;+#[allow(clippy::excessive_precision)]+const SQRT_TWO_OVER_PI_F64: f64 = 0.79788456080286535587989211986876373;++/// Tanh based approximation of the `gelu` operation+/// GeluErf is the more precise one.+/// <https://en.wikipedia.org/wiki/Activation_function#Comparison_of_activation_functions>+impl UnaryOpT for Gelu {+ const NAME: &'static str = "gelu";+ const V: Self = Gelu;+ #[inline(always)]+ fn bf16(v: bf16) -> bf16 {+ bf16::from_f32_const(0.5)+ * v+ * (bf16::ONE+ + bf16::tanh(+ bf16::from_f32_const(SQRT_TWO_OVER_PI_F32)+ * v+ * (bf16::ONE + bf16::from_f32_const(0.044715) * v * v),+ ))+ }+ #[inline(always)]+ fn f16(v: f16) -> f16 {+ f16::from_f32_const(0.5)+ * v+ * (f16::ONE+ + f16::tanh(+ f16::from_f32_const(SQRT_TWO_OVER_PI_F32)+ * v+ * (f16::ONE + f16::from_f32_const(0.044715) * v * v),+ ))+ }+ #[inline(always)]+ fn f32(v: f32) -> f32 {+ 0.5 * v * (1.0 + f32::tanh(SQRT_TWO_OVER_PI_F32 * v * (1.0 + 0.044715 * v * v)))+ }+ #[inline(always)]+ fn f64(v: f64) -> f64 {+ 0.5 * v * (1.0 + f64::tanh(SQRT_TWO_OVER_PI_F64 * v * (1.0 + 0.044715 * v * v)))+ }+ #[inline(always)]+ fn u8(_: u8) -> u8 {+ 0+ }+ #[inline(always)]+ fn u32(_: u32) -> u32 {+ 0+ }+ #[inline(always)]+ fn i16(_: i16) -> i16 {+ 0+ }+ #[inline(always)]+ fn i32(_: i32) -> i32 {+ 0+ }+ #[inline(always)]+ fn i64(_: i64) -> i64 {+ 0+ }+ #[inline(always)]+ fn f8e4m3(v: f8e4m3) -> f8e4m3 {+ f8e4m3::from_f32(0.5)+ * v+ * (f8e4m3::ONE+ + f8e4m3::tanh(+ f8e4m3::from_f32(SQRT_TWO_OVER_PI_F32)+ * v+ * (f8e4m3::ONE + f8e4m3::from_f32(0.044715) * v * v),+ ))+ }+ const KERNEL: &'static str = "ugelu";++ #[cfg(feature = "mkl")]+ #[inline(always)]+ fn f32_vec(xs: &[f32], ys: &mut [f32]) {+ crate::mkl::vs_gelu(xs, ys)+ }++ #[cfg(feature = "mkl")]+ #[inline(always)]+ fn f64_vec(xs: &[f64], ys: &mut [f64]) {+ crate::mkl::vd_gelu(xs, ys)+ }++ #[cfg(feature = "accelerate")]+ #[inline(always)]+ fn f32_vec(xs: &[f32], ys: &mut [f32]) {+ crate::accelerate::vs_gelu(xs, ys)+ }++ #[cfg(feature = "accelerate")]+ #[inline(always)]+ fn f64_vec(xs: &[f64], ys: &mut [f64]) {+ crate::accelerate::vd_gelu(xs, ys)+ }+}++/// `erf` operation+/// <https://en.wikipedia.org/wiki/Error_function>+impl UnaryOpT for Erf {+ const NAME: &'static str = "erf";+ const KERNEL: &'static str = "uerf";+ const V: Self = Erf;+ #[inline(always)]+ fn bf16(v: bf16) -> bf16 {+ bf16::from_f64(Self::f64(v.to_f64()))+ }+ #[inline(always)]+ fn f16(v: f16) -> f16 {+ f16::from_f64(Self::f64(v.to_f64()))+ }+ #[inline(always)]+ fn f32(v: f32) -> f32 {+ crate::cpu::erf::erf_f32(v)+ }+ #[inline(always)]+ fn f64(v: f64) -> f64 {+ crate::cpu::erf::erf_f64(v)+ }+ #[inline(always)]+ fn u8(_: u8) -> u8 {+ 0+ }+ #[inline(always)]+ fn u32(_: u32) -> u32 {+ 0+ }+ #[inline(always)]+ fn i16(_: i16) -> i16 {+ 0+ }+ #[inline(always)]+ fn i32(_: i32) -> i32 {+ 0+ }+ #[inline(always)]+ fn i64(_: i64) -> i64 {+ 0+ }+ #[inline(always)]+ fn f8e4m3(v: f8e4m3) -> f8e4m3 {+ f8e4m3::from_f64(Self::f64(v.to_f64()))+ }+}++/// Silu operation+impl UnaryOpT for Silu {+ const NAME: &'static str = "silu";+ const V: Self = Silu;+ #[inline(always)]+ fn bf16(v: bf16) -> bf16 {+ v / (bf16::ONE + (-v).exp())+ }+ #[inline(always)]+ fn f16(v: f16) -> f16 {+ v / (f16::ONE + (-v).exp())+ }+ #[inline(always)]+ fn f32(v: f32) -> f32 {+ v / (1.0 + (-v).exp())+ }+ #[inline(always)]+ fn f64(v: f64) -> f64 {+ v / (1.0 + (-v).exp())+ }+ #[inline(always)]+ fn u8(_: u8) -> u8 {+ 0+ }+ #[inline(always)]+ fn u32(_: u32) -> u32 {+ 0+ }+ #[inline(always)]+ fn i16(_: i16) -> i16 {+ 0+ }+ #[inline(always)]+ fn i32(_: i32) -> i32 {+ 0+ }+ #[inline(always)]+ fn i64(_: i64) -> i64 {+ 0+ }+ #[inline(always)]+ fn f8e4m3(v: f8e4m3) -> f8e4m3 {+ v / (f8e4m3::ONE + (-v).exp())+ }+ const KERNEL: &'static str = "usilu";++ #[cfg(feature = "mkl")]+ #[inline(always)]+ fn f32_vec(xs: &[f32], ys: &mut [f32]) {+ crate::mkl::vs_silu(xs, ys)+ }++ #[cfg(feature = "mkl")]+ #[inline(always)]+ fn f64_vec(xs: &[f64], ys: &mut [f64]) {+ crate::mkl::vd_silu(xs, ys)+ }++ #[cfg(feature = "accelerate")]+ #[inline(always)]+ fn f32_vec(xs: &[f32], ys: &mut [f32]) {+ crate::accelerate::vs_silu(xs, ys)+ }++ #[cfg(feature = "accelerate")]+ #[inline(always)]+ fn f64_vec(xs: &[f64], ys: &mut [f64]) {+ crate::accelerate::vd_silu(xs, ys)+ }+}++impl UnaryOpT for Abs {+ const NAME: &'static str = "abs";+ const KERNEL: &'static str = "uabs";+ const V: Self = Abs;+ #[inline(always)]+ fn bf16(v: bf16) -> bf16 {+ v.abs()+ }+ #[inline(always)]+ fn f16(v: f16) -> f16 {+ v.abs()+ }+ #[inline(always)]+ fn f32(v: f32) -> f32 {+ v.abs()+ }+ #[inline(always)]+ fn f64(v: f64) -> f64 {+ v.abs()+ }+ #[inline(always)]+ fn u8(v: u8) -> u8 {+ v+ }+ #[inline(always)]+ fn u32(v: u32) -> u32 {+ v+ }+ #[inline(always)]+ fn i16(v: i16) -> i16 {+ v.abs()+ }+ #[inline(always)]+ fn i32(v: i32) -> i32 {+ v.abs()+ }+ #[inline(always)]+ fn i64(v: i64) -> i64 {+ v.abs()+ }+ #[inline(always)]+ fn f8e4m3(v: f8e4m3) -> f8e4m3 {+ v.abs()+ }+}++impl UnaryOpT for Ceil {+ const NAME: &'static str = "ceil";+ const KERNEL: &'static str = "uceil";+ const V: Self = Ceil;+ #[inline(always)]+ fn bf16(v: bf16) -> bf16 {+ v.ceil()+ }+ #[inline(always)]+ fn f16(v: f16) -> f16 {+ v.ceil()+ }+ #[inline(always)]+ fn f32(v: f32) -> f32 {+ v.ceil()+ }+ #[inline(always)]+ fn f64(v: f64) -> f64 {+ v.ceil()+ }+ #[inline(always)]+ fn u8(v: u8) -> u8 {+ v+ }+ #[inline(always)]+ fn u32(v: u32) -> u32 {+ v+ }+ #[inline(always)]+ fn i16(v: i16) -> i16 {+ v+ }+ #[inline(always)]+ fn i32(v: i32) -> i32 {+ v+ }+ #[inline(always)]+ fn i64(v: i64) -> i64 {+ v+ }+ #[inline(always)]+ fn f8e4m3(v: f8e4m3) -> f8e4m3 {+ v.ceil()+ }+}++impl UnaryOpT for Floor {+ const NAME: &'static str = "floor";+ const KERNEL: &'static str = "ufloor";+ const V: Self = Floor;+ #[inline(always)]+ fn bf16(v: bf16) -> bf16 {+ v.floor()+ }+ #[inline(always)]+ fn f16(v: f16) -> f16 {+ v.floor()+ }+ #[inline(always)]+ fn f32(v: f32) -> f32 {+ v.floor()+ }+ #[inline(always)]+ fn f64(v: f64) -> f64 {+ v.floor()+ }+ #[inline(always)]+ fn u8(v: u8) -> u8 {+ v+ }+ #[inline(always)]+ fn u32(v: u32) -> u32 {+ v+ }+ #[inline(always)]+ fn i16(v: i16) -> i16 {+ v+ }+ #[inline(always)]+ fn i32(v: i32) -> i32 {+ v+ }+ #[inline(always)]+ fn i64(v: i64) -> i64 {+ v+ }+ #[inline(always)]+ fn f8e4m3(v: f8e4m3) -> f8e4m3 {+ v.floor()+ }+}++impl UnaryOpT for Round {+ const NAME: &'static str = "round";+ const KERNEL: &'static str = "uround";+ const V: Self = Round;+ #[inline(always)]+ fn bf16(v: bf16) -> bf16 {+ v.round()+ }+ #[inline(always)]+ fn f16(v: f16) -> f16 {+ v.round()+ }+ #[inline(always)]+ fn f32(v: f32) -> f32 {+ v.round()+ }+ #[inline(always)]+ fn f64(v: f64) -> f64 {+ v.round()+ }+ #[inline(always)]+ fn u8(v: u8) -> u8 {+ v+ }+ #[inline(always)]+ fn u32(v: u32) -> u32 {+ v+ }+ #[inline(always)]+ fn i16(v: i16) -> i16 {+ v+ }+ #[inline(always)]+ fn i32(v: i32) -> i32 {+ v+ }+ #[inline(always)]+ fn i64(v: i64) -> i64 {+ v+ }+ #[inline(always)]+ fn f8e4m3(v: f8e4m3) -> f8e4m3 {+ v.round()+ }+}++impl UnaryOpT for GeluErf {+ const NAME: &'static str = "gelu_erf";+ const KERNEL: &'static str = "ugelu_erf";+ const V: Self = GeluErf;+ #[inline(always)]+ fn bf16(v: bf16) -> bf16 {+ bf16::from_f64(Self::f64(v.to_f64()))+ }+ #[inline(always)]+ fn f16(v: f16) -> f16 {+ f16::from_f64(Self::f64(v.to_f64()))+ }+ #[inline(always)]+ fn f32(v: f32) -> f32 {+ (crate::cpu::erf::erf_f32(v * std::f32::consts::FRAC_1_SQRT_2) + 1.) * 0.5 * v+ }+ #[inline(always)]+ fn f64(v: f64) -> f64 {+ (crate::cpu::erf::erf_f64(v * std::f64::consts::FRAC_1_SQRT_2) + 1.) * 0.5 * v+ }+ #[inline(always)]+ fn u8(_: u8) -> u8 {+ 0+ }+ #[inline(always)]+ fn u32(_: u32) -> u32 {+ 0+ }+ #[inline(always)]+ fn i16(_: i16) -> i16 {+ 0+ }+ #[inline(always)]+ fn i32(_: i32) -> i32 {+ 0+ }+ #[inline(always)]+ fn i64(_: i64) -> i64 {+ 0+ }+ #[inline(always)]+ fn f8e4m3(v: f8e4m3) -> f8e4m3 {+ f8e4m3::from_f32(Self::f32(v.to_f32()))+ }+}++impl UnaryOpT for Relu {+ const NAME: &'static str = "relu";+ const KERNEL: &'static str = "urelu";+ const V: Self = Relu;+ #[inline(always)]+ fn bf16(v: bf16) -> bf16 {+ v.max(bf16::ZERO)+ }+ #[inline(always)]+ fn f16(v: f16) -> f16 {+ v.max(f16::ZERO)+ }+ #[inline(always)]+ fn f32(v: f32) -> f32 {+ v.max(0f32)+ }+ #[inline(always)]+ fn f64(v: f64) -> f64 {+ v.max(0f64)+ }+ #[inline(always)]+ fn u8(v: u8) -> u8 {+ v+ }+ #[inline(always)]+ fn u32(v: u32) -> u32 {+ v+ }+ #[inline(always)]+ fn i16(v: i16) -> i16 {+ v.max(0)+ }+ #[inline(always)]+ fn i32(v: i32) -> i32 {+ v.max(0)+ }+ #[inline(always)]+ fn i64(v: i64) -> i64 {+ v.max(0)+ }+ #[inline(always)]+ fn f8e4m3(v: f8e4m3) -> f8e4m3 {+ v.max(f8e4m3::ZERO)+ }+}++/// `BackpropOp` is a wrapper around `Option<Op>`. The main goal is to ensure that dependencies are+/// properly checked when creating a new value+#[derive(Clone)]+pub struct BackpropOp(Option<Op>);++impl BackpropOp {+ pub fn none() -> Self {+ BackpropOp(None)+ }++ pub(crate) fn new1(arg: &Tensor, f: impl Fn(Tensor) -> Op) -> Self {+ let op = if arg.track_op() {+ Some(f(arg.clone()))+ } else {+ None+ };+ Self(op)+ }++ pub(crate) fn new2(arg1: &Tensor, arg2: &Tensor, f: impl Fn(Tensor, Tensor) -> Op) -> Self {+ let op = if arg1.track_op() || arg2.track_op() {+ Some(f(arg1.clone(), arg2.clone()))+ } else {+ None+ };+ Self(op)+ }++ pub(crate) fn new3(+ arg1: &Tensor,+ arg2: &Tensor,+ arg3: &Tensor,+ f: impl Fn(Tensor, Tensor, Tensor) -> Op,+ ) -> Self {+ let op = if arg1.track_op() || arg2.track_op() || arg3.track_op() {+ Some(f(arg1.clone(), arg2.clone(), arg3.clone()))+ } else {+ None+ };+ Self(op)+ }++ pub(crate) fn new<A: AsRef<Tensor>>(args: &[A], f: impl Fn(Vec<Tensor>) -> Op) -> Self {+ let op = if args.iter().any(|arg| arg.as_ref().track_op()) {+ let args: Vec<Tensor> = args.iter().map(|arg| arg.as_ref().clone()).collect();+ Some(f(args))+ } else {+ None+ };+ Self(op)+ }++ pub(crate) fn is_none(&self) -> bool {+ self.0.is_none()+ }+}++impl std::ops::Deref for BackpropOp {+ type Target = Option<Op>;+ fn deref(&self) -> &Self::Target {+ &self.0+ }+}++impl UnaryOpT for Sign {+ const NAME: &'static str = "sign";+ const KERNEL: &'static str = "usign";+ const V: Self = Sign;+ #[inline(always)]+ fn bf16(v: bf16) -> bf16 {+ bf16::from((v > bf16::ZERO) as i8) - bf16::from((v < bf16::ZERO) as i8)+ }+ #[inline(always)]+ fn f16(v: f16) -> f16 {+ f16::from((v > f16::ZERO) as i8) - f16::from((v < f16::ZERO) as i8)+ }+ #[inline(always)]+ fn f32(v: f32) -> f32 {+ f32::from(v > 0.) - f32::from(v < 0.)+ }+ #[inline(always)]+ fn f64(v: f64) -> f64 {+ f64::from(v > 0.) - f64::from(v < 0.)+ }+ #[inline(always)]+ fn u8(v: u8) -> u8 {+ u8::min(1, v)+ }+ #[inline(always)]+ fn u32(v: u32) -> u32 {+ u32::min(1, v)+ }+ #[inline(always)]+ fn i16(v: i16) -> i16 {+ (v > 0) as i16 - (v < 0) as i16+ }+ #[inline(always)]+ fn i32(v: i32) -> i32 {+ (v > 0) as i32 - (v < 0) as i32+ }+ #[inline(always)]+ fn i64(v: i64) -> i64 {+ (v > 0) as i64 - (v < 0) as i64+ }+ #[inline(always)]+ fn f8e4m3(v: f8e4m3) -> f8e4m3 {+ if v > f8e4m3::ZERO {+ f8e4m3::ONE+ } else if v < f8e4m3::ZERO {+ -f8e4m3::ONE+ } else {+ f8e4m3::ZERO+ }+ }+}diff --git a/vendor/candle-core/src/pickle.rs b/vendor/candle-core/src/pickle.rsnew file mode 100644index 0000000..dd65b9d--- /dev/null+++ b/vendor/candle-core/src/pickle.rs@@ -0,0 +1,841 @@+//! Just enough pickle support to be able to read PyTorch checkpoints.+// This hardcodes objects that are required for tensor reading, we may want to make this a bit more+// composable/tensor agnostic at some point.+use crate::{Context, DType, Error as E, Layout, Result, Tensor};+use byteorder::{LittleEndian, ReadBytesExt};+use std::collections::HashMap;+use std::io::BufRead;++const VERBOSE: bool = false;++// https://docs.juliahub.com/Pickle/LAUNc/0.1.0/opcode/+#[repr(u8)]+#[derive(Debug, Eq, PartialEq, Clone)]+pub enum OpCode {+ // https://github.com/python/cpython/blob/ed25f097160b5cbb0c9a1f9a746d2f1bbc96515a/Lib/pickletools.py#L2123+ Proto = 0x80,+ Global = b'c',+ BinPut = b'q',+ LongBinPut = b'r',+ EmptyTuple = b')',+ Reduce = b'R',+ Mark = b'(',+ BinUnicode = b'X',+ BinInt = b'J',+ Tuple = b't',+ BinPersId = b'Q',+ BinInt1 = b'K',+ BinInt2 = b'M',+ Tuple1 = 0x85,+ Tuple2 = 0x86,+ Tuple3 = 0x87,+ NewTrue = 0x88,+ NewFalse = 0x89,+ None = b'N',+ BinGet = b'h',+ LongBinGet = b'j',+ SetItem = b's',+ SetItems = b'u',+ EmptyDict = b'}',+ Dict = b'd',+ Build = b'b',+ Stop = b'.',+ NewObj = 0x81,+ EmptyList = b']',+ BinFloat = b'G',+ Append = b'a',+ Appends = b'e',+ Long1 = 0x8a,+}++// Avoid using FromPrimitive so as not to drag another dependency.+impl TryFrom<u8> for OpCode {+ type Error = u8;+ fn try_from(value: u8) -> std::result::Result<Self, Self::Error> {+ match value {+ 0x80 => Ok(Self::Proto),+ b'c' => Ok(Self::Global),+ b'q' => Ok(Self::BinPut),+ b'r' => Ok(Self::LongBinPut),+ b')' => Ok(Self::EmptyTuple),+ b'R' => Ok(Self::Reduce),+ b'(' => Ok(Self::Mark),+ b'X' => Ok(Self::BinUnicode),+ b'J' => Ok(Self::BinInt),+ b't' => Ok(Self::Tuple),+ b'Q' => Ok(Self::BinPersId),+ b'K' => Ok(Self::BinInt1),+ b'M' => Ok(Self::BinInt2),+ b'N' => Ok(Self::None),+ 0x85 => Ok(Self::Tuple1),+ 0x86 => Ok(Self::Tuple2),+ 0x87 => Ok(Self::Tuple3),+ 0x88 => Ok(Self::NewTrue),+ 0x89 => Ok(Self::NewFalse),+ b'h' => Ok(Self::BinGet),+ b'j' => Ok(Self::LongBinGet),+ b's' => Ok(Self::SetItem),+ b'u' => Ok(Self::SetItems),+ b'}' => Ok(Self::EmptyDict),+ b'd' => Ok(Self::EmptyDict),+ b'b' => Ok(Self::Build),+ b'.' => Ok(Self::Stop),+ 0x81 => Ok(Self::NewObj),+ b']' => Ok(Self::EmptyList),+ b'G' => Ok(Self::BinFloat),+ b'a' => Ok(Self::Append),+ b'e' => Ok(Self::Appends),+ 0x8a => Ok(Self::Long1),+ value => Err(value),+ }+ }+}++fn read_to_newline<R: BufRead>(r: &mut R) -> Result<Vec<u8>> {+ let mut data: Vec<u8> = Vec::with_capacity(32);+ r.read_until(b'\n', &mut data)?;+ data.pop();+ if data.last() == Some(&b'\r') {+ data.pop();+ }+ Ok(data)+}++#[derive(Debug, Clone, PartialEq)]+pub enum Object {+ Class {+ module_name: String,+ class_name: String,+ },+ Int(i32),+ Long(i64),+ Float(f64),+ Unicode(String),+ Bool(bool),+ None,+ Tuple(Vec<Object>),+ List(Vec<Object>),+ Mark,+ Dict(Vec<(Object, Object)>),+ Reduce {+ callable: Box<Object>,+ args: Box<Object>,+ },+ Build {+ callable: Box<Object>,+ args: Box<Object>,+ },+ PersistentLoad(Box<Object>),+}++type OResult<T> = std::result::Result<T, Object>;++impl Object {+ pub fn unicode(self) -> OResult<String> {+ match self {+ Self::Unicode(t) => Ok(t),+ _ => Err(self),+ }+ }++ pub fn reduce(self) -> OResult<(Self, Self)> {+ match self {+ Self::Reduce { callable, args } => Ok((*callable, *args)),+ _ => Err(self),+ }+ }++ pub fn none(self) -> OResult<()> {+ match self {+ Self::None => Ok(()),+ _ => Err(self),+ }+ }++ pub fn persistent_load(self) -> OResult<Self> {+ match self {+ Self::PersistentLoad(t) => Ok(*t),+ _ => Err(self),+ }+ }++ pub fn bool(self) -> OResult<bool> {+ match self {+ Self::Bool(t) => Ok(t),+ _ => Err(self),+ }+ }++ pub fn int(self) -> OResult<i32> {+ match self {+ Self::Int(t) => Ok(t),+ _ => Err(self),+ }+ }++ pub fn int_or_long(self) -> OResult<i64> {+ match self {+ Self::Int(t) => Ok(t as i64),+ Self::Long(t) => Ok(t),+ _ => Err(self),+ }+ }++ pub fn tuple(self) -> OResult<Vec<Self>> {+ match self {+ Self::Tuple(t) => Ok(t),+ _ => Err(self),+ }+ }++ pub fn dict(self) -> OResult<Vec<(Self, Self)>> {+ match self {+ Self::Dict(t) => Ok(t),+ _ => Err(self),+ }+ }++ pub fn class(self) -> OResult<(String, String)> {+ match self {+ Self::Class {+ module_name,+ class_name,+ } => Ok((module_name, class_name)),+ _ => Err(self),+ }+ }++ pub fn into_tensor_info(+ self,+ name: Self,+ dir_name: &std::path::Path,+ ) -> Result<Option<TensorInfo>> {+ let name = match name.unicode() {+ Ok(name) => name,+ Err(_) => return Ok(None),+ };+ let (callable, args) = match self.reduce() {+ Ok(callable_args) => callable_args,+ _ => return Ok(None),+ };+ let (callable, args) = match callable {+ Object::Class {+ module_name,+ class_name,+ } if module_name == "torch._tensor" && class_name == "_rebuild_from_type_v2" => {+ let mut args = args.tuple()?;+ let callable = args.remove(0);+ let args = args.remove(1);+ (callable, args)+ }+ Object::Class {+ module_name,+ class_name,+ } if module_name == "torch._utils" && class_name == "_rebuild_parameter" => {+ let mut args = args.tuple()?;+ args.remove(0).reduce()?+ }+ _ => (callable, args),+ };+ match callable {+ Object::Class {+ module_name,+ class_name,+ } if module_name == "torch._utils" && class_name == "_rebuild_tensor_v2" => {}+ _ => return Ok(None),+ };+ let (layout, dtype, file_path, storage_size) = rebuild_args(args)?;+ Ok(Some(TensorInfo {+ name,+ dtype,+ layout,+ path: format!("{}/{}", dir_name.to_string_lossy(), file_path),+ storage_size,+ }))+ }+}++impl TryFrom<Object> for String {+ type Error = Object;+ fn try_from(value: Object) -> std::result::Result<Self, Self::Error> {+ match value {+ Object::Unicode(s) => Ok(s),+ other => Err(other),+ }+ }+}++impl TryFrom<Object> for usize {+ type Error = Object;+ fn try_from(value: Object) -> std::result::Result<Self, Self::Error> {+ match value {+ Object::Int(s) if s >= 0 => Ok(s as usize),+ other => Err(other),+ }+ }+}++impl<T: TryFrom<Object, Error = Object>> TryFrom<Object> for Vec<T> {+ type Error = Object;+ fn try_from(value: Object) -> std::result::Result<Self, Self::Error> {+ match value {+ Object::Tuple(values) => {+ // This does not return the appropriate value in the error case but instead return+ // the object related to the first error.+ values+ .into_iter()+ .map(|v| T::try_from(v))+ .collect::<std::result::Result<Vec<T>, Self::Error>>()+ }+ other => Err(other),+ }+ }+}++#[derive(Debug)]+pub struct Stack {+ stack: Vec<Object>,+ memo: HashMap<u32, Object>,+}++impl Stack {+ pub fn empty() -> Self {+ Self {+ stack: Vec::with_capacity(512),+ memo: HashMap::new(),+ }+ }++ pub fn stack(&self) -> &[Object] {+ self.stack.as_slice()+ }++ pub fn read_loop<R: BufRead>(&mut self, r: &mut R) -> Result<()> {+ loop {+ if self.read(r)? {+ break;+ }+ }+ Ok(())+ }++ pub fn finalize(mut self) -> Result<Object> {+ self.pop()+ }++ fn push(&mut self, obj: Object) {+ self.stack.push(obj)+ }++ fn pop(&mut self) -> Result<Object> {+ match self.stack.pop() {+ None => crate::bail!("unexpected empty stack"),+ Some(obj) => Ok(obj),+ }+ }++ // https://docs.juliahub.com/Pickle/LAUNc/0.1.0/opcode/#Pickle.OpCodes.BUILD+ fn build(&mut self) -> Result<()> {+ let args = self.pop()?;+ let obj = self.pop()?;+ let obj = match (obj, args) {+ (Object::Dict(mut obj), Object::Dict(mut args)) => {+ obj.append(&mut args);+ Object::Dict(obj)+ }+ (obj, args) => Object::Build {+ callable: Box::new(obj),+ args: Box::new(args),+ },+ };+ self.push(obj);+ Ok(())+ }++ fn reduce(&mut self) -> Result<()> {+ let args = self.pop()?;+ let callable = self.pop()?;+ #[allow(clippy::single_match)]+ let reduced = match &callable {+ Object::Class {+ module_name,+ class_name,+ } => {+ if module_name == "collections"+ && (class_name == "OrderedDict" || class_name == "defaultdict")+ {+ // TODO: have a separate ordered dict and a separate default dict.+ Some(Object::Dict(vec![]))+ } else {+ None+ }+ }+ _ => None,+ };+ let reduced = reduced.unwrap_or_else(|| Object::Reduce {+ callable: Box::new(callable),+ args: Box::new(args),+ });+ self.push(reduced);+ Ok(())+ }++ fn last(&mut self) -> Result<&mut Object> {+ match self.stack.last_mut() {+ None => crate::bail!("unexpected empty stack"),+ Some(obj) => Ok(obj),+ }+ }++ fn memo_get(&self, id: u32) -> Result<Object> {+ match self.memo.get(&id) {+ None => crate::bail!("missing object in memo {id}"),+ Some(obj) => {+ // Maybe we should use refcounting rather than doing potential large clones here.+ Ok(obj.clone())+ }+ }+ }++ fn memo_put(&mut self, id: u32) -> Result<()> {+ let obj = self.last()?.clone();+ self.memo.insert(id, obj);+ Ok(())+ }++ fn persistent_load(&self, id: Object) -> Result<Object> {+ Ok(Object::PersistentLoad(Box::new(id)))+ }++ fn new_obj(&self, class: Object, args: Object) -> Result<Object> {+ Ok(Object::Reduce {+ callable: Box::new(class),+ args: Box::new(args),+ })+ }++ fn pop_to_marker(&mut self) -> Result<Vec<Object>> {+ let mut mark_idx = None;+ for (idx, obj) in self.stack.iter().enumerate().rev() {+ if obj == &Object::Mark {+ mark_idx = Some(idx);+ break;+ }+ }+ match mark_idx {+ Some(mark_idx) => {+ let objs = self.stack.split_off(mark_idx + 1);+ self.stack.pop();+ Ok(objs)+ }+ None => {+ crate::bail!("marker object not found")+ }+ }+ }++ pub fn read<R: BufRead>(&mut self, r: &mut R) -> Result<bool> {+ let op_code = match OpCode::try_from(r.read_u8()?) {+ Ok(op_code) => op_code,+ Err(op_code) => {+ crate::bail!("unknown op-code {op_code}")+ }+ };+ // println!("op: {op_code:?}");+ // println!("{:?}", self.stack);+ match op_code {+ OpCode::Proto => {+ let version = r.read_u8()?;+ if VERBOSE {+ println!("proto {version}");+ }+ }+ OpCode::Global => {+ let module_name = read_to_newline(r)?;+ let class_name = read_to_newline(r)?;+ let module_name = String::from_utf8_lossy(&module_name).to_string();+ let class_name = String::from_utf8_lossy(&class_name).to_string();+ self.push(Object::Class {+ module_name,+ class_name,+ })+ }+ OpCode::BinInt1 => {+ let arg = r.read_u8()?;+ self.push(Object::Int(arg as i32))+ }+ OpCode::BinInt2 => {+ let arg = r.read_u16::<LittleEndian>()?;+ self.push(Object::Int(arg as i32))+ }+ OpCode::BinInt => {+ let arg = r.read_i32::<LittleEndian>()?;+ self.push(Object::Int(arg))+ }+ OpCode::BinFloat => {+ // Somehow floats are encoded using BigEndian whereas int types use LittleEndian.+ // https://github.com/python/cpython/blob/0c80da4c14d904a367968955544dd6ae58c8101c/Lib/pickletools.py#L855+ // https://github.com/pytorch/pytorch/blob/372d078f361e726bb4ac0884ac334b04c58179ef/torch/_weights_only_unpickler.py#L243+ let arg = r.read_f64::<byteorder::BigEndian>()?;+ self.push(Object::Float(arg))+ }+ OpCode::BinUnicode => {+ let len = r.read_u32::<LittleEndian>()?;+ let mut data = vec![0u8; len as usize];+ r.read_exact(&mut data)?;+ let data = String::from_utf8(data).map_err(E::wrap)?;+ self.push(Object::Unicode(data))+ }+ OpCode::BinPersId => {+ let id = self.pop()?;+ let obj = self.persistent_load(id)?;+ self.push(obj)+ }+ OpCode::Tuple => {+ let objs = self.pop_to_marker()?;+ self.push(Object::Tuple(objs))+ }+ OpCode::Tuple1 => {+ let obj = self.pop()?;+ self.push(Object::Tuple(vec![obj]))+ }+ OpCode::Tuple2 => {+ let obj2 = self.pop()?;+ let obj1 = self.pop()?;+ self.push(Object::Tuple(vec![obj1, obj2]))+ }+ OpCode::Tuple3 => {+ let obj3 = self.pop()?;+ let obj2 = self.pop()?;+ let obj1 = self.pop()?;+ self.push(Object::Tuple(vec![obj1, obj2, obj3]))+ }+ OpCode::NewTrue => self.push(Object::Bool(true)),+ OpCode::NewFalse => self.push(Object::Bool(false)),+ OpCode::Append => {+ let value = self.pop()?;+ let pylist = self.last()?;+ if let Object::List(d) = pylist {+ d.push(value)+ } else {+ crate::bail!("expected a list, got {pylist:?}")+ }+ }+ OpCode::Appends => {+ let objs = self.pop_to_marker()?;+ let pylist = self.last()?;+ if let Object::List(d) = pylist {+ d.extend(objs)+ } else {+ crate::bail!("expected a list, got {pylist:?}")+ }+ }+ OpCode::SetItem => {+ let value = self.pop()?;+ let key = self.pop()?;+ let pydict = self.last()?;+ if let Object::Dict(d) = pydict {+ d.push((key, value))+ } else {+ crate::bail!("expected a dict, got {pydict:?}")+ }+ }+ OpCode::SetItems => {+ let mut objs = self.pop_to_marker()?;+ let pydict = self.last()?;+ if let Object::Dict(d) = pydict {+ if objs.len() % 2 != 0 {+ crate::bail!("setitems: not an even number of objects")+ }+ while let Some(value) = objs.pop() {+ let key = objs.pop().context("empty objs")?;+ d.push((key, value))+ }+ } else {+ crate::bail!("expected a dict, got {pydict:?}")+ }+ }+ OpCode::None => self.push(Object::None),+ OpCode::Stop => {+ return Ok(true);+ }+ OpCode::Build => self.build()?,+ OpCode::EmptyDict => self.push(Object::Dict(vec![])),+ OpCode::Dict => {+ let mut objs = self.pop_to_marker()?;+ let mut pydict = vec![];+ if objs.len() % 2 != 0 {+ crate::bail!("setitems: not an even number of objects")+ }+ while let Some(value) = objs.pop() {+ let key = objs.pop().context("empty objs")?;+ pydict.push((key, value))+ }+ self.push(Object::Dict(pydict))+ }+ OpCode::Mark => self.push(Object::Mark),+ OpCode::Reduce => self.reduce()?,+ OpCode::EmptyTuple => self.push(Object::Tuple(vec![])),+ OpCode::EmptyList => self.push(Object::List(vec![])),+ OpCode::BinGet => {+ let arg = r.read_u8()?;+ let obj = self.memo_get(arg as u32)?;+ self.push(obj)+ }+ OpCode::LongBinGet => {+ let arg = r.read_u32::<LittleEndian>()?;+ let obj = self.memo_get(arg)?;+ self.push(obj)+ }+ OpCode::BinPut => {+ let arg = r.read_u8()?;+ self.memo_put(arg as u32)?+ }+ OpCode::LongBinPut => {+ let arg = r.read_u32::<LittleEndian>()?;+ self.memo_put(arg)?+ }+ OpCode::NewObj => {+ let args = self.pop()?;+ let class = self.pop()?;+ let obj = self.new_obj(class, args)?;+ self.push(obj)+ }+ OpCode::Long1 => {+ let n_bytes = r.read_u8()?;+ let mut v = 0;+ // Decode the next n bytes in little endian+ for i in 0..n_bytes {+ v |= (r.read_u8()? as i64) << (i * 8);+ }+ self.push(Object::Long(v))+ }+ }+ Ok(false)+ }+}++impl From<Object> for E {+ fn from(value: Object) -> Self {+ E::Msg(format!("conversion error on {value:?}"))+ }+}++// https://github.com/pytorch/pytorch/blob/4eac43d046ded0f0a5a5fa8db03eb40f45bf656e/torch/_utils.py#L198+// Arguments: storage, storage_offset, size, stride, requires_grad, backward_hooks+fn rebuild_args(args: Object) -> Result<(Layout, DType, String, usize)> {+ let mut args = args.tuple()?;+ let stride = Vec::<usize>::try_from(args.remove(3))?;+ let size = Vec::<usize>::try_from(args.remove(2))?;+ let offset = args.remove(1).int_or_long()? as usize;+ let storage = args.remove(0).persistent_load()?;+ let mut storage = storage.tuple()?;+ let storage_size = storage.remove(4).int_or_long()? as usize;+ let path = storage.remove(2).unicode()?;+ let (_module_name, class_name) = storage.remove(1).class()?;+ let dtype = match class_name.as_str() {+ "FloatStorage" => DType::F32,+ "DoubleStorage" => DType::F64,+ "HalfStorage" => DType::F16,+ "BFloat16Storage" => DType::BF16,+ "ByteStorage" => DType::U8,+ "LongStorage" => DType::I64,+ other => {+ crate::bail!("unsupported storage type {other}")+ }+ };+ let layout = Layout::new(+ crate::Shape::from(size),+ stride,+ offset * dtype.size_in_bytes(),+ );+ Ok((layout, dtype, path, storage_size))+}++#[derive(Debug, Clone)]+pub struct TensorInfo {+ pub name: String,+ pub dtype: DType,+ pub layout: Layout,+ pub path: String,+ pub storage_size: usize,+}++/// Read the tensor info from a .pth file.+///+/// # Arguments+/// * `file` - The path to the .pth file.+/// * `verbose` - Whether to print debug information.+/// * `key` - Optional key to retrieve `state_dict` from the pth file.+pub fn read_pth_tensor_info<P: AsRef<std::path::Path>>(+ file: P,+ verbose: bool,+ key: Option<&str>,+) -> Result<Vec<TensorInfo>> {+ let file = std::fs::File::open(file)?;+ let zip_reader = std::io::BufReader::new(file);+ let mut zip = zip::ZipArchive::new(zip_reader)?;+ let zip_file_names = zip+ .file_names()+ .map(|f| f.to_string())+ .collect::<Vec<String>>();++ let mut tensor_infos = vec![];+ for file_name in zip_file_names.iter() {+ if !file_name.ends_with("data.pkl") {+ continue;+ }+ let dir_name = std::path::PathBuf::from(file_name.strip_suffix(".pkl").context("no .pkl")?);+ let reader = zip.by_name(file_name)?;+ let mut reader = std::io::BufReader::new(reader);+ let mut stack = Stack::empty();+ stack.read_loop(&mut reader)?;+ let obj = stack.finalize()?;+ if VERBOSE || verbose {+ println!("{obj:#?}");+ }++ let obj = match obj {+ Object::Build { callable, args } => match *callable {+ Object::Reduce { callable, args: _ } => match *callable {+ Object::Class {+ module_name,+ class_name,+ } if module_name == "__torch__" && class_name == "Module" => *args,+ _ => continue,+ },+ _ => continue,+ },+ obj => obj,+ };++ // If key is provided, then we need to extract the state_dict from the object.+ let obj = if let Some(key) = key {+ if let Object::Dict(key_values) = obj {+ key_values+ .into_iter()+ .find(|(k, _)| *k == Object::Unicode(key.to_owned()))+ .map(|(_, v)| v)+ .ok_or_else(|| E::Msg(format!("key {key} not found")))?+ } else {+ obj+ }+ } else {+ obj+ };++ // If the object is a dict, then we can extract the tensor info from it.+ // NOTE: We are assuming that the `obj` is state_dict by this stage.+ if let Object::Dict(key_values) = obj {+ for (name, value) in key_values.into_iter() {+ match value.into_tensor_info(name, &dir_name) {+ Ok(Some(tensor_info)) => tensor_infos.push(tensor_info),+ Ok(None) => {}+ Err(err) => eprintln!("skipping: {err:?}"),+ }+ }+ }+ }+ Ok(tensor_infos)+}++/// Lazy tensor loader.+pub struct PthTensors {+ tensor_infos: HashMap<String, TensorInfo>,+ path: std::path::PathBuf,+ // We do not store a zip reader as it needs mutable access to extract data. Instead we+ // re-create a zip reader for each tensor.+}++impl PthTensors {+ pub fn new<P: AsRef<std::path::Path>>(path: P, key: Option<&str>) -> Result<Self> {+ let tensor_infos = read_pth_tensor_info(path.as_ref(), false, key)?;+ let tensor_infos = tensor_infos+ .into_iter()+ .map(|ti| (ti.name.to_string(), ti))+ .collect();+ let path = path.as_ref().to_owned();+ Ok(Self { tensor_infos, path })+ }++ pub fn tensor_infos(&self) -> &HashMap<String, TensorInfo> {+ &self.tensor_infos+ }++ pub fn get(&self, name: &str) -> Result<Option<Tensor>> {+ use std::io::Read;+ let tensor_info = match self.tensor_infos.get(name) {+ None => return Ok(None),+ Some(tensor_info) => tensor_info,+ };+ // We hope that the file has not changed since first reading it.+ let zip_reader = std::io::BufReader::new(std::fs::File::open(&self.path)?);+ let mut zip = zip::ZipArchive::new(zip_reader)?;+ let mut reader = zip.by_name(&tensor_info.path)?;+ let is_fortran_contiguous = tensor_info.layout.is_fortran_contiguous();+ let rank = tensor_info.layout.shape().rank();++ // Reading the data is a bit tricky as it can be strided, for now only support the basic+ // case and when the tensor is fortran contiguous.+ if !tensor_info.layout.is_contiguous() && !is_fortran_contiguous {+ crate::bail!(+ "cannot retrieve non-contiguous tensors {:?}",+ tensor_info.layout+ )+ }+ let start_offset = tensor_info.layout.start_offset();+ if start_offset > 0 {+ std::io::copy(+ &mut reader.by_ref().take(start_offset as u64),+ &mut std::io::sink(),+ )?;+ }+ let tensor = Tensor::from_reader(+ tensor_info.layout.shape().clone(),+ tensor_info.dtype,+ &mut reader,+ )?;++ if rank > 1 && is_fortran_contiguous {+ // Reverse the shape, e.g. Shape(2, 3, 4) -> Shape(4, 3, 2)+ let shape_reversed: Vec<_> = tensor_info.layout.dims().iter().rev().cloned().collect();+ let tensor = tensor.reshape(shape_reversed)?;++ // Permute (transpose) the dimensions, e.g. Shape(4, 3, 2) -> Shape(2, 3, 4)+ let dim_indices_reversed: Vec<_> = (0..rank).rev().collect();+ let tensor = tensor.permute(dim_indices_reversed)?;+ Ok(Some(tensor))+ } else {+ Ok(Some(tensor))+ }+ }+}++/// Read all the tensors from a PyTorch pth file with a given key.+///+/// # Arguments+/// * `path` - Path to the pth file.+/// * `key` - Optional key to retrieve `state_dict` from the pth file. Sometimes the pth file+/// contains multiple objects and the state_dict is the one we are interested in.+pub fn read_all_with_key<P: AsRef<std::path::Path>>(+ path: P,+ key: Option<&str>,+) -> Result<Vec<(String, Tensor)>> {+ let pth = PthTensors::new(path, key)?;+ let tensor_names = pth.tensor_infos.keys();+ let mut tensors = Vec::with_capacity(tensor_names.len());+ for name in tensor_names {+ if let Some(tensor) = pth.get(name)? {+ tensors.push((name.to_string(), tensor))+ }+ }+ Ok(tensors)+}++/// Read all the tensors from a PyTorch pth file.+///+/// # Arguments+/// * `path` - Path to the pth file.+pub fn read_all<P: AsRef<std::path::Path>>(path: P) -> Result<Vec<(String, Tensor)>> {+ read_all_with_key(path, None)+}diff --git a/vendor/candle-core/src/quantized/avx.rs b/vendor/candle-core/src/quantized/avx.rsnew file mode 100644index 0000000..01fdaf5--- /dev/null+++ b/vendor/candle-core/src/quantized/avx.rs@@ -0,0 +1,677 @@+use super::k_quants::{+ BlockQ2K, BlockQ3K, BlockQ4K, BlockQ4_0, BlockQ5K, BlockQ6K, BlockQ8K, BlockQ8_0, QK8_0, QK_K,+};+use byteorder::{ByteOrder, LittleEndian};+use half::f16;++#[cfg(target_arch = "x86")]+use core::arch::x86::*;+#[cfg(target_arch = "x86_64")]+use core::arch::x86_64::*;++#[target_feature(enable = "avx2,fma")]+pub(crate) fn sum_i16_pairs_float(x: __m256i) -> __m256 {+ let ones = _mm256_set1_epi16(1);+ let summed_pairs = _mm256_madd_epi16(ones, x);+ _mm256_cvtepi32_ps(summed_pairs)+}++#[target_feature(enable = "avx2,fma")]+pub(crate) fn mul_sum_us8_pairs_float(ax: __m256i, sy: __m256i) -> __m256 {+ let dot = _mm256_maddubs_epi16(ax, sy);+ sum_i16_pairs_float(dot)+}++#[target_feature(enable = "avx2,fma")]+pub(crate) fn hsum_float_8(x: __m256) -> f32 {+ let res = _mm256_extractf128_ps(x, 1);+ let res = _mm_add_ps(res, _mm256_castps256_ps128(x));+ let res = _mm_add_ps(res, _mm_movehl_ps(res, res));+ let res = _mm_add_ss(res, _mm_movehdup_ps(res));+ _mm_cvtss_f32(res)+}++/// # Safety+/// `rsi` must point to at least 16 readable bytes.+#[target_feature(enable = "avx2,fma")]+pub(crate) unsafe fn bytes_from_nibbles_32(rsi: *const u8) -> __m256i {+ let tmp = unsafe { _mm_loadu_si128(rsi as *const __m128i) };+ let bytes = _mm256_insertf128_si256::<1>(_mm256_castsi128_si256(tmp), _mm_srli_epi16(tmp, 4));+ let low_mask = _mm256_set1_epi8(0xF);+ _mm256_and_si256(low_mask, bytes)+}++#[target_feature(enable = "avx2,fma")]+pub(crate) fn mul_sum_i8_pairs_float(x: __m256i, y: __m256i) -> __m256 {+ let ax = _mm256_sign_epi8(x, x);+ let sy = _mm256_sign_epi8(y, x);+ mul_sum_us8_pairs_float(ax, sy)+}++#[target_feature(enable = "avx2,fma")]+pub(crate) fn vec_dot_q4_0_q8_0(n: usize, xs: &[BlockQ4_0], ys: &[BlockQ8_0]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK8_0),+ "vec_dot_q4_0_q8_0: {n} is not divisible by {QK8_0}"+ );+ unsafe {+ let mut acc = _mm256_setzero_ps();+ for (x, y) in xs.iter().zip(ys.iter()) {+ let d = _mm256_set1_ps(f16::to_f32(x.d) * f16::to_f32(y.d));+ let bx = bytes_from_nibbles_32(x.qs.as_ptr());+ let off = _mm256_set1_epi8(8);+ let bx = _mm256_sub_epi8(bx, off);+ let by = _mm256_loadu_si256(y.qs.as_ptr() as *const __m256i);+ let q = mul_sum_i8_pairs_float(bx, by);+ acc = _mm256_fmadd_ps(d, q, acc);+ }+ hsum_float_8(acc)+ }+}++#[target_feature(enable = "avx2,fma")]+pub(crate) fn vec_dot_q8_0_q8_0(n: usize, xs: &[BlockQ8_0], ys: &[BlockQ8_0]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK8_0),+ "vec_dot_q8_0_q8_0: {n} is not divisible by {QK8_0}"+ );+ unsafe {+ let mut acc = _mm256_setzero_ps();+ for (x, y) in xs.iter().zip(ys.iter()) {+ let d = _mm256_set1_ps(f16::to_f32(x.d) * f16::to_f32(y.d));+ let bx = _mm256_loadu_si256(x.qs.as_ptr() as *const __m256i);+ let by = _mm256_loadu_si256(y.qs.as_ptr() as *const __m256i);+ let q = mul_sum_i8_pairs_float(bx, by);+ acc = _mm256_fmadd_ps(d, q, acc);+ }+ hsum_float_8(acc)+ }+}++/// # Safety+/// `i` must be less than 8.+#[target_feature(enable = "avx2,fma")]+unsafe fn get_scale_shuffle(i: usize) -> __m128i {+ const K_SHUFFLE: [u8; 128] = [+ 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 3, 3, 3, 3, 3, 3,+ 3, 3, 4, 4, 4, 4, 4, 4, 4, 4, 5, 5, 5, 5, 5, 5, 5, 5, 6, 6, 6, 6, 6, 6, 6, 6, 7, 7, 7, 7,+ 7, 7, 7, 7, 8, 8, 8, 8, 8, 8, 8, 8, 9, 9, 9, 9, 9, 9, 9, 9, 10, 10, 10, 10, 10, 10, 10, 10,+ 11, 11, 11, 11, 11, 11, 11, 11, 12, 12, 12, 12, 12, 12, 12, 12, 13, 13, 13, 13, 13, 13, 13,+ 13, 14, 14, 14, 14, 14, 14, 14, 14, 15, 15, 15, 15, 15, 15, 15, 15,+ ];+ unsafe { _mm_loadu_si128((K_SHUFFLE.as_ptr() as *const __m128i).add(i)) }+}++/// # Safety+/// `i` must be less than 8.+#[target_feature(enable = "avx2,fma")]+unsafe fn get_scale_shuffle_k4(i: usize) -> __m256i {+ const K_SHUFFLE: [u8; 256] = [+ 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1,+ 0, 1, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3,+ 2, 3, 2, 3, 4, 5, 4, 5, 4, 5, 4, 5, 4, 5, 4, 5, 4, 5, 4, 5, 4, 5, 4, 5, 4, 5, 4, 5, 4, 5,+ 4, 5, 4, 5, 4, 5, 6, 7, 6, 7, 6, 7, 6, 7, 6, 7, 6, 7, 6, 7, 6, 7, 6, 7, 6, 7, 6, 7, 6, 7,+ 6, 7, 6, 7, 6, 7, 6, 7, 8, 9, 8, 9, 8, 9, 8, 9, 8, 9, 8, 9, 8, 9, 8, 9, 8, 9, 8, 9, 8, 9,+ 8, 9, 8, 9, 8, 9, 8, 9, 8, 9, 10, 11, 10, 11, 10, 11, 10, 11, 10, 11, 10, 11, 10, 11, 10,+ 11, 10, 11, 10, 11, 10, 11, 10, 11, 10, 11, 10, 11, 10, 11, 10, 11, 12, 13, 12, 13, 12, 13,+ 12, 13, 12, 13, 12, 13, 12, 13, 12, 13, 12, 13, 12, 13, 12, 13, 12, 13, 12, 13, 12, 13, 12,+ 13, 12, 13, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15,+ 14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15,+ ];+ unsafe { _mm256_loadu_si256((K_SHUFFLE.as_ptr() as *const __m256i).add(i)) }+}++/// # Safety+/// `i` must be less than 4.+#[target_feature(enable = "avx2,fma")]+unsafe fn get_scale_shuffle_q3k(i: usize) -> __m256i {+ const K_SHUFFLE: [u8; 128] = [+ 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 0, 1, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3, 2, 3,+ 2, 3, 4, 5, 4, 5, 4, 5, 4, 5, 4, 5, 4, 5, 4, 5, 4, 5, 6, 7, 6, 7, 6, 7, 6, 7, 6, 7, 6, 7,+ 6, 7, 6, 7, 8, 9, 8, 9, 8, 9, 8, 9, 8, 9, 8, 9, 8, 9, 8, 9, 10, 11, 10, 11, 10, 11, 10, 11,+ 10, 11, 10, 11, 10, 11, 10, 11, 12, 13, 12, 13, 12, 13, 12, 13, 12, 13, 12, 13, 12, 13, 12,+ 13, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15, 14, 15,+ ];+ unsafe { _mm256_loadu_si256((K_SHUFFLE.as_ptr() as *const __m256i).add(i)) }+}++#[target_feature(enable = "avx2,fma")]+pub(crate) fn vec_dot_q6k_q8k(n: usize, xs: &[BlockQ6K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q6k_8k: {n} is not divisible by {QK_K}"+ );++ unsafe {+ let m4 = _mm256_set1_epi8(0xF);+ let m2 = _mm256_set1_epi8(3);+ let m32s = _mm256_set1_epi8(32);+ let mut acc = _mm256_setzero_ps();+ for (x, y) in xs.iter().zip(ys.iter()) {+ let d = y.d * x.d.to_f32();+ let mut q4 = x.ql.as_ptr();+ let mut qh = x.qh.as_ptr();+ let mut q8 = y.qs.as_ptr();++ let scales = _mm_loadu_si128(x.scales.as_ptr() as *const __m128i);+ let mut sumi = _mm256_setzero_si256();++ for j in 0..QK_K / 128 {+ let is = j * 4;+ let scale_0 = _mm_shuffle_epi8(scales, get_scale_shuffle(is));+ let scale_1 = _mm_shuffle_epi8(scales, get_scale_shuffle(is + 1));+ let scale_2 = _mm_shuffle_epi8(scales, get_scale_shuffle(is + 2));+ let scale_3 = _mm_shuffle_epi8(scales, get_scale_shuffle(is + 3));++ let q4bits1 = _mm256_loadu_si256(q4 as *const __m256i);+ q4 = q4.add(32);+ let q4bits2 = _mm256_loadu_si256(q4 as *const __m256i);+ q4 = q4.add(32);+ let q4bits_h = _mm256_loadu_si256(qh as *const __m256i);+ qh = qh.add(32);++ let q4h_0 = _mm256_slli_epi16(_mm256_and_si256(q4bits_h, m2), 4);+ let q4h_1 =+ _mm256_slli_epi16(_mm256_and_si256(_mm256_srli_epi16(q4bits_h, 2), m2), 4);+ let q4h_2 =+ _mm256_slli_epi16(_mm256_and_si256(_mm256_srli_epi16(q4bits_h, 4), m2), 4);+ let q4h_3 =+ _mm256_slli_epi16(_mm256_and_si256(_mm256_srli_epi16(q4bits_h, 6), m2), 4);++ let q4_0 = _mm256_or_si256(_mm256_and_si256(q4bits1, m4), q4h_0);+ let q4_1 = _mm256_or_si256(_mm256_and_si256(q4bits2, m4), q4h_1);+ let q4_2 =+ _mm256_or_si256(_mm256_and_si256(_mm256_srli_epi16(q4bits1, 4), m4), q4h_2);+ let q4_3 =+ _mm256_or_si256(_mm256_and_si256(_mm256_srli_epi16(q4bits2, 4), m4), q4h_3);++ let q8_0 = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);+ let q8_1 = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);+ let q8_2 = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);+ let q8_3 = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);++ let q8s_0 = _mm256_maddubs_epi16(m32s, q8_0);+ let q8s_1 = _mm256_maddubs_epi16(m32s, q8_1);+ let q8s_2 = _mm256_maddubs_epi16(m32s, q8_2);+ let q8s_3 = _mm256_maddubs_epi16(m32s, q8_3);++ let p16_0 = _mm256_maddubs_epi16(q4_0, q8_0);+ let p16_1 = _mm256_maddubs_epi16(q4_1, q8_1);+ let p16_2 = _mm256_maddubs_epi16(q4_2, q8_2);+ let p16_3 = _mm256_maddubs_epi16(q4_3, q8_3);++ let p16_0 = _mm256_sub_epi16(p16_0, q8s_0);+ let p16_1 = _mm256_sub_epi16(p16_1, q8s_1);+ let p16_2 = _mm256_sub_epi16(p16_2, q8s_2);+ let p16_3 = _mm256_sub_epi16(p16_3, q8s_3);++ let p16_0 = _mm256_madd_epi16(_mm256_cvtepi8_epi16(scale_0), p16_0);+ let p16_1 = _mm256_madd_epi16(_mm256_cvtepi8_epi16(scale_1), p16_1);+ let p16_2 = _mm256_madd_epi16(_mm256_cvtepi8_epi16(scale_2), p16_2);+ let p16_3 = _mm256_madd_epi16(_mm256_cvtepi8_epi16(scale_3), p16_3);++ sumi = _mm256_add_epi32(sumi, _mm256_add_epi32(p16_0, p16_1));+ sumi = _mm256_add_epi32(sumi, _mm256_add_epi32(p16_2, p16_3));+ }+ acc = _mm256_fmadd_ps(_mm256_broadcast_ss(&d), _mm256_cvtepi32_ps(sumi), acc);+ }+ hsum_float_8(acc)+ }+}++#[target_feature(enable = "avx2,fma")]+fn mm256_set_m128i(a: __m128i, b: __m128i) -> __m256i {+ _mm256_insertf128_si256(_mm256_castsi128_si256(b), a, 1)+}++#[target_feature(enable = "avx2,fma")]+pub(crate) fn vec_dot_q2k_q8k(n: usize, xs: &[BlockQ2K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q2k_q8k: {n} is not divisible by {QK_K}"+ );++ unsafe {+ let m3 = _mm256_set1_epi8(3);+ let m4 = _mm_set1_epi8(0xF);++ let mut acc = _mm256_setzero_ps();++ for (x, y) in xs.iter().zip(ys.iter()) {+ let d = y.d * x.d.to_f32();+ let dmin = -y.d * x.dmin.to_f32();++ let mut q2 = x.qs.as_ptr();+ let mut q8 = y.qs.as_ptr();++ let mins_and_scales = _mm_loadu_si128(x.scales.as_ptr() as *const __m128i);+ let scales8 = _mm_and_si128(mins_and_scales, m4);+ let mins8 = _mm_and_si128(_mm_srli_epi16(mins_and_scales, 4), m4);+ let mins = _mm256_cvtepi8_epi16(mins8);+ let prod =+ _mm256_madd_epi16(mins, _mm256_loadu_si256(y.bsums.as_ptr() as *const __m256i));++ acc = _mm256_fmadd_ps(_mm256_broadcast_ss(&dmin), _mm256_cvtepi32_ps(prod), acc);++ let all_scales = _mm256_cvtepi8_epi16(scales8);+ let l_scales = _mm256_extracti128_si256(all_scales, 0);+ let h_scales = _mm256_extracti128_si256(all_scales, 1);+ let scales = [+ mm256_set_m128i(l_scales, l_scales),+ mm256_set_m128i(h_scales, h_scales),+ ];++ let mut sumi = _mm256_setzero_si256();++ for scale in scales {+ let q2bits = _mm256_loadu_si256(q2 as *const __m256i);+ q2 = q2.add(32);++ let q8_0 = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);+ let q8_1 = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);+ let q8_2 = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);+ let q8_3 = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);++ let q2_0 = _mm256_and_si256(q2bits, m3);+ let q2_1 = _mm256_and_si256(_mm256_srli_epi16(q2bits, 2), m3);+ let q2_2 = _mm256_and_si256(_mm256_srli_epi16(q2bits, 4), m3);+ let q2_3 = _mm256_and_si256(_mm256_srli_epi16(q2bits, 6), m3);++ let p0 = _mm256_maddubs_epi16(q2_0, q8_0);+ let p1 = _mm256_maddubs_epi16(q2_1, q8_1);+ let p2 = _mm256_maddubs_epi16(q2_2, q8_2);+ let p3 = _mm256_maddubs_epi16(q2_3, q8_3);++ let p0 =+ _mm256_madd_epi16(_mm256_shuffle_epi8(scale, get_scale_shuffle_q3k(0)), p0);+ let p1 =+ _mm256_madd_epi16(_mm256_shuffle_epi8(scale, get_scale_shuffle_q3k(1)), p1);+ let p2 =+ _mm256_madd_epi16(_mm256_shuffle_epi8(scale, get_scale_shuffle_q3k(2)), p2);+ let p3 =+ _mm256_madd_epi16(_mm256_shuffle_epi8(scale, get_scale_shuffle_q3k(3)), p3);++ let p0 = _mm256_add_epi32(p0, p1);+ let p2 = _mm256_add_epi32(p2, p3);++ sumi = _mm256_add_epi32(sumi, _mm256_add_epi32(p0, p2));+ }+ acc = _mm256_fmadd_ps(_mm256_broadcast_ss(&d), _mm256_cvtepi32_ps(sumi), acc);+ }++ hsum_float_8(acc)+ }+}++#[target_feature(enable = "avx2,fma")]+pub(crate) fn vec_dot_q3k_q8k(n: usize, xs: &[BlockQ3K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q3k_q8k: {n} is not divisible by {QK_K}"+ );++ const KMASK1: u32 = 0x03030303;+ const KMASK2: u32 = 0x0f0f0f0f;++ let mut aux = [0u32; 3];++ unsafe {+ let m3 = _mm256_set1_epi8(3);+ let mone = _mm256_set1_epi8(1);+ let m32 = _mm_set1_epi8(32);++ let mut acc = _mm256_setzero_ps();+ for (x, y) in xs.iter().zip(ys.iter()) {+ let d = y.d * x.d.to_f32();++ let mut q3 = x.qs.as_ptr();+ let mut q8 = y.qs.as_ptr();++ LittleEndian::read_u32_into(&x.scales, &mut aux);+ let scales128 = _mm_set_epi32(+ (((aux[1] >> 4) & KMASK2) | (((aux[2] >> 6) & KMASK1) << 4)) as i32,+ (((aux[0] >> 4) & KMASK2) | (((aux[2] >> 4) & KMASK1) << 4)) as i32,+ ((aux[1] & KMASK2) | (((aux[2] >> 2) & KMASK1) << 4)) as i32,+ ((aux[0] & KMASK2) | (((aux[2]) & KMASK1) << 4)) as i32,+ );+ let scales128 = _mm_sub_epi8(scales128, m32);+ let all_scales = _mm256_cvtepi8_epi16(scales128);+ let l_scales = _mm256_extracti128_si256(all_scales, 0);+ let h_scales = _mm256_extracti128_si256(all_scales, 1);+ let scales = [+ mm256_set_m128i(l_scales, l_scales),+ mm256_set_m128i(h_scales, h_scales),+ ];++ // high bit+ let hbits = _mm256_loadu_si256(x.hmask.as_ptr() as *const __m256i);++ let mut sumi = _mm256_setzero_si256();++ for (j, scale) in scales.iter().enumerate() {+ // load low 2 bits+ let q3bits = _mm256_loadu_si256(q3 as *const __m256i);+ q3 = q3.add(32);++ // Prepare low and high bits+ // We hardcode the shifts here to avoid loading them into a separate register+ let q3l_0 = _mm256_and_si256(q3bits, m3);+ let q3h_0 = if j == 0 {+ _mm256_srli_epi16(_mm256_andnot_si256(hbits, _mm256_slli_epi16(mone, 0)), 0)+ } else {+ _mm256_srli_epi16(_mm256_andnot_si256(hbits, _mm256_slli_epi16(mone, 4)), 4)+ };+ let q3h_0 = _mm256_slli_epi16(q3h_0, 2);++ let q3l_1 = _mm256_and_si256(_mm256_srli_epi16(q3bits, 2), m3);+ let q3h_1 = if j == 0 {+ _mm256_srli_epi16(_mm256_andnot_si256(hbits, _mm256_slli_epi16(mone, 1)), 1)+ } else {+ _mm256_srli_epi16(_mm256_andnot_si256(hbits, _mm256_slli_epi16(mone, 5)), 5)+ };+ let q3h_1 = _mm256_slli_epi16(q3h_1, 2);++ let q3l_2 = _mm256_and_si256(_mm256_srli_epi16(q3bits, 4), m3);+ let q3h_2 = if j == 0 {+ _mm256_srli_epi16(_mm256_andnot_si256(hbits, _mm256_slli_epi16(mone, 2)), 2)+ } else {+ _mm256_srli_epi16(_mm256_andnot_si256(hbits, _mm256_slli_epi16(mone, 6)), 6)+ };+ let q3h_2 = _mm256_slli_epi16(q3h_2, 2);++ let q3l_3 = _mm256_and_si256(_mm256_srli_epi16(q3bits, 6), m3);+ let q3h_3 = if j == 0 {+ _mm256_srli_epi16(_mm256_andnot_si256(hbits, _mm256_slli_epi16(mone, 3)), 3)+ } else {+ _mm256_srli_epi16(_mm256_andnot_si256(hbits, _mm256_slli_epi16(mone, 7)), 7)+ };+ let q3h_3 = _mm256_slli_epi16(q3h_3, 2);++ // load Q8 quants+ let q8_0 = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);+ let q8_1 = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);+ let q8_2 = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);+ let q8_3 = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);++ // Dot product: we multiply the 2 low bits and 1 high bit part separately, so we+ // can use _mm256_maddubs_epi16, and then subtract. The high bit part has the 2+ // already subtracted (and so, it is zero if the high bit was not set, and 2 if the+ // high bit was set)+ let q8s_0 = _mm256_maddubs_epi16(q3h_0, q8_0);+ let q8s_1 = _mm256_maddubs_epi16(q3h_1, q8_1);+ let q8s_2 = _mm256_maddubs_epi16(q3h_2, q8_2);+ let q8s_3 = _mm256_maddubs_epi16(q3h_3, q8_3);++ let p16_0 = _mm256_maddubs_epi16(q3l_0, q8_0);+ let p16_1 = _mm256_maddubs_epi16(q3l_1, q8_1);+ let p16_2 = _mm256_maddubs_epi16(q3l_2, q8_2);+ let p16_3 = _mm256_maddubs_epi16(q3l_3, q8_3);++ let p16_0 = _mm256_sub_epi16(p16_0, q8s_0);+ let p16_1 = _mm256_sub_epi16(p16_1, q8s_1);+ let p16_2 = _mm256_sub_epi16(p16_2, q8s_2);+ let p16_3 = _mm256_sub_epi16(p16_3, q8s_3);++ // multiply with scales+ let p16_0 =+ _mm256_madd_epi16(_mm256_shuffle_epi8(*scale, get_scale_shuffle_q3k(0)), p16_0);+ let p16_1 =+ _mm256_madd_epi16(_mm256_shuffle_epi8(*scale, get_scale_shuffle_q3k(1)), p16_1);+ let p16_2 =+ _mm256_madd_epi16(_mm256_shuffle_epi8(*scale, get_scale_shuffle_q3k(2)), p16_2);+ let p16_3 =+ _mm256_madd_epi16(_mm256_shuffle_epi8(*scale, get_scale_shuffle_q3k(3)), p16_3);++ // accumulate+ let p16_0 = _mm256_add_epi32(p16_0, p16_1);+ let p16_2 = _mm256_add_epi32(p16_2, p16_3);+ sumi = _mm256_add_epi32(sumi, _mm256_add_epi32(p16_0, p16_2));+ }++ // multiply with block scale and accumulate+ acc = _mm256_fmadd_ps(_mm256_broadcast_ss(&d), _mm256_cvtepi32_ps(sumi), acc);+ }+ hsum_float_8(acc)+ }+}++#[target_feature(enable = "avx2,fma")]+pub(crate) fn vec_dot_q4k_q8k(n: usize, xs: &[BlockQ4K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q4k_q8k: {n} is not divisible by {QK_K}"+ );+ let mut utmp = [0u32; 4];+ const KMASK1: u32 = 0x3f3f3f3f;+ const KMASK2: u32 = 0x0f0f0f0f;+ const KMASK3: u32 = 0x03030303;++ unsafe {+ let m4 = _mm256_set1_epi8(0xF);++ let mut acc = _mm256_setzero_ps();+ let mut acc_m = _mm_setzero_ps();++ for (x, y) in xs.iter().zip(ys.iter()) {+ let d = y.d * x.d.to_f32();+ let dmin = -y.d * x.dmin.to_f32();++ LittleEndian::read_u32_into(&x.scales, &mut utmp[0..3]);++ utmp[3] = ((utmp[2] >> 4) & KMASK2) | (((utmp[1] >> 6) & KMASK3) << 4);+ let uaux = utmp[1] & KMASK1;+ utmp[1] = (utmp[2] & KMASK2) | (((utmp[0] >> 6) & KMASK3) << 4);+ utmp[2] = uaux;+ utmp[0] &= KMASK1;++ let mut q4 = x.qs.as_ptr();+ let mut q8 = y.qs.as_ptr();++ let mins_and_scales = _mm256_cvtepu8_epi16(_mm_set_epi32(+ utmp[3] as i32,+ utmp[2] as i32,+ utmp[1] as i32,+ utmp[0] as i32,+ ));++ let q8sums = _mm256_loadu_si256(y.bsums.as_ptr() as *const __m256i);+ let q8s = _mm_hadd_epi16(+ _mm256_extracti128_si256(q8sums, 0),+ _mm256_extracti128_si256(q8sums, 1),+ );+ let prod = _mm_madd_epi16(_mm256_extracti128_si256(mins_and_scales, 1), q8s);+ acc_m = _mm_fmadd_ps(_mm_set1_ps(dmin), _mm_cvtepi32_ps(prod), acc_m);++ let sc128 = _mm256_extracti128_si256(mins_and_scales, 0);+ let scales = mm256_set_m128i(sc128, sc128);++ let mut sumi = _mm256_setzero_si256();++ for j in 0..QK_K / 64 {+ let scale_l = _mm256_shuffle_epi8(scales, get_scale_shuffle_k4(2 * j));+ let scale_h = _mm256_shuffle_epi8(scales, get_scale_shuffle_k4(2 * j + 1));++ let q4bits = _mm256_loadu_si256(q4 as *const __m256i);+ q4 = q4.add(32);+ let q4l = _mm256_and_si256(q4bits, m4);+ let q4h = _mm256_and_si256(_mm256_srli_epi16(q4bits, 4), m4);++ let q8l = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);+ let p16l = _mm256_maddubs_epi16(q4l, q8l);+ let p16l = _mm256_madd_epi16(scale_l, p16l);+ sumi = _mm256_add_epi32(sumi, p16l);++ let q8h = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);+ let p16h = _mm256_maddubs_epi16(q4h, q8h);+ let p16h = _mm256_madd_epi16(scale_h, p16h);+ sumi = _mm256_add_epi32(sumi, p16h);+ }++ let vd = _mm256_set1_ps(d);+ acc = _mm256_fmadd_ps(vd, _mm256_cvtepi32_ps(sumi), acc);+ }++ let acc_m = _mm_add_ps(acc_m, _mm_movehl_ps(acc_m, acc_m));+ let acc_m = _mm_add_ss(acc_m, _mm_movehdup_ps(acc_m));++ hsum_float_8(acc) + _mm_cvtss_f32(acc_m)+ }+}++#[target_feature(enable = "avx2,fma")]+pub(crate) fn vec_dot_q5k_q8k(n: usize, xs: &[BlockQ5K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q5k_q8k: {n} is not divisible by {QK_K}"+ );+ let mut utmp = [0u32; 4];+ const KMASK1: u32 = 0x3f3f3f3f;+ const KMASK2: u32 = 0x0f0f0f0f;+ const KMASK3: u32 = 0x03030303;++ unsafe {+ let m4 = _mm256_set1_epi8(0xF);+ let mzero = _mm_setzero_si128();+ let mone = _mm256_set1_epi8(1);++ let mut acc = _mm256_setzero_ps();+ let mut summs = 0.0;++ for (x, y) in xs.iter().zip(ys.iter()) {+ let d = y.d * x.d.to_f32();+ let dmin = -y.d * x.dmin.to_f32();++ LittleEndian::read_u32_into(&x.scales, &mut utmp[0..3]);++ utmp[3] = ((utmp[2] >> 4) & KMASK2) | (((utmp[1] >> 6) & KMASK3) << 4);+ let uaux = utmp[1] & KMASK1;+ utmp[1] = (utmp[2] & KMASK2) | (((utmp[0] >> 6) & KMASK3) << 4);+ utmp[2] = uaux;+ utmp[0] &= KMASK1;++ let mut q5 = x.qs.as_ptr();+ let mut q8 = y.qs.as_ptr();++ let mins_and_scales = _mm256_cvtepu8_epi16(_mm_set_epi32(+ utmp[3] as i32,+ utmp[2] as i32,+ utmp[1] as i32,+ utmp[0] as i32,+ ));++ let q8sums = _mm256_loadu_si256(y.bsums.as_ptr() as *const __m256i);+ let q8s = _mm_hadd_epi16(+ _mm256_extracti128_si256(q8sums, 0),+ _mm256_extracti128_si256(q8sums, 1),+ );+ let prod = _mm_madd_epi16(_mm256_extracti128_si256(mins_and_scales, 1), q8s);+ let hsum = _mm_hadd_epi32(_mm_hadd_epi32(prod, mzero), mzero);+ summs += dmin * _mm_extract_epi32(hsum, 0) as f32;++ let sc128 = _mm256_extracti128_si256(mins_and_scales, 0);+ let scales = mm256_set_m128i(sc128, sc128);++ let hbits = _mm256_loadu_si256(x.qh.as_ptr() as *const __m256i);+ let mut hmask = mone;++ let mut sumi = _mm256_setzero_si256();++ for j in 0..QK_K / 64 {+ let scale_0 = _mm256_shuffle_epi8(scales, get_scale_shuffle_k4(2 * j));+ let scale_1 = _mm256_shuffle_epi8(scales, get_scale_shuffle_k4(2 * j + 1));++ let q5bits = _mm256_loadu_si256(q5 as *const __m256i);+ q5 = q5.add(32);++ //Similar to q3k we hardcode the shifts here to avoid loading them into a separate register+ let q5l_0 = _mm256_and_si256(q5bits, m4);+ let q5l_0_shift_input = _mm256_and_si256(hbits, hmask);+ let q5l_0_right_shift = match j {+ 0 => _mm256_srli_epi16(q5l_0_shift_input, 0),+ 1 => _mm256_srli_epi16(q5l_0_shift_input, 2),+ 2 => _mm256_srli_epi16(q5l_0_shift_input, 4),+ 3 => _mm256_srli_epi16(q5l_0_shift_input, 6),+ _ => unreachable!(),+ };+ let q5h_0 = _mm256_slli_epi16(q5l_0_right_shift, 4);+ let q5_0 = _mm256_add_epi8(q5l_0, q5h_0);+ hmask = _mm256_slli_epi16(hmask, 1);++ let q5l_1 = _mm256_and_si256(_mm256_srli_epi16(q5bits, 4), m4);+ let q5l_1_shift_input = _mm256_and_si256(hbits, hmask);+ let q5l_1_right_shift = match j {+ 0 => _mm256_srli_epi16(q5l_1_shift_input, 1),+ 1 => _mm256_srli_epi16(q5l_1_shift_input, 3),+ 2 => _mm256_srli_epi16(q5l_1_shift_input, 5),+ 3 => _mm256_srli_epi16(q5l_1_shift_input, 7),+ _ => unreachable!(),+ };++ let q5h_1 = _mm256_slli_epi16(q5l_1_right_shift, 4);+ let q5_1 = _mm256_add_epi8(q5l_1, q5h_1);+ hmask = _mm256_slli_epi16(hmask, 1);++ let q8_0 = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);+ let q8_1 = _mm256_loadu_si256(q8 as *const __m256i);+ q8 = q8.add(32);++ let p16_0 = _mm256_maddubs_epi16(q5_0, q8_0);+ let p16_1 = _mm256_maddubs_epi16(q5_1, q8_1);++ let p16_0 = _mm256_madd_epi16(scale_0, p16_0);+ let p16_1 = _mm256_madd_epi16(scale_1, p16_1);++ sumi = _mm256_add_epi32(sumi, _mm256_add_epi32(p16_0, p16_1));+ }+ let vd = _mm256_set1_ps(d);+ acc = _mm256_fmadd_ps(vd, _mm256_cvtepi32_ps(sumi), acc);+ }+ hsum_float_8(acc) + summs+ }+}++#[target_feature(enable = "avx2,fma")]+pub(crate) fn vec_dot_q8k_q8k(n: usize, xs: &[BlockQ8K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q8k_8k: {n} is not divisible by {QK_K}"+ );+ unsafe {+ let mut acc = _mm256_setzero_ps();+ for (xs, ys) in xs.iter().zip(ys.iter()) {+ let mut sumi = _mm256_setzero_si256();+ let x_qs = xs.qs.as_ptr();+ let y_qs = ys.qs.as_ptr();+ for j in (0..QK_K).step_by(32) {+ let xs = _mm256_loadu_si256(x_qs.add(j) as *const __m256i);+ let ys = _mm256_loadu_si256(y_qs.add(j) as *const __m256i);++ let xs0 = _mm256_cvtepi8_epi16(_mm256_extracti128_si256(xs, 0));+ let ys0 = _mm256_cvtepi8_epi16(_mm256_extracti128_si256(ys, 0));+ sumi = _mm256_add_epi32(sumi, _mm256_madd_epi16(xs0, ys0));++ let xs1 = _mm256_cvtepi8_epi16(_mm256_extracti128_si256(xs, 1));+ let ys1 = _mm256_cvtepi8_epi16(_mm256_extracti128_si256(ys, 1));+ sumi = _mm256_add_epi32(sumi, _mm256_madd_epi16(xs1, ys1));+ }+ let d = _mm256_set1_ps(xs.d * ys.d);+ acc = _mm256_fmadd_ps(d, _mm256_cvtepi32_ps(sumi), acc);+ }+ hsum_float_8(acc)+ }+}diff --git a/vendor/candle-core/src/quantized/cuda.rs b/vendor/candle-core/src/quantized/cuda.rsnew file mode 100644index 0000000..bd8efc3--- /dev/null+++ b/vendor/candle-core/src/quantized/cuda.rs@@ -0,0 +1,1130 @@+use super::{GgmlDType, QStorage};+use crate::quantized::k_quants::GgmlType;+use crate::{backend::BackendDevice, cuda_backend::WrapErr};+use crate::{builder_arg as barg, CudaDevice, CudaStorage, Result};+use half::f16;++use cudarc::driver::{CudaSlice, CudaStream, CudaView, DevicePtr, PushKernelArg, SyncOnDrop};++#[derive(Clone, Debug)]+struct PaddedCudaSlice {+ inner: CudaSlice<u8>,+ len: usize,+}++#[derive(Clone, Debug)]+pub struct QCudaStorage {+ data: PaddedCudaSlice,+ dtype: GgmlDType,+ device: CudaDevice,+}++pub(crate) static FORCE_DMMV: std::sync::atomic::AtomicBool =+ std::sync::atomic::AtomicBool::new(false);++pub fn set_force_dmmv(f: bool) {+ FORCE_DMMV.store(f, std::sync::atomic::Ordering::Relaxed)+}++pub const WARP_SIZE: usize = 32;+pub const MMQ_X_Q4_0_AMPERE: usize = 4;+pub const MMQ_Y_Q4_0_AMPERE: usize = 32;+pub const NWARPS_Q4_0_AMPERE: usize = 4;+pub const GGML_CUDA_MMV_X: usize = 32;+pub const GGML_CUDA_MMV_Y: usize = 1;+pub const CUDA_QUANTIZE_BLOCK_SIZE: usize = 256;+pub const CUDA_DEQUANTIZE_BLOCK_SIZE: usize = 256;+pub const CUDA_GET_ROWS_BLOCK_SIZE: usize = 256;+pub const MATRIX_ROW_PADDING: usize = 512;++fn ceil_div(p: usize, q: usize) -> usize {+ p.div_ceil(q)+}++fn pad(p: usize, q: usize) -> usize {+ ceil_div(p, q) * q+}++fn quantize_q8_1(+ src: &CudaView<f32>,+ dst: &mut CudaSlice<u8>,+ k: usize,+ ky: usize,+ dev: &CudaDevice,+) -> Result<()> {+ let kx_padded = pad(k, MATRIX_ROW_PADDING);+ let num_blocks = ceil_div(kx_padded, CUDA_QUANTIZE_BLOCK_SIZE);++ let total_rows = ky;+ // Get Q8_1 metadata.+ let q8_1_block_size = GgmlDType::Q8_1.block_size();+ let q8_1_type_size = GgmlDType::Q8_1.type_size();++ // Calculate the size of the output buffer in bytes.+ let num_blocks_per_row = kx_padded / q8_1_block_size;+ let dst_row_size_bytes = num_blocks_per_row * q8_1_type_size;++ const CHUNK_SIZE: usize = 65535; // gridDim.y limit+ let func = dev.get_or_load_func("quantize_q8_1", &candle_kernels::QUANTIZED)?;++ let mut rows_processed = 0;+ while rows_processed < total_rows {+ // --- calculate the number of rows for this chunk ---+ let remaining_rows = total_rows - rows_processed;+ // This is our gridDim.y, now <= 65535+ let rows_in_chunk = std::cmp::min(CHUNK_SIZE, remaining_rows);++ // --- slice the source (f32) tensor by elements ---+ let src_start_elem = rows_processed * k;+ let src_num_elems = rows_in_chunk * k;+ let src_chunk = src.slice(src_start_elem..(src_start_elem + src_num_elems));++ // --- slice the destination (u8) tensor by bytes ---+ let dst_start_byte = rows_processed * dst_row_size_bytes;+ let dst_num_bytes = rows_in_chunk * dst_row_size_bytes;+ let dst_chunk = dst.slice(dst_start_byte..(dst_start_byte + dst_num_bytes));++ let cfg = cudarc::driver::LaunchConfig {+ grid_dim: (num_blocks as u32, rows_in_chunk as u32, 1),+ block_dim: (CUDA_QUANTIZE_BLOCK_SIZE as u32, 1, 1),+ shared_mem_bytes: 0,+ };++ let mut builder = func.builder();+ builder.arg(&src_chunk);+ builder.arg(&dst_chunk);+ barg!(builder, k as i32, kx_padded as i32);+ unsafe { builder.launch(cfg) }.w()?;++ rows_processed += rows_in_chunk;+ }++ Ok(())+}++fn dequantize_f32(+ data: &PaddedCudaSlice,+ dtype: GgmlDType,+ elem_count: usize,+ dev: &CudaDevice,+) -> Result<CudaStorage> {+ let nb = elem_count.div_ceil(256);+ let (kernel_name, is_k, block_dim, num_blocks) = match dtype {+ GgmlDType::Q4_0 => ("dequantize_block_q4_0_f32", false, 32, nb),+ GgmlDType::Q4_1 => ("dequantize_block_q4_1_f32", false, 32, nb),+ GgmlDType::Q5_0 => (+ "dequantize_block_q5_0_f32",+ false,+ CUDA_DEQUANTIZE_BLOCK_SIZE,+ ceil_div(elem_count, 2 * CUDA_DEQUANTIZE_BLOCK_SIZE),+ ),+ GgmlDType::Q5_1 => (+ "dequantize_block_q5_1_f32",+ false,+ CUDA_DEQUANTIZE_BLOCK_SIZE,+ ceil_div(elem_count, 2 * CUDA_DEQUANTIZE_BLOCK_SIZE),+ ),+ GgmlDType::Q8_0 => ("dequantize_block_q8_0_f32", false, 32, nb),+ GgmlDType::Q2K => ("dequantize_block_q2_K_f32", true, 64, nb),+ GgmlDType::Q3K => ("dequantize_block_q3_K_f32", true, 64, nb),+ GgmlDType::Q4K => ("dequantize_block_q4_K_f32", true, 32, nb),+ GgmlDType::Q5K => ("dequantize_block_q5_K_f32", true, 64, nb),+ GgmlDType::Q6K => ("dequantize_block_q6_K_f32", true, 64, nb),+ GgmlDType::Q8K => ("dequantize_block_q8_K_f32", true, 32, nb),+ _ => crate::bail!("unsupported dtype for dequantize {dtype:?}"),+ };+ let func = dev.get_or_load_func(kernel_name, &candle_kernels::QUANTIZED)?;+ let dst = unsafe { dev.alloc::<f32>(elem_count)? };+ // See e.g.+ // https://github.com/ggerganov/llama.cpp/blob/cbbd1efa06f8c09f9dff58ff9d9af509cc4c152b/ggml-cuda.cu#L7270+ let cfg = cudarc::driver::LaunchConfig {+ grid_dim: (num_blocks as u32, 1, 1),+ block_dim: (block_dim as u32, 1, 1),+ shared_mem_bytes: 0,+ };++ if is_k {+ let mut builder = func.builder();+ builder.arg(&data.inner);+ builder.arg(&dst);+ unsafe { builder.launch(cfg) }.w()?;+ } else {+ let nb32 = match dtype {+ GgmlDType::Q5_0 | GgmlDType::Q5_1 => elem_count,+ _ => elem_count / 32,+ };+ let mut builder = func.builder();+ builder.arg(&data.inner);+ builder.arg(&dst);+ barg!(builder, nb32 as i32);+ unsafe { builder.launch(cfg) }.w()?;+ }+ Ok(CudaStorage::wrap_cuda_slice(dst, dev.clone()))+}++fn dequantize_f16(+ data: &PaddedCudaSlice,+ dtype: GgmlDType,+ elem_count: usize,+ dev: &CudaDevice,+) -> Result<CudaStorage> {+ let nb = elem_count.div_ceil(256);+ let (kernel_name, is_k, block_dim, num_blocks) = match dtype {+ GgmlDType::Q4_0 => ("dequantize_block_q4_0_f16", false, 32, nb),+ GgmlDType::Q4_1 => ("dequantize_block_q4_1_f16", false, 32, nb),+ GgmlDType::Q5_0 => (+ "dequantize_block_q5_0_f16",+ false,+ CUDA_DEQUANTIZE_BLOCK_SIZE,+ ceil_div(elem_count, 2 * CUDA_DEQUANTIZE_BLOCK_SIZE),+ ),+ GgmlDType::Q5_1 => (+ "dequantize_block_q5_1_f16",+ false,+ CUDA_DEQUANTIZE_BLOCK_SIZE,+ ceil_div(elem_count, 2 * CUDA_DEQUANTIZE_BLOCK_SIZE),+ ),+ GgmlDType::Q8_0 => ("dequantize_block_q8_0_f16", false, 32, nb),+ GgmlDType::Q2K => ("dequantize_block_q2_K_f16", true, 64, nb),+ GgmlDType::Q3K => ("dequantize_block_q3_K_f16", true, 64, nb),+ GgmlDType::Q4K => ("dequantize_block_q4_K_f16", true, 32, nb),+ GgmlDType::Q5K => ("dequantize_block_q5_K_f16", true, 64, nb),+ GgmlDType::Q6K => ("dequantize_block_q6_K_f16", true, 64, nb),+ GgmlDType::Q8K => ("dequantize_block_q8_K_f16", true, 32, nb),+ _ => crate::bail!("unsupported dtype for dequantize {dtype:?}"),+ };+ let func = dev.get_or_load_func(kernel_name, &candle_kernels::QUANTIZED)?;+ let dst = unsafe { dev.alloc::<f16>(elem_count)? };+ // See e.g.+ // https://github.com/ggerganov/llama.cpp/blob/cbbd1efa06f8c09f9dff58ff9d9af509cc4c152b/ggml-cuda.cu#L7270+ let cfg = cudarc::driver::LaunchConfig {+ grid_dim: (num_blocks as u32, 1, 1),+ block_dim: (block_dim as u32, 1, 1),+ shared_mem_bytes: 0,+ };++ if is_k {+ let mut builder = func.builder();+ builder.arg(&data.inner);+ builder.arg(&dst);+ unsafe { builder.launch(cfg) }.w()?;+ } else {+ let nb32 = match dtype {+ GgmlDType::Q5_0 | GgmlDType::Q5_1 => elem_count,+ _ => elem_count / 32,+ };+ let mut builder = func.builder();+ builder.arg(&data.inner);+ builder.arg(&dst);+ barg!(builder, nb32 as i32);+ unsafe { builder.launch(cfg) }.w()?;+ }+ Ok(CudaStorage::wrap_cuda_slice(dst, dev.clone()))+}++fn get_rows(+ data: &PaddedCudaSlice,+ dtype: GgmlDType,+ hidden: usize,+ ids: &CudaView<u32>,+ dev: &CudaDevice,+) -> Result<CudaStorage> {+ let (kernel_name, block_dim, block_num_y, can_stride_y) = match dtype {+ GgmlDType::F32 => (+ "get_rows_f32",+ CUDA_GET_ROWS_BLOCK_SIZE,+ ceil_div(hidden, CUDA_GET_ROWS_BLOCK_SIZE),+ true,+ ),+ GgmlDType::F16 => (+ "get_rows_f16",+ CUDA_GET_ROWS_BLOCK_SIZE,+ ceil_div(hidden, CUDA_GET_ROWS_BLOCK_SIZE),+ true,+ ),+ GgmlDType::BF16 => (+ "get_rows_bf16",+ CUDA_GET_ROWS_BLOCK_SIZE,+ ceil_div(hidden, CUDA_GET_ROWS_BLOCK_SIZE),+ true,+ ),+ GgmlDType::Q4_0 => (+ "get_rows_q4_0",+ CUDA_GET_ROWS_BLOCK_SIZE,+ ceil_div(hidden, 2 * CUDA_GET_ROWS_BLOCK_SIZE),+ true,+ ),+ GgmlDType::Q4_1 => (+ "get_rows_q4_1",+ CUDA_GET_ROWS_BLOCK_SIZE,+ ceil_div(hidden, 2 * CUDA_GET_ROWS_BLOCK_SIZE),+ true,+ ),+ GgmlDType::Q5_0 => (+ "get_rows_q5_0",+ CUDA_GET_ROWS_BLOCK_SIZE,+ ceil_div(hidden, 2 * CUDA_GET_ROWS_BLOCK_SIZE),+ true,+ ),+ GgmlDType::Q5_1 => (+ "get_rows_q5_1",+ CUDA_GET_ROWS_BLOCK_SIZE,+ ceil_div(hidden, 2 * CUDA_GET_ROWS_BLOCK_SIZE),+ true,+ ),+ GgmlDType::Q8_0 => (+ "get_rows_q8_0",+ CUDA_GET_ROWS_BLOCK_SIZE,+ ceil_div(hidden, 2 * CUDA_GET_ROWS_BLOCK_SIZE),+ true,+ ),+ GgmlDType::Q2K => ("get_rows_q2_K", 64, hidden / dtype.block_size(), false),+ GgmlDType::Q3K => ("get_rows_q3_K", 64, hidden / dtype.block_size(), false),+ GgmlDType::Q4K => ("get_rows_q4_K", 32, hidden / dtype.block_size(), false),+ GgmlDType::Q5K => ("get_rows_q5_K", 64, hidden / dtype.block_size(), false),+ GgmlDType::Q6K => ("get_rows_q6_K", 64, hidden / dtype.block_size(), false),+ _ => crate::bail!("unsupported dtype for CUDA quantized embedding {dtype:?}"),+ };+ let func = dev.get_or_load_func(kernel_name, &candle_kernels::QUANTIZED)?;+ let ids_len = ids.len();+ let dst = unsafe { dev.alloc::<f32>(ids_len * hidden)? };+ if ids_len == 0 {+ return Ok(CudaStorage::wrap_cuda_slice(dst, dev.clone()));+ }+ if !can_stride_y && block_num_y > u16::MAX as usize {+ crate::bail!("quantized embedding hidden size {hidden} exceeds CUDA grid y limit")+ }+ let grid_y = if can_stride_y {+ block_num_y.min(u16::MAX as usize)+ } else {+ block_num_y+ };+ let cfg = cudarc::driver::LaunchConfig {+ grid_dim: (ids_len as u32, grid_y as u32, 1),+ block_dim: (block_dim as u32, 1, 1),+ shared_mem_bytes: 0,+ };+ let row_stride = hidden * dtype.type_size() / dtype.block_size();++ let mut builder = func.builder();+ builder.arg(&data.inner);+ builder.arg(ids);+ builder.arg(&dst);+ barg!(builder, hidden as i64, row_stride);+ unsafe { builder.launch(cfg) }.w()?;+ Ok(CudaStorage::wrap_cuda_slice(dst, dev.clone()))+}++fn dequantize_mul_mat_vec(+ data: &PaddedCudaSlice,+ y: &CudaView<f32>,+ dtype: GgmlDType,+ ncols: usize,+ nrows: usize,+ dev: &CudaDevice,+) -> Result<CudaStorage> {+ let data_elems = data.len / dtype.type_size() * dtype.block_size();+ if data_elems < ncols * nrows {+ crate::bail!("unexpected data size {}, ncols {ncols} {nrows}", data_elems)+ }+ if y.len() != ncols {+ crate::bail!("unexpected y size {}, ncols {ncols} {nrows}", y.len())+ }+ let kernel_name = match dtype {+ GgmlDType::Q4_0 => "dequantize_mul_mat_vec_q4_0_cuda",+ GgmlDType::Q4_1 => "dequantize_mul_mat_vec_q4_1_cuda",+ GgmlDType::Q5_0 => "dequantize_mul_mat_vec_q5_0_cuda",+ GgmlDType::Q5_1 => "dequantize_mul_mat_vec_q5_1_cuda",+ GgmlDType::Q8_0 => "dequantize_mul_mat_vec_q8_0_cuda",+ GgmlDType::Q2K => "dequantize_mul_mat_vec_q2_k",+ GgmlDType::Q3K => "dequantize_mul_mat_vec_q3_k",+ GgmlDType::Q4K => "dequantize_mul_mat_vec_q4_k",+ GgmlDType::Q5K => "dequantize_mul_mat_vec_q5_k",+ GgmlDType::Q6K => "dequantize_mul_mat_vec_q6_k",+ _ => crate::bail!("unsupported dtype for quantized matmul {dtype:?}"),+ };+ let func = dev.get_or_load_func(kernel_name, &candle_kernels::QUANTIZED)?;+ let dst = unsafe { dev.alloc::<f32>(nrows)? };+ let block_num_y = ceil_div(nrows, GGML_CUDA_MMV_Y);+ let cfg = cudarc::driver::LaunchConfig {+ grid_dim: (block_num_y as u32, 1, 1),+ block_dim: (WARP_SIZE as u32, GGML_CUDA_MMV_Y as u32, 1),+ shared_mem_bytes: 0,+ };++ let mut builder = func.builder();+ builder.arg(&data.inner);+ builder.arg(y);+ builder.arg(&dst);+ barg!(builder, ncols as i32, nrows as i32);+ unsafe { builder.launch(cfg) }.w()?;+ Ok(CudaStorage::wrap_cuda_slice(dst, dev.clone()))+}++fn mul_mat_vec_via_q8_1(+ data: &PaddedCudaSlice,+ y: &CudaView<f32>,+ dtype: GgmlDType,+ ncols: usize,+ nrows: usize,+ b_size: usize,+ dev: &CudaDevice,+) -> Result<CudaStorage> {+ let data_elems = data.len / dtype.type_size() * dtype.block_size();+ if data_elems < ncols * nrows {+ crate::bail!("unexpected data size {}, ncols {ncols} {nrows}", data_elems)+ }+ if y.len() != ncols * b_size {+ crate::bail!("unexpected y size {}, ncols {ncols} {nrows}", y.len())+ }+ if b_size == 0 || b_size > 8 {+ crate::bail!("only bsize between 1 and 8 are supported, got {b_size}")+ }+ // Start by quantizing y+ let ncols_padded = pad(ncols, MATRIX_ROW_PADDING);+ let y_size_in_bytes =+ b_size * ncols_padded * GgmlDType::Q8_1.type_size() / GgmlDType::Q8_1.block_size();+ let mut y_q8_1 = dev.alloc_zeros::<u8>(y_size_in_bytes)?;+ quantize_q8_1(y, &mut y_q8_1, ncols, b_size, dev)?;++ let kernel_name = match dtype {+ GgmlDType::Q4_0 => "mul_mat_vec_q4_0_q8_1_cuda",+ GgmlDType::Q4_1 => "mul_mat_vec_q4_1_q8_1_cuda",+ GgmlDType::Q5_0 => "mul_mat_vec_q5_0_q8_1_cuda",+ GgmlDType::Q5_1 => "mul_mat_vec_q5_1_q8_1_cuda",+ GgmlDType::Q8_0 => "mul_mat_vec_q8_0_q8_1_cuda",+ GgmlDType::Q2K => "mul_mat_vec_q2_K_q8_1_cuda",+ GgmlDType::Q3K => "mul_mat_vec_q3_K_q8_1_cuda",+ GgmlDType::Q4K => "mul_mat_vec_q4_K_q8_1_cuda",+ GgmlDType::Q5K => "mul_mat_vec_q5_K_q8_1_cuda",+ GgmlDType::Q6K => "mul_mat_vec_q6_K_q8_1_cuda",+ _ => crate::bail!("unsupported dtype for quantized matmul {dtype:?}"),+ };+ let kernel_name = format!("{kernel_name}{b_size}");+ let func = dev.get_or_load_func(&kernel_name, &candle_kernels::QUANTIZED)?;+ let dst = dev.alloc_zeros::<f32>(nrows * b_size)?;+ // https://github.com/ggerganov/llama.cpp/blob/facb8b56f8fd3bb10a693bf0943ae9d69d0828ef/ggml-cuda/mmvq.cu#L98+ let (nblocks, nwarps) = match b_size {+ 1 => (nrows as u32, 4),+ 2..=4 => ((nrows as u32).div_ceil(2), 4),+ 5..=8 => ((nrows as u32).div_ceil(2), 2),+ _ => crate::bail!("unexpected bsize {b_size}"),+ };+ let cfg = cudarc::driver::LaunchConfig {+ grid_dim: (nblocks, 1, 1),+ block_dim: (WARP_SIZE as u32, nwarps, 1),+ shared_mem_bytes: 0,+ };++ let mut builder = func.builder();+ builder.arg(&data.inner);+ builder.arg(&y_q8_1);+ builder.arg(&dst);+ barg!(+ builder,+ /* ncols_x */ ncols as i32,+ /* nrows_x */ nrows as i32,+ /* nrows_y */ ncols_padded as i32,+ /* nrows_dst */ nrows as i32+ );+ unsafe { builder.launch(cfg) }.w()?;+ Ok(CudaStorage::wrap_cuda_slice(dst, dev.clone()))+}++#[allow(clippy::too_many_arguments)]+fn mul_mat_via_q8_1(+ data: &PaddedCudaSlice,+ y: &CudaView<f32>,+ dtype: GgmlDType,+ x_rows: usize,+ x_cols: usize,+ y_rows: usize,+ y_cols: usize,+ dev: &CudaDevice,+) -> Result<CudaStorage> {+ let data_elems = data.len / dtype.type_size() * dtype.block_size();+ if data_elems < x_rows * x_cols {+ crate::bail!("unexpected lhs size {}, {x_rows} {x_cols}", data_elems)+ }+ if y.len() != y_rows * y_cols {+ crate::bail!("unexpected y size {}, {y_rows} {y_cols}", y.len())+ }+ if x_cols != y_rows {+ crate::bail!("unexpected x/y size {x_rows} {x_cols} {y_rows} {y_cols}")+ }+ let k = x_cols;+ // Start by quantizing y+ let k_padded = pad(k, MATRIX_ROW_PADDING);+ let y_size_in_bytes =+ k_padded * y_cols * GgmlDType::Q8_1.type_size() / GgmlDType::Q8_1.block_size();+ let mut y_q8_1 = dev.alloc_zeros::<u8>(y_size_in_bytes)?;+ quantize_q8_1(y, &mut y_q8_1, k, y_cols, dev)?;++ let (kernel_name, mmq_x, mmq_y) = match dtype {+ GgmlDType::Q4_0 => ("mul_mat_q4_0", 64, 128),+ GgmlDType::Q4_1 => ("mul_mat_q4_1", 64, 128),+ GgmlDType::Q5_0 => ("mul_mat_q5_0", 128, 64),+ GgmlDType::Q5_1 => ("mul_mat_q5_1", 128, 64),+ GgmlDType::Q8_0 => ("mul_mat_q8_0", 128, 64),+ GgmlDType::Q2K => ("mul_mat_q2_K", 64, 128),+ GgmlDType::Q3K => ("mul_mat_q3_K", 128, 128),+ GgmlDType::Q4K => ("mul_mat_q4_K", 64, 128),+ GgmlDType::Q5K => ("mul_mat_q5_K", 64, 128),+ GgmlDType::Q6K => ("mul_mat_q6_K", 64, 64),+ _ => crate::bail!("unsupported dtype for quantized matmul {dtype:?}"),+ };+ let func = dev.get_or_load_func(kernel_name, &candle_kernels::QUANTIZED)?;+ let dst = dev.alloc_zeros::<f32>(x_rows * y_cols)?;+ let cfg = cudarc::driver::LaunchConfig {+ grid_dim: (+ ceil_div(x_rows, mmq_y) as u32,+ ceil_div(y_cols, mmq_x) as u32,+ 1,+ ),+ block_dim: (WARP_SIZE as u32, 4, 1),+ shared_mem_bytes: 0,+ };++ let mut builder = func.builder();+ builder.arg(/* vx */ &data.inner);+ builder.arg(/* vy */ &y_q8_1);+ builder.arg(/* dst */ &dst);+ barg!(+ builder,+ /* ncols_x */ x_cols as i32,+ /* nrows_x */ x_rows as i32,+ /* ncols_y */ y_cols as i32,+ /* nrows_y */ k_padded as i32,+ /* nrows_dst */ x_rows as i32+ );+ unsafe { builder.launch(cfg) }.w()?;+ Ok(CudaStorage::wrap_cuda_slice(dst, dev.clone()))+}++#[allow(clippy::too_many_arguments)]+fn indexed_moe_forward_fused_q8_1_input(+ weight: &CudaView<u8>,+ w_shape: &crate::Shape, //[num_experts, n, k]+ w_dtype: GgmlDType,+ input: &CudaSlice<f32>,+ in_shape: &crate::Shape, //[batch, topk or 1, k]+ ids: &CudaView<u32>,+ idx_shape: &crate::Shape, //[batch, topk]+ dev: &CudaDevice,+) -> Result<(CudaStorage, crate::Shape)> {+ let (_, n, k) = w_shape.dims3()?;+ let batch = in_shape.dims()[0];+ let input_dim1 = in_shape.dims()[1];++ let topk = idx_shape.dims()[1];+ assert!(batch == idx_shape.dims()[0], "batch dim not match!");++ // Quantize input into q8_1.+ let total_rows = batch * input_dim1;+ let k_padded = pad(k, MATRIX_ROW_PADDING);+ // Get Q8_1 metadata.+ let q8_1_block_size = GgmlDType::Q8_1.block_size();+ let q8_1_type_size = GgmlDType::Q8_1.type_size();++ // Calculate the size of the output buffer in bytes.+ let num_blocks_per_row = k_padded / q8_1_block_size;+ let dst_row_size_bytes = num_blocks_per_row * q8_1_type_size;+ let y_size_in_bytes = total_rows * dst_row_size_bytes;+ let mut input_quant = dev.alloc_zeros::<u8>(y_size_in_bytes)?;++ let input_view = input.slice(0..);+ quantize_q8_1(&input_view, &mut input_quant, k, total_rows, dev)?;++ // output buffer+ let outsize = batch * topk * n;+ let out = dev.alloc_zeros::<f32>(outsize)?;++ let kernel_name = match w_dtype {+ GgmlDType::Q2K => "indexed_moe_forward_q2k_q8_1",+ GgmlDType::Q3K => "indexed_moe_forward_q3k_q8_1",+ GgmlDType::Q4K => "indexed_moe_forward_q4k_q8_1",+ GgmlDType::Q5K => "indexed_moe_forward_q5k_q8_1",+ GgmlDType::Q6K => "indexed_moe_forward_q6k_q8_1",+ GgmlDType::Q8_0 => "indexed_moe_forward_q8_0_q8_1",+ _ => crate::bail!("unsupported dtype for indexed_moe_forward {w_dtype:?}"),+ };+ let func = dev.get_or_load_func(kernel_name, &candle_kernels::QUANTIZED)?;+ let (nblocks, nwarps) = (n as u32, 4);+ let cfg = cudarc::driver::LaunchConfig {+ grid_dim: (nblocks, batch as u32, topk as u32),+ block_dim: (WARP_SIZE as u32, nwarps, 1),+ shared_mem_bytes: 0,+ };++ let mut builder = func.builder();+ builder.arg(weight);+ builder.arg(&input_quant);+ builder.arg(ids);+ builder.arg(&out);++ barg!(+ builder,+ n as i32,+ k as i32,+ batch as i32,+ topk as i32,+ k_padded as i32,+ input_dim1 as i32+ );+ unsafe { builder.launch(cfg) }.w()?;++ let mut out_shape = in_shape.dims().to_vec();+ out_shape.pop();+ out_shape.push(n);+ out_shape[1] = topk;+ Ok((+ CudaStorage::wrap_cuda_slice(out, dev.clone()),+ out_shape.into(),+ ))+}++impl QCudaStorage {+ pub fn indexed_moe_forward(+ &self,+ self_shape: &crate::Shape, //[num_experts, n, k]+ input: &CudaStorage, //[batch, topk or 1, k]+ input_l: &crate::Layout,+ ids: &CudaStorage, //[batch, topk]+ ids_l: &crate::Layout,+ ) -> Result<(CudaStorage, crate::Shape)> {+ if matches!(+ self.dtype(),+ GgmlDType::Q8_0+ | GgmlDType::Q2K+ | GgmlDType::Q3K+ | GgmlDType::Q4K+ | GgmlDType::Q5K+ | GgmlDType::Q6K+ ) {+ let input_storage = input.as_cuda_slice::<f32>()?;+ let ids_storage = ids.as_cuda_slice::<u32>()?;+ indexed_moe_forward_fused_q8_1_input(+ &self.data.inner.slice(0..),+ self_shape, //[num_experts, n, k]+ self.dtype(),+ input_storage,+ input_l.shape(), //[batch, topk or 1, k]+ &ids_storage.slice(0..),+ ids_l.shape(), //[batch, topk]+ &self.device,+ )+ } else {+ crate::bail!(+ "The given quantized dtype {:?} is not supported for indexed_moe_forward!",+ self.dtype()+ );+ }+ }++ pub fn zeros(device: &CudaDevice, el_count: usize, dtype: GgmlDType) -> Result<Self> {+ let size_in_bytes = ceil_div(el_count, dtype.block_size()) * dtype.type_size();+ let padded_size_in_bytes =+ ceil_div(el_count + MATRIX_ROW_PADDING, dtype.block_size()) * dtype.type_size();+ let inner = device.alloc_zeros::<u8>(padded_size_in_bytes)?;+ Ok(QCudaStorage {+ data: PaddedCudaSlice {+ inner,+ len: size_in_bytes,+ },+ device: device.clone(),+ dtype,+ })+ }++ pub fn dtype(&self) -> GgmlDType {+ self.dtype+ }++ pub fn device(&self) -> &CudaDevice {+ &self.device+ }++ pub fn dequantize(&self, elem_count: usize) -> Result<CudaStorage> {+ fn deq<T: GgmlType>(buffer: &[u8], n: usize, dst: &mut [f32]) {+ let slice = unsafe { std::slice::from_raw_parts(buffer.as_ptr() as *const T, n) };+ let vec = slice.to_vec();+ T::to_float(&vec, dst)+ }++ let fast_kernel = matches!(+ self.dtype,+ GgmlDType::Q4_0+ | GgmlDType::Q4_1+ | GgmlDType::Q5_0+ | GgmlDType::Q5_1+ | GgmlDType::Q8_0+ | GgmlDType::Q2K+ | GgmlDType::Q3K+ | GgmlDType::Q4K+ | GgmlDType::Q5K+ | GgmlDType::Q6K+ | GgmlDType::Q8K+ );+ if fast_kernel {+ return dequantize_f32(&self.data, self.dtype, elem_count, self.device());+ }+ // Run the dequantization on cpu.++ let buffer = self+ .device+ .clone_dtoh(&self.data.inner.slice(..self.data.len))?;+ let mut out = vec![0.0; elem_count];+ let block_len = elem_count / self.dtype.block_size();+ match self.dtype {+ GgmlDType::F32 => deq::<f32>(&buffer, block_len, &mut out),+ GgmlDType::F16 => deq::<half::f16>(&buffer, block_len, &mut out),+ GgmlDType::BF16 => deq::<half::bf16>(&buffer, block_len, &mut out),+ GgmlDType::Q4_0 => deq::<crate::quantized::BlockQ4_0>(&buffer, block_len, &mut out),+ GgmlDType::Q4_1 => deq::<crate::quantized::BlockQ4_1>(&buffer, block_len, &mut out),+ GgmlDType::Q5_0 => deq::<crate::quantized::BlockQ5_0>(&buffer, block_len, &mut out),+ GgmlDType::Q5_1 => deq::<crate::quantized::BlockQ5_1>(&buffer, block_len, &mut out),+ GgmlDType::Q8_0 => deq::<crate::quantized::BlockQ8_0>(&buffer, block_len, &mut out),+ GgmlDType::Q8_1 => deq::<crate::quantized::BlockQ8_1>(&buffer, block_len, &mut out),+ GgmlDType::Q2K => deq::<crate::quantized::BlockQ2K>(&buffer, block_len, &mut out),+ GgmlDType::Q3K => deq::<crate::quantized::BlockQ3K>(&buffer, block_len, &mut out),+ GgmlDType::Q4K => deq::<crate::quantized::BlockQ4K>(&buffer, block_len, &mut out),+ GgmlDType::Q5K => deq::<crate::quantized::BlockQ5K>(&buffer, block_len, &mut out),+ GgmlDType::Q6K => deq::<crate::quantized::BlockQ6K>(&buffer, block_len, &mut out),+ GgmlDType::Q8K => deq::<crate::quantized::BlockQ8K>(&buffer, block_len, &mut out),+ }++ self.device+ .storage_from_cpu_storage(&crate::CpuStorage::F32(out))+ }++ pub fn dequantize_f16(&self, elem_count: usize) -> Result<CudaStorage> {+ dequantize_f16(&self.data, self.dtype, elem_count, self.device())+ }++ pub fn quantize(&mut self, src: &CudaStorage) -> Result<()> {+ // Run the quantization on cpu.+ let src = match &src.slice {+ crate::cuda_backend::CudaStorageSlice::F32(data) => self.device.clone_dtoh(data)?,+ _ => crate::bail!("only f32 can be quantized"),+ };+ let src_len = src.len();+ let src = crate::Storage::Cpu(crate::CpuStorage::F32(src));+ let mut qcpu_storage = crate::Device::Cpu.qzeros(src_len, self.dtype)?;+ qcpu_storage.quantize(&src)?;+ let data = qcpu_storage.data()?;+ let padded_len =+ data.len() + MATRIX_ROW_PADDING * self.dtype.type_size() / self.dtype.block_size();+ let mut inner = unsafe { self.device.alloc::<u8>(padded_len)? };+ self.device+ .memcpy_htod(&*data, &mut inner.slice_mut(..data.len()))?;+ self.data = PaddedCudaSlice {+ inner,+ len: data.len(),+ };+ Ok(())+ }++ pub fn quantize_imatrix(+ &mut self,+ src: &CudaStorage,+ imatrix_weights: &[f32],+ n_per_row: usize,+ ) -> Result<()> {+ // Run the quantization on cpu.+ let src = match &src.slice {+ crate::cuda_backend::CudaStorageSlice::F32(data) => self.device.clone_dtoh(data)?,+ _ => crate::bail!("only f32 can be quantized"),+ };+ let src_len = src.len();+ let src = crate::Storage::Cpu(crate::CpuStorage::F32(src));+ let mut qcpu_storage = crate::Device::Cpu.qzeros(src_len, self.dtype)?;+ qcpu_storage.quantize_imatrix(&src, imatrix_weights, n_per_row)?;+ let data = qcpu_storage.data()?;+ let padded_len =+ data.len() + MATRIX_ROW_PADDING * self.dtype.type_size() / self.dtype.block_size();+ let mut inner = unsafe { self.device.alloc::<u8>(padded_len)? };+ self.device+ .memcpy_htod(&*data, &mut inner.slice_mut(..data.len()))?;+ self.data = PaddedCudaSlice {+ inner,+ len: data.len(),+ };+ Ok(())+ }++ pub fn quantize_imatrix_onto(+ &mut self,+ src: &crate::CpuStorage,+ imatrix_weights: &[f32],+ n_per_row: usize,+ ) -> Result<()> {+ // Run the quantization on cpu.+ let src_len = src.as_slice::<f32>()?.len();+ let mut qcpu_storage = crate::Device::Cpu.qzeros(src_len, self.dtype)?;++ if let QStorage::Cpu(storage) = &mut qcpu_storage {+ storage.from_float_imatrix(src.as_slice::<f32>()?, imatrix_weights, n_per_row);+ } else {+ unreachable!()+ }++ let data = qcpu_storage.data()?;+ let padded_len =+ data.len() + MATRIX_ROW_PADDING * self.dtype.type_size() / self.dtype.block_size();+ let mut inner = unsafe { self.device.alloc::<u8>(padded_len)? };+ self.device+ .memcpy_htod(&*data, &mut inner.slice_mut(..data.len()))?;+ self.data = PaddedCudaSlice {+ inner,+ len: data.len(),+ };+ Ok(())+ }++ pub fn quantize_onto(&mut self, src: &crate::CpuStorage) -> Result<()> {+ // Run the quantization on cpu.+ let src_len = src.as_slice::<f32>()?.len();+ let mut qcpu_storage = crate::Device::Cpu.qzeros(src_len, self.dtype)?;++ if let QStorage::Cpu(storage) = &mut qcpu_storage {+ storage.from_float(src.as_slice::<f32>()?);+ } else {+ unreachable!()+ }++ let data = qcpu_storage.data()?;+ let padded_len =+ data.len() + MATRIX_ROW_PADDING * self.dtype.type_size() / self.dtype.block_size();+ let mut inner = unsafe { self.device.alloc::<u8>(padded_len)? };+ self.device+ .memcpy_htod(&*data, &mut inner.slice_mut(..data.len()))?;+ self.data = PaddedCudaSlice {+ inner,+ len: data.len(),+ };+ Ok(())+ }++ pub fn storage_size_in_bytes(&self) -> usize {+ self.data.len+ }++ pub fn embedding(+ &self,+ rows: usize,+ hidden: usize,+ ids: &CudaStorage,+ ids_l: &crate::Layout,+ ) -> Result<CudaStorage> {+ if !ids_l.is_contiguous() {+ crate::bail!("quantized embedding requires contiguous ids")+ }+ if !hidden.is_multiple_of(self.dtype.block_size()) {+ crate::bail!(+ "quantized embedding hidden size {hidden} is not divisible by block size {}",+ self.dtype.block_size()+ )+ }+ let expected_size = rows * hidden * self.dtype.type_size() / self.dtype.block_size();+ if self.storage_size_in_bytes() != expected_size {+ crate::bail!(+ "quantized tensor has {} bytes, expected {expected_size}",+ self.storage_size_in_bytes()+ )+ }+ let ids = ids.as_cuda_slice::<u32>()?;+ let ids = match ids_l.contiguous_offsets() {+ Some((o1, o2)) => ids.slice(o1..o2),+ None => Err(crate::Error::RequiresContiguous {+ op: "quantized-embedding",+ }+ .bt())?,+ };+ get_rows(&self.data, self.dtype, hidden, &ids, self.device())+ }++ pub fn fwd(+ &self,+ self_shape: &crate::Shape,+ storage: &CudaStorage,+ layout: &crate::Layout,+ ) -> Result<(CudaStorage, crate::Shape)> {+ // Optimized MMVQ and MMQ paths (support most paths: BF16/F16/F32, batch 1-8, all quant types, reuses per-device workspace).+ if !FORCE_DMMV.load(std::sync::atomic::Ordering::Relaxed) {+ if let Some(result) = super::fast_mmvq::try_fwd(self, self_shape, storage, layout)? {+ return Ok(result);+ }+ if let Some(result) = super::fast_mmq::try_fwd(self, self_shape, storage, layout)? {+ return Ok(result);+ }+ }++ // Fallback+ let max_bm = if FORCE_DMMV.load(std::sync::atomic::Ordering::Relaxed) {+ 1+ } else {+ 8+ };+ let use_vec_kernel = match layout.shape().dims() {+ [b, m, _k] => b * m <= max_bm,+ [b, _k] => *b <= max_bm,+ _ => false,+ };+ if use_vec_kernel {+ self.dequantize_matmul_vec(self_shape, storage, layout)+ } else {+ self.dequantize_matmul(self_shape, storage, layout)+ }+ }++ pub fn data(&self) -> Result<Vec<u8>> {+ let mut out = vec![0u8; self.data.len];+ self.device+ .memcpy_dtoh(&self.data.inner.slice(..self.data.len), &mut out)?;+ Ok(out)+ }++ pub fn device_ptr(&self) -> Result<*const u8> {+ Ok(self.data.inner.device_ptr(self.data.inner.stream()).0 as *const u8)+ }++ pub fn device_ptr_with_guard<'a>(+ &'a self,+ stream: &'a CudaStream,+ ) -> Result<(*const u8, SyncOnDrop<'a>)> {+ let (ptr, guard) = self.data.inner.device_ptr(stream);+ Ok((ptr as *const u8, guard))+ }+}++impl QCudaStorage {+ fn dequantize_matmul_vec(+ &self,+ self_shape: &crate::Shape,+ rhs: &CudaStorage,+ rhs_l: &crate::Layout,+ ) -> Result<(CudaStorage, crate::Shape)> {+ let (nrows, ncols) = self_shape.dims2()?;+ let rhs = rhs.as_cuda_slice::<f32>()?;+ let rhs = match rhs_l.contiguous_offsets() {+ Some((o1, o2)) => rhs.slice(o1..o2),+ None => Err(crate::Error::RequiresContiguous { op: "dmmv" }.bt())?,+ };+ let (b_size, k) = match rhs_l.shape().dims() {+ [b, m, k] => (b * m, *k),+ [b, k] => (*b, *k),+ _ => crate::bail!("unexpected rhs shape in dmmv {:?}", rhs_l.shape()),+ };+ if ncols != k {+ crate::bail!("mismatch on matmul dim {self_shape:?} {:?}", rhs_l.shape())+ }++ let out = if FORCE_DMMV.load(std::sync::atomic::Ordering::Relaxed) {+ dequantize_mul_mat_vec(&self.data, &rhs, self.dtype, ncols, nrows, self.device())?+ } else {+ mul_mat_vec_via_q8_1(+ &self.data,+ &rhs,+ self.dtype,+ ncols,+ nrows,+ b_size,+ self.device(),+ )?+ };+ let mut out_shape = rhs_l.shape().dims().to_vec();+ out_shape.pop();+ out_shape.push(nrows);+ Ok((out, out_shape.into()))+ }++ fn dequantize_matmul(+ &self,+ self_shape: &crate::Shape,+ storage: &CudaStorage,+ layout: &crate::Layout,+ ) -> Result<(CudaStorage, crate::Shape)> {+ use crate::backend::BackendStorage;+ let (n, k) = self_shape.dims2()?;+ let (b, m, k2) = match layout.shape().dims() {+ &[b, m, k2] => (b, m, k2),+ &[m, k2] => (1, m, k2),+ s => crate::bail!("unexpected shape for input {s:?}"),+ };+ if k2 != k {+ crate::bail!("mismatch on matmul dim {self_shape:?} {:?}", layout.shape())+ }++ let out = if FORCE_DMMV.load(std::sync::atomic::Ordering::Relaxed) {+ let data_f32 = self.dequantize(n * k)?;+ let rhs_l = crate::Layout::new((k, n).into(), vec![1, k], 0).broadcast_as((b, k, n))?;+ storage.matmul(&data_f32, (b, m, n, k), layout, &rhs_l)?+ } else {+ let storage = storage.as_cuda_slice::<f32>()?;+ let storage = match layout.contiguous_offsets() {+ Some((o1, o2)) => storage.slice(o1..o2),+ None => Err(crate::Error::RequiresContiguous {+ op: "quantized-matmul",+ }+ .bt())?,+ };+ mul_mat_via_q8_1(+ &self.data,+ &storage,+ self.dtype,+ /* x_rows */ n,+ /* x_cols */ k,+ /* y_rows */ k,+ /* y_cols */ b * m,+ self.device(),+ )?+ };+ let mut out_shape = layout.shape().dims().to_vec();+ out_shape.pop();+ out_shape.push(n);+ Ok((out, out_shape.into()))+ }+}++pub fn load_quantized<T: super::GgmlType + Send + Sync + 'static>(+ device: &CudaDevice,+ data: &[T],+) -> Result<super::QStorage> {+ let data = unsafe {+ std::slice::from_raw_parts(data.as_ptr() as *const u8, core::mem::size_of_val(data))+ };+ let dtype = T::DTYPE;+ let padded_len = data.len() + MATRIX_ROW_PADDING * dtype.type_size() / dtype.block_size();+ let mut inner = device.alloc_zeros::<u8>(padded_len)?;+ device.memcpy_htod(data, &mut inner.slice_mut(..data.len()))?;+ Ok(QStorage::Cuda(QCudaStorage {+ data: PaddedCudaSlice {+ inner,+ len: data.len(),+ },+ device: device.clone(),+ dtype,+ }))+}++#[cfg(test)]+mod test {+ use super::*;++ #[test]+ fn cuda_quantize_q8_1() -> Result<()> {+ let dev = CudaDevice::new(0)?;+ let el = 256;+ let el_padded = pad(el, MATRIX_ROW_PADDING);+ let y_size_in_bytes =+ el_padded * GgmlDType::Q8_1.type_size() / GgmlDType::Q8_1.block_size();+ let mut y_q8_1 = unsafe { dev.alloc::<u8>(y_size_in_bytes)? };+ let vs: Vec<f32> = (0..el).map(|v| v as f32).collect();+ let y = dev.clone_htod(&vs)?;+ quantize_q8_1(&y.as_view(), &mut y_q8_1, el, 1, &dev)?;+ Ok(())+ }++ #[test]+ fn cuda_mmv_q8_1() -> Result<()> {+ let dev = CudaDevice::new(0)?;+ let ncols = 256;+ let vs: Vec<f32> = (0..ncols).map(|v| v as f32).collect();+ let y = dev.clone_htod(&vs)?;+ let mut xs = QCudaStorage::zeros(&dev, ncols, GgmlDType::Q4_0)?;+ xs.quantize(&CudaStorage::wrap_cuda_slice(y.clone(), dev.clone()))?;+ let cuda_storage = mul_mat_vec_via_q8_1(+ &xs.data,+ &y.as_view(),+ /* dtype */ GgmlDType::Q4_0,+ /* ncols */ ncols,+ /* nrows */ 1,+ /* b_size */ 1,+ &dev,+ )?;+ let vs = cuda_storage.as_cuda_slice::<f32>()?;+ let vs = dev.clone_dtoh(&vs.as_view())?;+ assert_eq!(vs.len(), 1);+ // for n = 255, n.(n+1).(2n+1) / 6 = 5559680+ // Q8 means 1/256 precision.+ assert_eq!(vs[0], 5561664.5);++ let cuda_storage = dequantize_mul_mat_vec(+ &xs.data,+ &y.as_view(),+ /* dtype */ GgmlDType::Q4_0,+ /* ncols */ ncols,+ /* nrows */ 1,+ &dev,+ )?;+ let vs = cuda_storage.as_cuda_slice::<f32>()?;+ let vs = dev.clone_dtoh(&vs.as_view())?;+ assert_eq!(vs.len(), 1);+ assert_eq!(vs[0], 5561851.0);+ Ok(())+ }++ #[test]+ fn cuda_mm_q8_1() -> Result<()> {+ let dev = CudaDevice::new(0)?;+ let ncols = 256;+ let vs: Vec<f32> = (0..ncols * 4).map(|v| v as f32 / 4.).collect();+ let y = dev.clone_htod(&vs)?;+ let mut xs = QCudaStorage::zeros(&dev, ncols * 4, GgmlDType::Q4_0)?;+ xs.quantize(&CudaStorage::wrap_cuda_slice(y.clone(), dev.clone()))?;+ let cuda_storage = mul_mat_via_q8_1(+ &xs.data,+ &y.as_view(),+ /* dtype */ GgmlDType::Q4_0,+ /* x_rows */ 4,+ /* x_cols */ ncols,+ /* y_rows */ ncols,+ /* y_cols */ 4,+ &dev,+ )?;+ let vs = cuda_storage.as_cuda_slice::<f32>()?;+ let vs = dev.clone_dtoh(&vs.as_view())?;++ /*+ x = torch.tensor([float(v) for v in range(1024)]).reshape(4, 256)+ x @ x.t() / 16+ tensor([[ 347480.0000, 869720.0000, 1391960.0000, 1914200.0000],+ [ 869720.0000, 2440536.0000, 4011352.0000, 5582166.5000],+ [ 1391960.0000, 4011352.0000, 6630742.0000, 9250132.0000],+ [ 1914200.0000, 5582166.5000, 9250132.0000, 12918099.0000]])+ */+ assert_eq!(vs.len(), 16);+ assert_eq!(vs[0], 347604.0);+ assert_eq!(vs[1], 888153.06);+ assert_eq!(vs[4], 869780.7);+ assert_eq!(vs[5], 2483145.0);+ assert_eq!(vs[11], 9407368.0);+ assert_eq!(vs[14], 9470856.0);+ assert_eq!(vs[15], 13138824.0);+ Ok(())+ }++ // The following test used to fail under compute-sanitizer until #2526.+ #[test]+ fn cuda_mm_q8_1_pad() -> Result<()> {+ let dev = CudaDevice::new(0)?;+ let (x_rows, ncols, y_cols) = (4, 16, 2048);+ let vs: Vec<f32> = (0..ncols * y_cols).map(|v| v as f32 / 256.).collect();+ let y = dev.clone_htod(&vs)?;+ let mut xs = QCudaStorage::zeros(&dev, ncols * x_rows, GgmlDType::Q4_0)?;+ xs.quantize(&CudaStorage::wrap_cuda_slice(y.clone(), dev.clone()))?;+ let cuda_storage = mul_mat_via_q8_1(+ &xs.data,+ &y.as_view(),+ /* dtype */ GgmlDType::Q4_0,+ /* x_rows */ x_rows,+ /* x_cols */ ncols,+ /* y_rows */ ncols,+ /* y_cols */ y_cols,+ &dev,+ )?;+ let vs = cuda_storage.as_cuda_slice::<f32>()?;+ let _vs = dev.clone_dtoh(&vs.as_view())?;+ Ok(())+ }+}diff --git a/vendor/candle-core/src/quantized/dummy_cuda.rs b/vendor/candle-core/src/quantized/dummy_cuda.rsnew file mode 100644index 0000000..754a991--- /dev/null+++ b/vendor/candle-core/src/quantized/dummy_cuda.rs@@ -0,0 +1,105 @@+#![allow(unused)]+use super::GgmlDType;+use crate::{CudaDevice, CudaStorage, Error, Result};++pub struct QCudaStorage {+ dtype: GgmlDType,+ device: CudaDevice,+}++impl QCudaStorage {+ pub fn zeros(_: &CudaDevice, _: usize, _: GgmlDType) -> Result<Self> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ pub fn dtype(&self) -> GgmlDType {+ self.dtype+ }++ pub fn device(&self) -> &CudaDevice {+ &self.device+ }++ pub fn dequantize(&self, _elem_count: usize) -> Result<CudaStorage> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ pub fn dequantize_f16(&self, _elem_count: usize) -> Result<CudaStorage> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ pub fn quantize(&mut self, _src: &CudaStorage) -> Result<()> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ pub fn quantize_imatrix(+ &mut self,+ _src: &CudaStorage,+ _imatrix_weights: &[f32],+ _n_per_row: usize,+ ) -> Result<()> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ pub fn quantize_imatrix_onto(+ &mut self,+ _src: &crate::CpuStorage,+ _imatrix_weights: &[f32],+ _n_per_row: usize,+ ) -> Result<()> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ pub fn quantize_onto(&mut self, _src: &crate::CpuStorage) -> Result<()> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ pub fn device_ptr(&self) -> Result<*const u8> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ pub fn storage_size_in_bytes(&self) -> usize {+ 0+ }++ pub fn embedding(+ &self,+ _rows: usize,+ _hidden: usize,+ _ids: &CudaStorage,+ _ids_l: &crate::Layout,+ ) -> Result<CudaStorage> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ pub fn fwd(+ &self,+ _self_shape: &crate::Shape,+ _storage: &CudaStorage,+ _layout: &crate::Layout,+ ) -> Result<(CudaStorage, crate::Shape)> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ pub fn data(&self) -> Result<Vec<u8>> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ pub fn indexed_moe_forward(+ &self,+ _: &crate::Shape,+ _: &CudaStorage,+ _: &crate::Layout,+ _: &CudaStorage,+ _: &crate::Layout,+ ) -> Result<(CudaStorage, crate::Shape)> {+ Err(Error::NotCompiledWithCudaSupport)+ }+}++pub fn load_quantized<T: super::GgmlType + Send + Sync + 'static>(+ _device: &CudaDevice,+ _data: &[T],+) -> Result<super::QStorage> {+ Err(Error::NotCompiledWithCudaSupport)+}diff --git a/vendor/candle-core/src/quantized/dummy_metal.rs b/vendor/candle-core/src/quantized/dummy_metal.rsnew file mode 100644index 0000000..32e598e--- /dev/null+++ b/vendor/candle-core/src/quantized/dummy_metal.rs@@ -0,0 +1,97 @@+#![allow(unused)]+use super::GgmlDType;+use crate::{Error, MetalDevice, MetalStorage, Result};++pub struct QMetalStorage {+ dtype: GgmlDType,+ device: MetalDevice,+}++impl QMetalStorage {+ pub fn zeros(_: &MetalDevice, _: usize, _: GgmlDType) -> Result<Self> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ pub fn dtype(&self) -> GgmlDType {+ self.dtype+ }++ pub fn device(&self) -> &MetalDevice {+ &self.device+ }++ pub fn dequantize(&self, _elem_count: usize) -> Result<MetalStorage> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ pub fn quantize(&mut self, _src: &MetalStorage) -> Result<()> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ pub fn quantize_imatrix(+ &mut self,+ _src: &MetalStorage,+ _imatrix_weights: &[f32],+ _n_per_row: usize,+ ) -> Result<()> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ pub fn quantize_imatrix_onto(+ &mut self,+ _src: &crate::CpuStorage,+ _imatrix_weights: &[f32],+ _n_per_row: usize,+ ) -> Result<()> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ pub fn quantize_onto(&mut self, _src: &crate::CpuStorage) -> Result<()> {+ Err(Error::NotCompiledWithCudaSupport)+ }++ pub fn storage_size_in_bytes(&self) -> usize {+ 0+ }++ pub fn embedding(+ &self,+ _rows: usize,+ _hidden: usize,+ _ids: &MetalStorage,+ _ids_l: &crate::Layout,+ ) -> Result<MetalStorage> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ pub fn fwd(+ &self,+ _self_shape: &crate::Shape,+ _storage: &MetalStorage,+ _layout: &crate::Layout,+ ) -> Result<(MetalStorage, crate::Shape)> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ pub fn data(&self) -> Result<Vec<u8>> {+ Err(Error::NotCompiledWithMetalSupport)+ }++ pub fn indexed_moe_forward(+ &self,+ _: &crate::Shape,+ _: &MetalStorage,+ _: &crate::Layout,+ _: &MetalStorage,+ _: &crate::Layout,+ ) -> Result<(MetalStorage, crate::Shape)> {+ Err(Error::NotCompiledWithMetalSupport)+ }+}++pub fn load_quantized<T: super::GgmlType + Send + Sync + 'static>(+ _device: &MetalDevice,+ _data: &[T],+) -> Result<super::QStorage> {+ Err(Error::NotCompiledWithMetalSupport)+}diff --git a/vendor/candle-core/src/quantized/fast_mmq.rs b/vendor/candle-core/src/quantized/fast_mmq.rsnew file mode 100644index 0000000..0e3fd0a--- /dev/null+++ b/vendor/candle-core/src/quantized/fast_mmq.rs@@ -0,0 +1,400 @@+//! CUDA fast path for GGUF tiled matmul (prompt/prefill phase).+//! Handles batch > 8 (complement to fast_mmvq which handles batch 1-8).++use std::collections::HashMap;+use std::sync::{Mutex, OnceLock};++use super::cuda::{QCudaStorage, MATRIX_ROW_PADDING};+use super::GgmlDType;+use crate::cuda_backend::DeviceId;+use crate::{backend::BackendStorage, CudaDevice, CudaStorage, DType, Result, Shape};++use cudarc::driver::{CudaSlice, DevicePtr};++const QK8_1: usize = 32;+const BLOCK_Q8_1_MMQ_SIZE: usize = 4 * QK8_1 + 4 * 4; // 128 qs + 16 scale bytes = 144++#[inline]+fn pad(p: usize, q: usize) -> usize {+ p.div_ceil(q) * q+}++/// Quant types supported by MMQ kernels (same as MMVQ).+fn supports(dtype: GgmlDType) -> bool {+ matches!(+ dtype,+ GgmlDType::Q4_0+ | GgmlDType::Q4_1+ | GgmlDType::Q5_0+ | GgmlDType::Q5_1+ | GgmlDType::Q8_0+ | GgmlDType::Q2K+ | GgmlDType::Q3K+ | GgmlDType::Q4K+ | GgmlDType::Q5K+ | GgmlDType::Q6K+ )+}++/// qk (block quantization size) per dtype.+fn qk_for(dtype: GgmlDType) -> usize {+ match dtype {+ GgmlDType::Q4_0 | GgmlDType::Q4_1 | GgmlDType::Q5_0 | GgmlDType::Q5_1 | GgmlDType::Q8_0 => {+ 32+ }+ GgmlDType::Q2K | GgmlDType::Q3K | GgmlDType::Q4K | GgmlDType::Q5K | GgmlDType::Q6K => 256,+ _ => unreachable!(),+ }+}++// ds_layout mapping: which Q8_1_mmq scale layout to use per weight type.+enum DsLayout {+ D4,+ DS4,+ D2S6,+}++fn ds_layout_for(dtype: GgmlDType) -> DsLayout {+ match dtype {+ GgmlDType::Q4_0 | GgmlDType::Q4_1 => DsLayout::DS4,+ GgmlDType::Q5_0 => DsLayout::D4,+ GgmlDType::Q5_1 => DsLayout::DS4,+ GgmlDType::Q8_0 => DsLayout::D4,+ GgmlDType::Q2K => DsLayout::D2S6,+ GgmlDType::Q3K => DsLayout::D4,+ GgmlDType::Q4K | GgmlDType::Q5K => DsLayout::DS4,+ GgmlDType::Q6K => DsLayout::D4,+ _ => unreachable!(),+ }+}++type QuantizeLauncher = unsafe extern "C" fn(+ x: *const std::ffi::c_void,+ ids: *const i32,+ vy: *mut std::ffi::c_void,+ type_x: i32,+ ne00: i64,+ s01: i64,+ s02: i64,+ s03: i64,+ ne0: i64,+ ne1: i64,+ ne2: i64,+ ne3: i64,+ stream: *mut std::ffi::c_void,+);++fn quantize_launcher(layout: DsLayout) -> QuantizeLauncher {+ use candle_kernels::ffi;+ match layout {+ DsLayout::D4 => ffi::launch_mmq_quantize_q8_1_D4,+ DsLayout::DS4 => ffi::launch_mmq_quantize_q8_1_DS4,+ DsLayout::D2S6 => ffi::launch_mmq_quantize_q8_1_D2S6,+ }+}++type MmqLauncher = unsafe extern "C" fn(+ tmp_fixup: *mut std::ffi::c_void,+ x: *const std::ffi::c_void,+ y: *const std::ffi::c_void,+ dst: *mut std::ffi::c_void,+ ncols_x: i64,+ nrows_x: i64,+ ncols_y: i64,+ stride_row_x: i64,+ stride_col_dst: i64,+ cc: i32,+ nsm: i32,+ smpbo: i64,+ warp_size: i32,+ stream: *mut std::ffi::c_void,+);++fn mmq_launcher(dtype: GgmlDType) -> Option<MmqLauncher> {+ use candle_kernels::ffi;+ let f: MmqLauncher = match dtype {+ GgmlDType::Q4_0 => ffi::launch_mmq_gguf_q4_0,+ GgmlDType::Q4_1 => ffi::launch_mmq_gguf_q4_1,+ GgmlDType::Q5_0 => ffi::launch_mmq_gguf_q5_0,+ GgmlDType::Q5_1 => ffi::launch_mmq_gguf_q5_1,+ GgmlDType::Q8_0 => ffi::launch_mmq_gguf_q8_0,+ GgmlDType::Q2K => ffi::launch_mmq_gguf_q2_k,+ GgmlDType::Q3K => ffi::launch_mmq_gguf_q3_k,+ GgmlDType::Q4K => ffi::launch_mmq_gguf_q4_k,+ GgmlDType::Q5K => ffi::launch_mmq_gguf_q5_k,+ GgmlDType::Q6K => ffi::launch_mmq_gguf_q6_k,+ _ => return None,+ };+ Some(f)+}++// ---------------------------------------------------------------------------+// Per-device workspaces (grows-only, reused across calls).+// ---------------------------------------------------------------------------++struct WorkspaceSlot {+ slice: CudaSlice<u8>,+ cap: usize,+}++type WsMap = Mutex<HashMap<DeviceId, &'static Mutex<WorkspaceSlot>>>;++static MMQ_WORKSPACE: OnceLock<WsMap> = OnceLock::new();+static FIXUP_WORKSPACE: OnceLock<WsMap> = OnceLock::new();++fn workspace_ensure(+ ws: &'static OnceLock<WsMap>,+ dev: &CudaDevice,+ bytes: usize,+) -> Result<(u64, std::sync::MutexGuard<'static, WorkspaceSlot>)> {+ let map = ws.get_or_init(|| Mutex::new(HashMap::new()));+ let device_key = dev.id();+ let device_mtx: &'static Mutex<WorkspaceSlot> = {+ let mut guard = map.lock().unwrap();+ match guard.get(&device_key).copied() {+ Some(mtx) => mtx,+ None => {+ let slice = unsafe { dev.alloc::<u8>(bytes.max(1))? };+ let leaked = Box::leak(Box::new(Mutex::new(WorkspaceSlot {+ slice,+ cap: bytes.max(1),+ })));+ guard.insert(device_key, leaked);+ leaked+ }+ }+ };+ let mut slot = device_mtx.lock().unwrap();+ if slot.cap < bytes {+ slot.slice = unsafe { dev.alloc::<u8>(bytes)? };+ slot.cap = bytes;+ }+ let ptr = slot.slice.device_ptr(slot.slice.stream()).0;+ Ok((ptr, slot))+}++// ---------------------------------------------------------------------------+// Per-device info cache (compute capability, SM count, etc.)+// ---------------------------------------------------------------------------++#[derive(Clone, Copy)]+struct DeviceInfo {+ cc: i32,+ nsm: i32,+ smpbo: i64,+ warp_size: i32,+}++static DEVICE_INFO: OnceLock<Mutex<HashMap<DeviceId, DeviceInfo>>> = OnceLock::new();++fn get_device_info(dev: &CudaDevice) -> DeviceInfo {+ use cudarc::driver::{result, sys};+ let map = DEVICE_INFO.get_or_init(|| Mutex::new(HashMap::new()));+ let key = dev.id();+ let mut guard = map.lock().unwrap();+ if let Some(info) = guard.get(&key) {+ return *info;+ }+ let cu_device = dev.cuda_stream().context().cu_device();+ let major = unsafe {+ result::device::get_attribute(+ cu_device,+ sys::CUdevice_attribute::CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MAJOR,+ )+ }+ .unwrap_or(8);+ let minor = unsafe {+ result::device::get_attribute(+ cu_device,+ sys::CUdevice_attribute::CU_DEVICE_ATTRIBUTE_COMPUTE_CAPABILITY_MINOR,+ )+ }+ .unwrap_or(0);+ let nsm = unsafe {+ result::device::get_attribute(+ cu_device,+ sys::CUdevice_attribute::CU_DEVICE_ATTRIBUTE_MULTIPROCESSOR_COUNT,+ )+ }+ .unwrap_or(1);+ let smpbo = unsafe {+ result::device::get_attribute(+ cu_device,+ sys::CUdevice_attribute::CU_DEVICE_ATTRIBUTE_MAX_SHARED_MEMORY_PER_BLOCK_OPTIN,+ )+ }+ .unwrap_or(49152);+ let warp_size = unsafe {+ result::device::get_attribute(+ cu_device,+ sys::CUdevice_attribute::CU_DEVICE_ATTRIBUTE_WARP_SIZE,+ )+ }+ .unwrap_or(32);+ let info = DeviceInfo {+ cc: major * 100 + minor * 10,+ nsm,+ smpbo: smpbo as i64,+ warp_size,+ };+ guard.insert(key, info);+ info+}++// ---------------------------------------------------------------------------+// Public entry point+// ---------------------------------------------------------------------------++/// Try the fast MMQ path. Returns `Ok(None)` when the fast path is not applicable:+/// - unsupported quant dtype+/// - batch size 0+/// - non-BF16/F16/F32 input+/// - non-contiguous input+pub fn try_fwd(+ qstorage: &QCudaStorage,+ self_shape: &Shape,+ rhs: &CudaStorage,+ rhs_l: &crate::Layout,+) -> Result<Option<(CudaStorage, Shape)>> {+ let w_dtype = qstorage.dtype();+ if !supports(w_dtype) {+ return Ok(None);+ }+ let input_dtype = rhs.dtype();+ if !matches!(input_dtype, DType::BF16 | DType::F16 | DType::F32) {+ return Ok(None);+ }++ let (nrows, ncols) = self_shape.dims2()?;++ let (b_size, k) = match rhs_l.shape().dims() {+ [b, m, k] => (b * m, *k),+ [b, k] => (*b, *k),+ _ => return Ok(None),+ };+ if ncols != k {+ return Ok(None);+ }+ if b_size == 0 {+ return Ok(None);+ }++ let qk = qk_for(w_dtype);+ if k % qk != 0 {+ return Ok(None);+ }++ let (o1, o2) = match rhs_l.contiguous_offsets() {+ Some(offsets) => offsets,+ None => return Ok(None),+ };++ let dev = qstorage.device();+ let stream = dev.cuda_stream();+ let stream_ptr = stream.cu_stream() as *mut std::ffi::c_void;++ // MMQ quantize expects f32 input. Convert if needed.+ let rhs_f32_storage: Option<CudaStorage> = if input_dtype != DType::F32 {+ let layout_for_cast =+ crate::Layout::contiguous_with_offset(rhs_l.shape(), rhs_l.start_offset());+ Some(rhs.to_dtype(&layout_for_cast, DType::F32)?)+ } else {+ None+ };++ // Get the f32 slice (either original or converted)+ let rhs_f32_ref = rhs_f32_storage.as_ref().unwrap_or(rhs);+ let rhs_f32_slice = rhs_f32_ref.as_cuda_slice::<f32>()?;+ // For the converted case, offset is 0 (to_dtype returns contiguous from 0).+ // For the original f32 case, apply the original offsets.+ let rhs_f32_slice = if rhs_f32_storage.is_some() {+ rhs_f32_slice.slice(..b_size * k)+ } else {+ rhs_f32_slice.slice(o1..o2)+ };+ let rhs_ptr = rhs_f32_slice.device_ptr(&stream).0 as *const std::ffi::c_void;++ // Compute padded dimensions+ let k_padded = pad(k, MATRIX_ROW_PADDING);+ // Must also be multiple of 4*QK8_1 = 128 for block_q8_1_mmq+ let k_padded = pad(k_padded, 4 * QK8_1);++ // Workspace for block_q8_1_mmq quantized activations+ let blocks_per_row = k_padded / (4 * QK8_1);+ let workspace_main = b_size * blocks_per_row * BLOCK_Q8_1_MMQ_SIZE;+ // Extra padding for mmq_x_max (128 for MMA path)+ let workspace_extra = 128 * BLOCK_Q8_1_MMQ_SIZE;+ let workspace_bytes = workspace_main + workspace_extra;++ let (scratch_ptr, _workspace_guard) = workspace_ensure(&MMQ_WORKSPACE, dev, workspace_bytes)?;+ let scratch_ptr = scratch_ptr as *mut std::ffi::c_void;++ // Stream-k fixup workspace+ const MMQ_X_MAX: usize = 128;+ const MMQ_Y_MAX: usize = 128;+ const MAX_SMS: usize = 256;+ let fixup_bytes = MAX_SMS * MMQ_X_MAX * MMQ_Y_MAX * std::mem::size_of::<f32>();+ let (fixup_ptr, _fixup_guard) = workspace_ensure(&FIXUP_WORKSPACE, dev, fixup_bytes)?;+ let fixup_ptr = fixup_ptr as *mut std::ffi::c_void;++ let weight_ptr = qstorage.device_ptr()? as *const std::ffi::c_void;+ let stride_row_x = (k / qk) as i64;+ let di = get_device_info(dev);++ let out = unsafe { dev.alloc::<f32>(nrows * b_size)? };+ let stride_col_dst = nrows as i64;++ let out_ptr = out.device_ptr(&stream).0 as *mut std::ffi::c_void;++ unsafe {+ let quantize = quantize_launcher(ds_layout_for(w_dtype));+ quantize(+ rhs_ptr,+ std::ptr::null(),+ scratch_ptr,+ 0,+ k as i64,+ k as i64,+ 0,+ 0,+ k_padded as i64,+ b_size as i64,+ 1,+ 1,+ stream_ptr,+ );++ let launcher = mmq_launcher(w_dtype).expect("supports() checked");+ launcher(+ fixup_ptr,+ weight_ptr,+ scratch_ptr as *const std::ffi::c_void,+ out_ptr,+ k as i64,+ nrows as i64,+ b_size as i64,+ stride_row_x,+ stride_col_dst,+ di.cc,+ di.nsm,+ di.smpbo,+ di.warp_size,+ stream_ptr,+ );+ }++ let mut out_shape = rhs_l.shape().dims().to_vec();+ out_shape.pop();+ out_shape.push(nrows);++ let out_storage = CudaStorage::wrap_cuda_slice(out, dev.clone());++ if input_dtype == DType::F32 {+ Ok(Some((out_storage, out_shape.into())))+ } else {+ // Cast f32 output back to input dtype+ let out_layout = crate::Layout::contiguous(Shape::from(out_shape.clone()));+ let cast_storage = out_storage.to_dtype(&out_layout, input_dtype)?;+ Ok(Some((cast_storage, out_shape.into())))+ }+}diff --git a/vendor/candle-core/src/quantized/fast_mmvq.rs b/vendor/candle-core/src/quantized/fast_mmvq.rsnew file mode 100644index 0000000..0b53552--- /dev/null+++ b/vendor/candle-core/src/quantized/fast_mmvq.rs@@ -0,0 +1,323 @@+//! CUDA fast path for GGUF matmul with BF16/F32 activations.++use std::collections::HashMap;+use std::sync::{Mutex, OnceLock};++use super::cuda::{QCudaStorage, MATRIX_ROW_PADDING};+use super::GgmlDType;+use crate::cuda_backend::DeviceId;+use crate::{backend::BackendStorage, CudaDevice, CudaStorage, DType, Result, Shape};++use cudarc::driver::{CudaSlice, DevicePtr};++const Q8_1_BLOCK_SIZE: usize = 32;+const Q8_1_TYPE_SIZE: usize = 36; // 2 halves (4 bytes) + QK8_1 int8 = 4 + 32 = 36++#[inline]+fn pad(p: usize, q: usize) -> usize {+ p.div_ceil(q) * q+}++/// Quant types supported by the fast MMVQ kernels.+fn supports(dtype: GgmlDType) -> bool {+ matches!(+ dtype,+ GgmlDType::Q4_0+ | GgmlDType::Q4_1+ | GgmlDType::Q5_0+ | GgmlDType::Q5_1+ | GgmlDType::Q8_0+ | GgmlDType::Q2K+ | GgmlDType::Q3K+ | GgmlDType::Q4K+ | GgmlDType::Q5K+ | GgmlDType::Q6K+ )+}++const MMVQ_MAX_BATCH: usize = 8;++// ---------------------------------------------------------------------------+// Per-device Q8_1 scratch workspace (grows-only, reused across calls).+// ---------------------------------------------------------------------------++struct WorkspaceSlot {+ slice: CudaSlice<u8>,+ cap: usize,+}++static WORKSPACE: OnceLock<Mutex<HashMap<DeviceId, WorkspaceSlot>>> = OnceLock::new();++/// Returns a device pointer to the scratch workspace, growing it if needed.+/// The returned `MutexGuard` must be held alive until the kernels using+/// this pointer have been launched (all launches are on the device's+/// default stream, so they are serialised).+fn workspace_ensure(+ dev: &CudaDevice,+ bytes: usize,+) -> Result<(+ u64,+ std::sync::MutexGuard<'static, HashMap<DeviceId, WorkspaceSlot>>,+)> {+ let map = WORKSPACE.get_or_init(|| Mutex::new(HashMap::new()));+ let device_key = dev.id();+ let mut guard = map.lock().unwrap();+ let slot = match guard.entry(device_key) {+ std::collections::hash_map::Entry::Occupied(entry) => {+ let slot = entry.into_mut();+ if slot.cap < bytes {+ slot.slice = unsafe { dev.alloc::<u8>(bytes)? };+ slot.cap = bytes;+ }+ slot+ }+ std::collections::hash_map::Entry::Vacant(entry) => {+ let slice = unsafe { dev.alloc::<u8>(bytes)? };+ entry.insert(WorkspaceSlot { slice, cap: bytes })+ }+ };+ let ptr = slot.slice.device_ptr(slot.slice.stream()).0;+ Ok((ptr, guard))+}++// ---------------------------------------------------------------------------+// Launcher dispatch by weight dtype and output dtype.+// ---------------------------------------------------------------------------++type PlainLauncher = unsafe extern "C" fn(+ vx: *const std::ffi::c_void,+ vy: *const std::ffi::c_void,+ dst: *mut std::ffi::c_void,+ ncols_x: i32,+ nrows_x: i32,+ stride_col_y: i32,+ stride_col_dst: i32,+ b_size: i32,+ stream: *mut std::ffi::c_void,+);++fn plain_launcher_bf16(dtype: GgmlDType) -> Option<PlainLauncher> {+ use candle_kernels::ffi;+ let f: PlainLauncher = match dtype {+ GgmlDType::Q4_0 => ffi::launch_mmvq_gguf_q4_0_bf16_plain,+ GgmlDType::Q4_1 => ffi::launch_mmvq_gguf_q4_1_bf16_plain,+ GgmlDType::Q5_0 => ffi::launch_mmvq_gguf_q5_0_bf16_plain,+ GgmlDType::Q5_1 => ffi::launch_mmvq_gguf_q5_1_bf16_plain,+ GgmlDType::Q8_0 => ffi::launch_mmvq_gguf_q8_0_bf16_plain,+ GgmlDType::Q2K => ffi::launch_mmvq_gguf_q2_k_bf16_plain,+ GgmlDType::Q3K => ffi::launch_mmvq_gguf_q3_k_bf16_plain,+ GgmlDType::Q4K => ffi::launch_mmvq_gguf_q4_k_bf16_plain,+ GgmlDType::Q5K => ffi::launch_mmvq_gguf_q5_k_bf16_plain,+ GgmlDType::Q6K => ffi::launch_mmvq_gguf_q6_k_bf16_plain,+ _ => return None,+ };+ Some(f)+}++fn plain_launcher_f16(dtype: GgmlDType) -> Option<PlainLauncher> {+ use candle_kernels::ffi;+ let f: PlainLauncher = match dtype {+ GgmlDType::Q4_0 => ffi::launch_mmvq_gguf_q4_0_f16_plain,+ GgmlDType::Q4_1 => ffi::launch_mmvq_gguf_q4_1_f16_plain,+ GgmlDType::Q5_0 => ffi::launch_mmvq_gguf_q5_0_f16_plain,+ GgmlDType::Q5_1 => ffi::launch_mmvq_gguf_q5_1_f16_plain,+ GgmlDType::Q8_0 => ffi::launch_mmvq_gguf_q8_0_f16_plain,+ GgmlDType::Q2K => ffi::launch_mmvq_gguf_q2_k_f16_plain,+ GgmlDType::Q3K => ffi::launch_mmvq_gguf_q3_k_f16_plain,+ GgmlDType::Q4K => ffi::launch_mmvq_gguf_q4_k_f16_plain,+ GgmlDType::Q5K => ffi::launch_mmvq_gguf_q5_k_f16_plain,+ GgmlDType::Q6K => ffi::launch_mmvq_gguf_q6_k_f16_plain,+ _ => return None,+ };+ Some(f)+}++fn plain_launcher_f32(dtype: GgmlDType) -> Option<PlainLauncher> {+ use candle_kernels::ffi;+ let f: PlainLauncher = match dtype {+ GgmlDType::Q4_0 => ffi::launch_mmvq_gguf_q4_0_f32_plain,+ GgmlDType::Q4_1 => ffi::launch_mmvq_gguf_q4_1_f32_plain,+ GgmlDType::Q5_0 => ffi::launch_mmvq_gguf_q5_0_f32_plain,+ GgmlDType::Q5_1 => ffi::launch_mmvq_gguf_q5_1_f32_plain,+ GgmlDType::Q8_0 => ffi::launch_mmvq_gguf_q8_0_f32_plain,+ GgmlDType::Q2K => ffi::launch_mmvq_gguf_q2_k_f32_plain,+ GgmlDType::Q3K => ffi::launch_mmvq_gguf_q3_k_f32_plain,+ GgmlDType::Q4K => ffi::launch_mmvq_gguf_q4_k_f32_plain,+ GgmlDType::Q5K => ffi::launch_mmvq_gguf_q5_k_f32_plain,+ GgmlDType::Q6K => ffi::launch_mmvq_gguf_q6_k_f32_plain,+ _ => return None,+ };+ Some(f)+}++// ---------------------------------------------------------------------------+// Public entry point+// ---------------------------------------------------------------------------++/// Try the fast MMVQ path. Returns `Ok(None)` when the fast path is not applicable:+/// - unsupported quant dtype+/// - batch too large+/// - non-BF16/F32 input+pub fn try_fwd(+ qstorage: &QCudaStorage,+ self_shape: &Shape,+ rhs: &CudaStorage,+ rhs_l: &crate::Layout,+) -> Result<Option<(CudaStorage, Shape)>> {+ use candle_kernels::ffi;++ // Gate checks.+ let w_dtype = qstorage.dtype();+ if !supports(w_dtype) {+ return Ok(None);+ }+ let input_dtype = rhs.dtype();+ if !matches!(input_dtype, DType::BF16 | DType::F16 | DType::F32) {+ return Ok(None);+ }++ let (nrows, ncols) = self_shape.dims2()?;++ let (b_size, k) = match rhs_l.shape().dims() {+ [b, m, k] => (b * m, *k),+ [b, k] => (*b, *k),+ _ => return Ok(None),+ };+ if ncols != k {+ return Ok(None);+ }+ if b_size == 0 || b_size > MMVQ_MAX_BATCH {+ return Ok(None);+ }++ let (o1, o2) = match rhs_l.contiguous_offsets() {+ Some(offsets) => offsets,+ None => return Ok(None),+ };++ let dev = qstorage.device();+ let stream_ptr = dev.cuda_stream().cu_stream() as *mut std::ffi::c_void;++ let k_padded = pad(k, MATRIX_ROW_PADDING);+ let num_blocks_per_row = k_padded / Q8_1_BLOCK_SIZE;+ let dst_row_bytes = num_blocks_per_row * Q8_1_TYPE_SIZE;+ let scratch_bytes = b_size * dst_row_bytes;++ let (scratch_ptr, _workspace_guard) = workspace_ensure(dev, scratch_bytes)?;+ let scratch_ptr = scratch_ptr as *mut std::ffi::c_void;+ let stride_col_y = (k_padded / Q8_1_BLOCK_SIZE) as i32;+ let stride_col_dst = nrows as i32;+ let weight_ptr = qstorage.device_ptr()? as *const std::ffi::c_void;++ let mut out_shape = rhs_l.shape().dims().to_vec();+ out_shape.pop();+ out_shape.push(nrows);++ let stream = dev.cuda_stream();++ match input_dtype {+ DType::BF16 => {+ let rhs_slice = rhs.as_cuda_slice::<half::bf16>()?;+ let rhs_slice = rhs_slice.slice(o1..o2);+ let out = unsafe { dev.alloc::<half::bf16>(nrows * b_size)? };++ let rhs_ptr = rhs_slice.device_ptr(&stream).0 as *const std::ffi::c_void;+ let out_ptr = out.device_ptr(&stream).0 as *mut std::ffi::c_void;++ unsafe {+ ffi::launch_mmvq_gguf_quantize_q8_1_bf16(+ rhs_ptr,+ scratch_ptr,+ k as i32,+ k_padded as i32,+ b_size as i32,+ stream_ptr,+ );+ let launcher = plain_launcher_bf16(w_dtype).unwrap();+ launcher(+ weight_ptr,+ scratch_ptr as *const std::ffi::c_void,+ out_ptr,+ k as i32,+ nrows as i32,+ stride_col_y,+ stride_col_dst,+ b_size as i32,+ stream_ptr,+ );+ }++ let out_storage = CudaStorage::wrap_cuda_slice(out, dev.clone());+ Ok(Some((out_storage, out_shape.into())))+ }+ DType::F16 => {+ let rhs_slice = rhs.as_cuda_slice::<half::f16>()?;+ let rhs_slice = rhs_slice.slice(o1..o2);+ let out = unsafe { dev.alloc::<half::f16>(nrows * b_size)? };++ let rhs_ptr = rhs_slice.device_ptr(&stream).0 as *const std::ffi::c_void;+ let out_ptr = out.device_ptr(&stream).0 as *mut std::ffi::c_void;++ unsafe {+ ffi::launch_mmvq_gguf_quantize_q8_1_f16(+ rhs_ptr,+ scratch_ptr,+ k as i32,+ k_padded as i32,+ b_size as i32,+ stream_ptr,+ );+ let launcher = plain_launcher_f16(w_dtype).unwrap();+ launcher(+ weight_ptr,+ scratch_ptr as *const std::ffi::c_void,+ out_ptr,+ k as i32,+ nrows as i32,+ stride_col_y,+ stride_col_dst,+ b_size as i32,+ stream_ptr,+ );+ }++ let out_storage = CudaStorage::wrap_cuda_slice(out, dev.clone());+ Ok(Some((out_storage, out_shape.into())))+ }+ DType::F32 => {+ let rhs_slice = rhs.as_cuda_slice::<f32>()?;+ let rhs_slice = rhs_slice.slice(o1..o2);+ let out = unsafe { dev.alloc::<f32>(nrows * b_size)? };++ let rhs_ptr = rhs_slice.device_ptr(&stream).0 as *const std::ffi::c_void;+ let out_ptr = out.device_ptr(&stream).0 as *mut std::ffi::c_void;++ unsafe {+ ffi::launch_mmvq_gguf_quantize_q8_1_f32(+ rhs_ptr,+ scratch_ptr,+ k as i32,+ k_padded as i32,+ b_size as i32,+ stream_ptr,+ );+ let launcher = plain_launcher_f32(w_dtype).unwrap();+ launcher(+ weight_ptr,+ scratch_ptr as *const std::ffi::c_void,+ out_ptr,+ k as i32,+ nrows as i32,+ stride_col_y,+ stride_col_dst,+ b_size as i32,+ stream_ptr,+ );+ }++ let out_storage = CudaStorage::wrap_cuda_slice(out, dev.clone());+ Ok(Some((out_storage, out_shape.into())))+ }+ _ => Ok(None),+ }+}diff --git a/vendor/candle-core/src/quantized/ggml_file.rs b/vendor/candle-core/src/quantized/ggml_file.rsnew file mode 100644index 0000000..8609370--- /dev/null+++ b/vendor/candle-core/src/quantized/ggml_file.rs@@ -0,0 +1,269 @@+//! Support for the GGML file format.++use super::{k_quants, GgmlDType, QStorage};+use crate::{Device, Result};+use byteorder::{LittleEndian, ReadBytesExt};+use std::collections::HashMap;++// https://github.com/ggerganov/llama.cpp/blob/468ea24fb4633a0d681f7ac84089566c1c6190cb/llama.h#L37+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+enum Magic {+ Ggjt,+ Ggla,+ Ggmf,+ Ggml,+ Ggsn,+}++impl TryFrom<u32> for Magic {+ type Error = crate::Error;+ fn try_from(value: u32) -> Result<Self> {+ let magic = match value {+ 0x67676a74 => Self::Ggjt,+ 0x67676c61 => Self::Ggla,+ 0x67676d66 => Self::Ggmf,+ 0x67676d6c => Self::Ggml,+ 0x6767736e => Self::Ggsn,+ _ => crate::bail!("unknown magic {value:08x}"),+ };+ Ok(magic)+ }+}++#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum VersionedMagic {+ GgmlUnversioned,+ GgmfV1,+ GgjtV1,+ GgjtV2,+ GgjtV3,+}++impl VersionedMagic {+ fn read<R: std::io::Read>(reader: &mut R) -> Result<Self> {+ let magic = reader.read_u32::<LittleEndian>()?;+ let magic = Magic::try_from(magic)?;+ if magic == Magic::Ggml {+ return Ok(Self::GgmlUnversioned);+ }+ let version = reader.read_u32::<LittleEndian>()?;+ let versioned_magic = match (magic, version) {+ (Magic::Ggmf, 1) => Self::GgmfV1,+ (Magic::Ggjt, 1) => Self::GgjtV1,+ (Magic::Ggjt, 2) => Self::GgjtV2,+ (Magic::Ggjt, 3) => Self::GgjtV3,+ _ => crate::bail!("ggml: unsupported magic/version {magic:?}/{version}"),+ };+ Ok(versioned_magic)+ }++ fn align32(&self) -> bool {+ match self {+ Self::GgmlUnversioned | Self::GgmfV1 => false,+ Self::GgjtV1 | Self::GgjtV2 | Self::GgjtV3 => true,+ }+ }+}++#[derive(Debug, Clone, PartialEq, Eq)]+pub struct HParams {+ pub n_vocab: u32,+ pub n_embd: u32,+ pub n_mult: u32,+ pub n_head: u32,+ pub n_layer: u32,+ pub n_rot: u32,+ pub ftype: u32,+}++impl HParams {+ fn read<R: std::io::Read>(reader: &mut R) -> Result<Self> {+ let n_vocab = reader.read_u32::<LittleEndian>()?;+ let n_embd = reader.read_u32::<LittleEndian>()?;+ let n_mult = reader.read_u32::<LittleEndian>()?;+ let n_head = reader.read_u32::<LittleEndian>()?;+ let n_layer = reader.read_u32::<LittleEndian>()?;+ let n_rot = reader.read_u32::<LittleEndian>()?;+ let ftype = reader.read_u32::<LittleEndian>()?;+ Ok(Self {+ n_vocab,+ n_embd,+ n_mult,+ n_head,+ n_layer,+ n_rot,+ ftype,+ })+ }+}++#[derive(Debug, Clone, PartialEq)]+pub struct Vocab {+ pub token_score_pairs: Vec<(Vec<u8>, f32)>,+}++impl Vocab {+ fn read<R: std::io::Read>(reader: &mut R, n_vocab: usize) -> Result<Self> {+ // https://github.com/ggerganov/llama.cpp/blob/468ea24fb4633a0d681f7ac84089566c1c6190cb/llama.cpp#L556+ let mut token_score_pairs = Vec::with_capacity(n_vocab);+ for _index in 0..n_vocab {+ let len = reader.read_u32::<LittleEndian>()? as usize;+ let mut word = vec![0u8; len];+ reader.read_exact(&mut word)?;+ let score = reader.read_f32::<LittleEndian>()?;+ token_score_pairs.push((word, score))+ }+ Ok(Self { token_score_pairs })+ }+}++fn from_raw_data<T: super::GgmlType + Send + Sync + 'static>(+ raw_data: &[u8],+ size_in_bytes: usize,+ dims: Vec<usize>,+ device: &Device,+) -> Result<super::QTensor> {+ let raw_data_ptr = raw_data.as_ptr();+ let n_blocks = size_in_bytes / std::mem::size_of::<T>();+ let data = unsafe { std::slice::from_raw_parts(raw_data_ptr as *const T, n_blocks) };+ let data: QStorage = match device {+ Device::Cpu => QStorage::Cpu(Box::new(data.to_vec())),+ Device::Metal(metal) => super::metal::load_quantized(metal, data)?,+ Device::Cuda(cuda) => super::cuda::load_quantized(cuda, data)?,+ };+ super::QTensor::new(data, dims)+}++/// Creates a [Tensor] from a raw GGML tensor.+pub fn qtensor_from_ggml(+ ggml_dtype: GgmlDType,+ raw_data: &[u8],+ dims: Vec<usize>,+ device: &Device,+) -> Result<super::QTensor> {+ let tensor_elems = dims.iter().product::<usize>();+ let block_size = ggml_dtype.block_size();+ if tensor_elems % block_size != 0 {+ crate::bail!(+ "the number of elements {tensor_elems} is not divisible by the block size {block_size}"+ )+ }+ let size_in_bytes = tensor_elems / block_size * ggml_dtype.type_size();++ match ggml_dtype {+ GgmlDType::F32 => from_raw_data::<f32>(raw_data, size_in_bytes, dims, device),+ GgmlDType::F16 => from_raw_data::<half::f16>(raw_data, size_in_bytes, dims, device),+ GgmlDType::BF16 => from_raw_data::<half::bf16>(raw_data, size_in_bytes, dims, device),+ GgmlDType::Q4_0 => {+ from_raw_data::<k_quants::BlockQ4_0>(raw_data, size_in_bytes, dims, device)+ }+ GgmlDType::Q4_1 => {+ from_raw_data::<k_quants::BlockQ4_1>(raw_data, size_in_bytes, dims, device)+ }+ GgmlDType::Q5_0 => {+ from_raw_data::<k_quants::BlockQ5_0>(raw_data, size_in_bytes, dims, device)+ }+ GgmlDType::Q5_1 => {+ from_raw_data::<k_quants::BlockQ5_1>(raw_data, size_in_bytes, dims, device)+ }+ GgmlDType::Q8_0 => {+ from_raw_data::<k_quants::BlockQ8_0>(raw_data, size_in_bytes, dims, device)+ }+ GgmlDType::Q2K => {+ from_raw_data::<k_quants::BlockQ2K>(raw_data, size_in_bytes, dims, device)+ }+ GgmlDType::Q3K => {+ from_raw_data::<k_quants::BlockQ3K>(raw_data, size_in_bytes, dims, device)+ }+ GgmlDType::Q4K => {+ from_raw_data::<k_quants::BlockQ4K>(raw_data, size_in_bytes, dims, device)+ }+ GgmlDType::Q5K => {+ from_raw_data::<k_quants::BlockQ5K>(raw_data, size_in_bytes, dims, device)+ }+ GgmlDType::Q6K => {+ from_raw_data::<k_quants::BlockQ6K>(raw_data, size_in_bytes, dims, device)+ }+ GgmlDType::Mxfp4 => {+ from_raw_data::<k_quants::BlockMxfp4>(raw_data, size_in_bytes, dims, device)+ }+ _ => crate::bail!("quantized type {ggml_dtype:?} is not supported yet"),+ }+}++fn read_one_tensor<R: std::io::Seek + std::io::Read>(+ reader: &mut R,+ magic: VersionedMagic,+ device: &Device,+) -> Result<(String, super::QTensor)> {+ let n_dims = reader.read_u32::<LittleEndian>()?;+ let name_len = reader.read_u32::<LittleEndian>()?;+ let ggml_dtype = reader.read_u32::<LittleEndian>()?;+ let ggml_dtype = GgmlDType::from_u32(ggml_dtype)?;+ let mut dims = vec![0u32; n_dims as usize];+ reader.read_u32_into::<LittleEndian>(&mut dims)?;+ // The dimensions are stored in reverse order, see for example:+ // https://github.com/ggerganov/llama.cpp/blob/b5ffb2849d23afe73647f68eec7b68187af09be6/convert.py#L969+ dims.reverse();+ let mut name = vec![0u8; name_len as usize];+ reader.read_exact(&mut name)?;+ let name = String::from_utf8_lossy(&name).into_owned();++ if magic.align32() {+ let pos = reader.stream_position()?;+ reader.seek(std::io::SeekFrom::Current(((32 - pos % 32) % 32) as i64))?;+ }+ let dims = dims.iter().map(|&u| u as usize).collect::<Vec<_>>();+ let tensor_elems = dims.iter().product::<usize>();+ let size_in_bytes = tensor_elems * ggml_dtype.type_size() / ggml_dtype.block_size();+ // TODO: Mmap version to avoid copying the data around?+ let mut raw_data = vec![0u8; size_in_bytes];+ reader.read_exact(&mut raw_data)?;+ match qtensor_from_ggml(ggml_dtype, &raw_data, dims, device) {+ Ok(tensor) => Ok((name, tensor)),+ Err(e) => crate::bail!("Error creating tensor {name}: {e}"),+ }+}++pub struct Content {+ pub magic: VersionedMagic,+ pub hparams: HParams,+ pub vocab: Vocab,+ pub tensors: HashMap<String, super::QTensor>,+ pub device: Device,+}++impl Content {+ pub fn read<R: std::io::Seek + std::io::Read>(+ reader: &mut R,+ device: &Device,+ ) -> Result<Content> {+ // https://github.com/ggerganov/llama.cpp/blob/468ea24fb4633a0d681f7ac84089566c1c6190cb/llama.cpp#L505+ let last_position = reader.seek(std::io::SeekFrom::End(0))?;+ reader.seek(std::io::SeekFrom::Start(0))?;+ let magic = VersionedMagic::read(reader)?;+ let hparams = HParams::read(reader)?;+ let vocab = Vocab::read(reader, hparams.n_vocab as usize)?;+ let mut tensors = HashMap::new();++ while reader.stream_position()? != last_position {+ let (name, tensor) = read_one_tensor(reader, magic, device)?;+ tensors.insert(name, tensor);+ }+ let device = device.clone();+ Ok(Self {+ magic,+ hparams,+ vocab,+ tensors,+ device,+ })+ }++ pub fn remove(&mut self, name: &str) -> Result<super::QTensor> {+ match self.tensors.remove(name) {+ None => crate::bail!("cannot find tensor with name '{name}'"),+ Some(tensor) => Ok(tensor),+ }+ }+}diff --git a/vendor/candle-core/src/quantized/gguf_file.rs b/vendor/candle-core/src/quantized/gguf_file.rsnew file mode 100644index 0000000..fa07510--- /dev/null+++ b/vendor/candle-core/src/quantized/gguf_file.rs@@ -0,0 +1,638 @@+//! Support for the [GGUF file format](https://github.com/philpax/ggml/blob/gguf-spec/docs/gguf.md).+//!+//! Spec: https://github.com/ggml-org/ggml/blob/master/docs/gguf.md++use super::{GgmlDType, QTensor};+use crate::{Context, Device, Result};+use byteorder::{LittleEndian, ReadBytesExt, WriteBytesExt};+use std::collections::HashMap;++pub const DEFAULT_ALIGNMENT: u64 = 32;++// Caps mirror ggml-org/llama.cpp#19856 (GGUF_MAX_STRING_LENGTH,+// GGUF_MAX_ARRAY_ELEMENTS) and GGML_MAX_DIMS from ggml.h. The depth cap+// bounds recursion through nested `Value::Array` so a crafted file cannot+// overflow the stack with a chain of arrays-of-arrays.+const GGUF_MAX_STRING_LENGTH: u64 = 1 << 30;+const GGUF_MAX_ARRAY_ELEMENTS: u64 = 1 << 30;+const GGUF_MAX_TENSOR_DIMS: u32 = 4;+const GGUF_MAX_VALUE_DEPTH: usize = 64;++// `file_size` is the byte length captured once up front, so this avoids+// seeking to the end and back on every length-prefixed read.+fn remaining_bytes<R: std::io::Seek>(reader: &mut R, file_size: u64) -> Result<u64> {+ let cur = reader.stream_position()?;+ Ok(file_size.saturating_sub(cur))+}++fn read_length<R: std::io::Read>(reader: &mut R, magic: &VersionedMagic) -> Result<u64> {+ match magic {+ VersionedMagic::GgufV1 => Ok(reader.read_u32::<LittleEndian>()? as u64),+ VersionedMagic::GgufV2 | VersionedMagic::GgufV3 => Ok(reader.read_u64::<LittleEndian>()?),+ }+}++#[derive(Debug, Clone, Copy, PartialEq, Eq)]+enum Magic {+ Gguf,+}++impl TryFrom<u32> for Magic {+ type Error = crate::Error;+ fn try_from(value: u32) -> Result<Self> {+ let magic = match value {+ 0x46554747 | 0x47475546 => Self::Gguf,+ _ => crate::bail!("unknown magic 0x{value:08x}"),+ };+ Ok(magic)+ }+}++#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub enum VersionedMagic {+ GgufV1,+ GgufV2,+ GgufV3,+}++impl VersionedMagic {+ fn read<R: std::io::Read>(reader: &mut R) -> Result<Self> {+ let magic = reader.read_u32::<LittleEndian>()?;+ let magic = Magic::try_from(magic)?;+ let version = reader.read_u32::<LittleEndian>()?;+ let versioned_magic = match (magic, version) {+ (Magic::Gguf, 1) => Self::GgufV1,+ (Magic::Gguf, 2) => Self::GgufV2,+ (Magic::Gguf, 3) => Self::GgufV3,+ _ => crate::bail!("gguf: unsupported magic/version {magic:?}/{version}"),+ };+ Ok(versioned_magic)+ }++ fn length_prefix_size(&self) -> u64 {+ match self {+ Self::GgufV1 => 4,+ Self::GgufV2 | Self::GgufV3 => 8,+ }+ }+}++#[derive(Debug)]+pub struct TensorInfo {+ pub ggml_dtype: GgmlDType,+ pub shape: crate::Shape,+ pub offset: u64,+}++impl TensorInfo {+ pub fn read<R: std::io::Seek + std::io::Read>(+ &self,+ reader: &mut R,+ tensor_data_offset: u64,+ device: &Device,+ ) -> Result<QTensor> {+ let tensor_elems = self.shape.elem_count();+ let block_size = self.ggml_dtype.block_size();+ if !tensor_elems.is_multiple_of(block_size) {+ crate::bail!(+ "the number of elements {tensor_elems} is not divisible by the block size {block_size}"+ )+ }+ let size_in_bytes = tensor_elems / block_size * self.ggml_dtype.type_size();+ let tensor_start = tensor_data_offset.saturating_add(self.offset);+ let file_size = reader.seek(std::io::SeekFrom::End(0))?;+ let remaining = file_size.saturating_sub(tensor_start);+ if size_in_bytes as u64 > remaining {+ crate::bail!(+ "tensor needs {size_in_bytes} bytes at offset {tensor_start}, only {remaining} remaining in file"+ )+ }+ let mut raw_data = vec![0u8; size_in_bytes];+ reader.seek(std::io::SeekFrom::Start(tensor_start))?;+ reader.read_exact(&mut raw_data)?;+ super::ggml_file::qtensor_from_ggml(+ self.ggml_dtype,+ &raw_data,+ self.shape.dims().to_vec(),+ device,+ )+ }+}++#[derive(Debug)]+pub struct Content {+ pub magic: VersionedMagic,+ pub metadata: HashMap<String, Value>,+ pub tensor_infos: HashMap<String, TensorInfo>,+ pub tensor_data_offset: u64,+}++fn read_string<R: std::io::Read + std::io::Seek>(+ reader: &mut R,+ magic: &VersionedMagic,+ file_size: u64,+) -> Result<String> {+ let len = read_length(reader, magic)?;+ if len > GGUF_MAX_STRING_LENGTH {+ crate::bail!("gguf: string length {len} exceeds max {GGUF_MAX_STRING_LENGTH}")+ }+ let remaining = remaining_bytes(reader, file_size)?;+ if len > remaining {+ crate::bail!("gguf: string length {len} exceeds remaining file bytes {remaining}")+ }+ let mut v = vec![0u8; len as usize];+ reader.read_exact(&mut v)?;+ // GGUF strings are supposed to be non-null terminated but in practice this happens.+ while let Some(0) = v.last() {+ v.pop();+ }+ // GGUF strings are utf8 encoded but there are cases that don't seem to be valid.+ Ok(String::from_utf8_lossy(&v).into_owned())+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]+pub enum ValueType {+ // The value is a 8-bit unsigned integer.+ U8,+ // The value is a 8-bit signed integer.+ I8,+ // The value is a 16-bit unsigned little-endian integer.+ U16,+ // The value is a 16-bit signed little-endian integer.+ I16,+ // The value is a 32-bit unsigned little-endian integer.+ U32,+ // The value is a 32-bit signed little-endian integer.+ I32,+ // The value is a 64-bit unsigned little-endian integer.+ U64,+ // The value is a 64-bit signed little-endian integer.+ I64,+ // The value is a 32-bit IEEE754 floating point number.+ F32,+ // The value is a 64-bit IEEE754 floating point number.+ F64,+ // The value is a boolean.+ // 1-byte value where 0 is false and 1 is true.+ // Anything else is invalid, and should be treated as either the model being invalid or the reader being buggy.+ Bool,+ // The value is a UTF-8 non-null-terminated string, with length prepended.+ String,+ // The value is an array of other values, with the length and type prepended.+ // Arrays can be nested, and the length of the array is the number of elements in the array, not the number of bytes.+ Array,+}++#[derive(Debug, Clone)]+pub enum Value {+ U8(u8),+ I8(i8),+ U16(u16),+ I16(i16),+ U32(u32),+ I32(i32),+ U64(u64),+ I64(i64),+ F32(f32),+ F64(f64),+ Bool(bool),+ String(String),+ Array(Vec<Value>),+}++impl Value {+ pub fn value_type(&self) -> ValueType {+ match self {+ Self::U8(_) => ValueType::U8,+ Self::I8(_) => ValueType::I8,+ Self::U16(_) => ValueType::U16,+ Self::I16(_) => ValueType::I16,+ Self::U32(_) => ValueType::U32,+ Self::I32(_) => ValueType::I32,+ Self::U64(_) => ValueType::U64,+ Self::I64(_) => ValueType::I64,+ Self::F32(_) => ValueType::F32,+ Self::F64(_) => ValueType::F64,+ Self::Bool(_) => ValueType::Bool,+ Self::String(_) => ValueType::String,+ Self::Array(_) => ValueType::Array,+ }+ }++ pub fn to_u8(&self) -> Result<u8> {+ match self {+ Self::U8(v) => Ok(*v),+ v => crate::bail!("not a u8 {v:?}"),+ }+ }++ pub fn to_i8(&self) -> Result<i8> {+ match self {+ Self::I8(v) => Ok(*v),+ v => crate::bail!("not a i8 {v:?}"),+ }+ }++ pub fn to_u16(&self) -> Result<u16> {+ match self {+ Self::U16(v) => Ok(*v),+ v => crate::bail!("not a u16 {v:?}"),+ }+ }++ pub fn to_i16(&self) -> Result<i16> {+ match self {+ Self::I16(v) => Ok(*v),+ v => crate::bail!("not a i16 {v:?}"),+ }+ }++ pub fn to_u32(&self) -> Result<u32> {+ match self {+ Self::U32(v) => Ok(*v),+ v => crate::bail!("not a u32 {v:?}"),+ }+ }++ pub fn to_i32(&self) -> Result<i32> {+ match self {+ Self::I32(v) => Ok(*v),+ v => crate::bail!("not a i32 {v:?}"),+ }+ }++ /// This will also automatically upcast any integral types which will not truncate.+ pub fn to_u64(&self) -> Result<u64> {+ match self {+ Self::U64(v) => Ok(*v),+ // Autoupcast cases here+ Self::U8(v) => Ok(*v as u64),+ Self::U16(v) => Ok(*v as u64),+ Self::U32(v) => Ok(*v as u64),+ Self::Bool(v) => Ok(*v as u64),+ v => crate::bail!("not a u64 or upcastable to u64 {v:?}"),+ }+ }++ pub fn to_i64(&self) -> Result<i64> {+ match self {+ Self::I64(v) => Ok(*v),+ v => crate::bail!("not a i64 {v:?}"),+ }+ }++ pub fn to_f32(&self) -> Result<f32> {+ match self {+ Self::F32(v) => Ok(*v),+ v => crate::bail!("not a f32 {v:?}"),+ }+ }++ pub fn to_f64(&self) -> Result<f64> {+ match self {+ Self::F64(v) => Ok(*v),+ v => crate::bail!("not a f64 {v:?}"),+ }+ }++ pub fn to_bool(&self) -> Result<bool> {+ match self {+ Self::Bool(v) => Ok(*v),+ v => crate::bail!("not a bool {v:?}"),+ }+ }++ pub fn to_vec(&self) -> Result<&Vec<Value>> {+ match self {+ Self::Array(v) => Ok(v),+ v => crate::bail!("not a vec {v:?}"),+ }+ }++ pub fn to_string(&self) -> Result<&String> {+ match self {+ Self::String(v) => Ok(v),+ v => crate::bail!("not a string {v:?}"),+ }+ }++ fn read<R: std::io::Read + std::io::Seek>(+ reader: &mut R,+ value_type: ValueType,+ magic: &VersionedMagic,+ depth: usize,+ file_size: u64,+ ) -> Result<Self> {+ if depth > GGUF_MAX_VALUE_DEPTH {+ crate::bail!("gguf: value nesting depth exceeds max {GGUF_MAX_VALUE_DEPTH}")+ }+ let v = match value_type {+ ValueType::U8 => Self::U8(reader.read_u8()?),+ ValueType::I8 => Self::I8(reader.read_i8()?),+ ValueType::U16 => Self::U16(reader.read_u16::<LittleEndian>()?),+ ValueType::I16 => Self::I16(reader.read_i16::<LittleEndian>()?),+ ValueType::U32 => Self::U32(reader.read_u32::<LittleEndian>()?),+ ValueType::I32 => Self::I32(reader.read_i32::<LittleEndian>()?),+ ValueType::U64 => Self::U64(reader.read_u64::<LittleEndian>()?),+ ValueType::I64 => Self::I64(reader.read_i64::<LittleEndian>()?),+ ValueType::F32 => Self::F32(reader.read_f32::<LittleEndian>()?),+ ValueType::F64 => Self::F64(reader.read_f64::<LittleEndian>()?),+ ValueType::Bool => match reader.read_u8()? {+ 0 => Self::Bool(false),+ 1 => Self::Bool(true),+ b => crate::bail!("unexpected bool value {b}"),+ },+ ValueType::String => Self::String(read_string(reader, magic, file_size)?),+ ValueType::Array => {+ let value_type = reader.read_u32::<LittleEndian>()?;+ let value_type = ValueType::from_u32(value_type)?;+ let len = read_length(reader, magic)?;+ if len > GGUF_MAX_ARRAY_ELEMENTS {+ crate::bail!("gguf: array length {len} exceeds max {GGUF_MAX_ARRAY_ELEMENTS}")+ }+ let needed = len.saturating_mul(value_type.min_disk_size(magic));+ let remaining = remaining_bytes(reader, file_size)?;+ if needed > remaining {+ crate::bail!(+ "gguf: array of {len} elements needs at least {needed} bytes, only {remaining} remaining"+ )+ }+ let mut vs = Vec::new();+ for _ in 0..len {+ vs.push(Value::read(+ reader,+ value_type,+ magic,+ depth + 1,+ file_size,+ )?)+ }+ Self::Array(vs)+ }+ };+ Ok(v)+ }++ fn write<W: std::io::Write>(&self, w: &mut W) -> Result<()> {+ match self {+ &Self::U8(v) => w.write_u8(v)?,+ &Self::I8(v) => w.write_i8(v)?,+ &Self::U16(v) => w.write_u16::<LittleEndian>(v)?,+ &Self::I16(v) => w.write_i16::<LittleEndian>(v)?,+ &Self::U32(v) => w.write_u32::<LittleEndian>(v)?,+ &Self::I32(v) => w.write_i32::<LittleEndian>(v)?,+ &Self::U64(v) => w.write_u64::<LittleEndian>(v)?,+ &Self::I64(v) => w.write_i64::<LittleEndian>(v)?,+ &Self::F32(v) => w.write_f32::<LittleEndian>(v)?,+ &Self::F64(v) => w.write_f64::<LittleEndian>(v)?,+ &Self::Bool(v) => w.write_u8(u8::from(v))?,+ Self::String(v) => write_string(w, v.as_str())?,+ Self::Array(v) => {+ // The `Value` type does not enforce that all the values in an Array have the same+ // type.+ let value_type = if v.is_empty() {+ // Doesn't matter, the array is empty.+ ValueType::U32+ } else {+ let value_type: std::collections::HashSet<_> =+ v.iter().map(|elem| elem.value_type()).collect();+ if value_type.len() != 1 {+ crate::bail!("multiple value-types in the same array {value_type:?}")+ }+ value_type.into_iter().next().context("empty value_type")?+ };+ w.write_u32::<LittleEndian>(value_type.to_u32())?;+ w.write_u64::<LittleEndian>(v.len() as u64)?;+ for elem in v.iter() {+ elem.write(w)?+ }+ }+ }+ Ok(())+ }+}++impl ValueType {+ fn from_u32(v: u32) -> Result<Self> {+ let v = match v {+ 0 => Self::U8,+ 1 => Self::I8,+ 2 => Self::U16,+ 3 => Self::I16,+ 4 => Self::U32,+ 5 => Self::I32,+ 6 => Self::F32,+ 7 => Self::Bool,+ 8 => Self::String,+ 9 => Self::Array,+ 10 => Self::U64,+ 11 => Self::I64,+ 12 => Self::F64,+ v => crate::bail!("unrecognized value-type {v:#08x}"),+ };+ Ok(v)+ }++ fn to_u32(self) -> u32 {+ match self {+ Self::U8 => 0,+ Self::I8 => 1,+ Self::U16 => 2,+ Self::I16 => 3,+ Self::U32 => 4,+ Self::I32 => 5,+ Self::F32 => 6,+ Self::Bool => 7,+ Self::String => 8,+ Self::Array => 9,+ Self::U64 => 10,+ Self::I64 => 11,+ Self::F64 => 12,+ }+ }++ /// Minimum on-disk size of one value of this type, used to reject array+ /// lengths that exceed the remaining file size before allocating.+ fn min_disk_size(&self, magic: &VersionedMagic) -> u64 {+ match self {+ Self::U8 | Self::I8 | Self::Bool => 1,+ Self::U16 | Self::I16 => 2,+ Self::U32 | Self::I32 | Self::F32 => 4,+ Self::U64 | Self::I64 | Self::F64 => 8,+ Self::String => magic.length_prefix_size(),+ Self::Array => 4 + magic.length_prefix_size(),+ }+ }+}++impl Content {+ pub fn read<R: std::io::Seek + std::io::Read>(reader: &mut R) -> Result<Self> {+ // Capture the file size once so the bounds checks below don't have to+ // seek to the end and back on every length-prefixed read.+ let start = reader.stream_position()?;+ let file_size = reader.seek(std::io::SeekFrom::End(0))?;+ reader.seek(std::io::SeekFrom::Start(start))?;++ let magic = VersionedMagic::read(reader)?;+ let tensor_count = read_length(reader, &magic)?;+ let metadata_kv_count = read_length(reader, &magic)?;++ if tensor_count > GGUF_MAX_ARRAY_ELEMENTS {+ crate::bail!("gguf: tensor_count {tensor_count} exceeds max {GGUF_MAX_ARRAY_ELEMENTS}")+ }+ if metadata_kv_count > GGUF_MAX_ARRAY_ELEMENTS {+ crate::bail!(+ "gguf: metadata_kv_count {metadata_kv_count} exceeds max {GGUF_MAX_ARRAY_ELEMENTS}"+ )+ }++ // Reject header-declared counts that can't fit in the file at minimum size.+ // Per-entry minima: a metadata kv is at least `key_len_prefix + u32 value_type+ // + 1 byte value`; a tensor info is at least `name_len_prefix + u32 n_dims+ // + u32 dtype + u64 offset`.+ let prefix = magic.length_prefix_size();+ let min_per_kv = prefix + 4 + 1;+ let min_per_tensor = prefix + 4 + 4 + 8;+ let needed = metadata_kv_count+ .saturating_mul(min_per_kv)+ .saturating_add(tensor_count.saturating_mul(min_per_tensor));+ let remaining = remaining_bytes(reader, file_size)?;+ if needed > remaining {+ crate::bail!(+ "gguf: header declares {tensor_count} tensors and {metadata_kv_count} metadata entries, needs at least {needed} bytes, only {remaining} remaining"+ )+ }++ let mut metadata = HashMap::new();+ for _idx in 0..metadata_kv_count {+ let key = read_string(reader, &magic, file_size)?;+ let value_type = reader.read_u32::<LittleEndian>()?;+ let value_type = ValueType::from_u32(value_type)?;+ let value = Value::read(reader, value_type, &magic, 0, file_size)?;+ metadata.insert(key, value);+ }+ let mut tensor_infos = HashMap::new();+ for _idx in 0..tensor_count {+ let tensor_name = read_string(reader, &magic, file_size)?;+ let n_dimensions = reader.read_u32::<LittleEndian>()?;+ if n_dimensions > GGUF_MAX_TENSOR_DIMS {+ crate::bail!(+ "gguf: tensor '{tensor_name}' has {n_dimensions} dimensions, max is {GGUF_MAX_TENSOR_DIMS}"+ )+ }++ let mut dimensions: Vec<usize> = match magic {+ VersionedMagic::GgufV1 => {+ let mut dimensions = vec![0; n_dimensions as usize];+ reader.read_u32_into::<LittleEndian>(&mut dimensions)?;+ dimensions.into_iter().map(|c| c as usize).collect()+ }+ VersionedMagic::GgufV2 | VersionedMagic::GgufV3 => {+ let mut dimensions = vec![0; n_dimensions as usize];+ reader.read_u64_into::<LittleEndian>(&mut dimensions)?;+ dimensions.into_iter().map(|c| c as usize).collect()+ }+ };++ dimensions.reverse();+ let ggml_dtype = reader.read_u32::<LittleEndian>()?;+ let ggml_dtype = GgmlDType::from_u32(ggml_dtype)?;+ let offset = reader.read_u64::<LittleEndian>()?;+ tensor_infos.insert(+ tensor_name,+ TensorInfo {+ shape: crate::Shape::from(dimensions),+ offset,+ ggml_dtype,+ },+ );+ }+ let position = reader.stream_position()?;+ let alignment = match metadata.get("general.alignment") {+ Some(Value::U8(v)) => *v as u64,+ Some(Value::U16(v)) => *v as u64,+ Some(Value::U32(v)) => *v as u64,+ Some(Value::I8(v)) if *v >= 0 => *v as u64,+ Some(Value::I16(v)) if *v >= 0 => *v as u64,+ Some(Value::I32(v)) if *v >= 0 => *v as u64,+ _ => DEFAULT_ALIGNMENT,+ };+ let tensor_data_offset = position.div_ceil(alignment) * alignment;+ Ok(Self {+ magic,+ metadata,+ tensor_infos,+ tensor_data_offset,+ })+ }++ pub fn tensor<R: std::io::Seek + std::io::Read>(+ &self,+ reader: &mut R,+ name: &str,+ device: &Device,+ ) -> Result<QTensor> {+ let tensor_info = match self.tensor_infos.get(name) {+ Some(tensor_info) => tensor_info,+ None => crate::bail!("cannot find tensor info for {name}"),+ };+ tensor_info.read(reader, self.tensor_data_offset, device)+ }+}++fn write_string<W: std::io::Write>(w: &mut W, str: &str) -> Result<()> {+ let bytes = str.as_bytes();+ w.write_u64::<LittleEndian>(bytes.len() as u64)?;+ w.write_all(bytes)?;+ Ok(())+}++pub fn write<W: std::io::Seek + std::io::Write>(+ w: &mut W,+ metadata: &[(&str, &Value)],+ tensors: &[(&str, &QTensor)],+) -> Result<()> {+ w.write_u32::<LittleEndian>(0x46554747)?;+ w.write_u32::<LittleEndian>(2)?; // version 2.+ w.write_u64::<LittleEndian>(tensors.len() as u64)?;+ w.write_u64::<LittleEndian>(metadata.len() as u64)?;+ for (name, value) in metadata.iter() {+ write_string(w, name)?;+ w.write_u32::<LittleEndian>(value.value_type().to_u32())?;+ value.write(w)?;+ }+ let mut offset = 0usize;+ let mut offsets = Vec::with_capacity(tensors.len());+ for (name, tensor) in tensors.iter() {+ write_string(w, name)?;+ let dims = tensor.shape().dims();+ w.write_u32::<LittleEndian>(dims.len() as u32)?;+ for &dim in dims.iter().rev() {+ w.write_u64::<LittleEndian>(dim as u64)?;+ }+ w.write_u32::<LittleEndian>(tensor.dtype().to_u32())?;+ w.write_u64::<LittleEndian>(offset as u64)?;+ offsets.push(offset);+ let size_in_bytes = tensor.storage_size_in_bytes();+ let padding = 31 - (31 + size_in_bytes) % 32;+ offset += size_in_bytes + padding;+ }+ let pos = w.stream_position()? as usize;+ let padding = 31 - (31 + pos) % 32;+ w.write_all(&vec![0u8; padding])?;+ let tensor_start_pos = w.stream_position()? as usize;+ for (offset, (_name, tensor)) in offsets.iter().zip(tensors.iter()) {+ let pos = w.stream_position()? as usize;+ if tensor_start_pos + offset != pos {+ crate::bail!(+ "internal error, unexpected current position {tensor_start_pos} {offset} {pos}"+ )+ }+ let data = tensor.data()?;+ let size_in_bytes = data.len();+ w.write_all(&data)?;+ let padding = 31 - (31 + size_in_bytes) % 32;+ w.write_all(&vec![0u8; padding])?;+ }+ Ok(())+}diff --git a/vendor/candle-core/src/quantized/imatrix_file.rs b/vendor/candle-core/src/quantized/imatrix_file.rsnew file mode 100644index 0000000..ed228b7--- /dev/null+++ b/vendor/candle-core/src/quantized/imatrix_file.rs@@ -0,0 +1,85 @@+use std::collections::HashMap;+use std::fs::File;+use std::io::{Cursor, Read};+use std::path::Path;++use byteorder::{LittleEndian, ReadBytesExt};++use crate::Result;++pub fn load_imatrix<P: AsRef<Path>>(fname: P) -> Result<HashMap<String, Vec<f32>>> {+ let mut all_data = HashMap::new();++ let mut file = File::open(&fname).map_err(|e| {+ crate::Error::msg(format!(+ "Failed to open {}: {}",+ fname.as_ref().display(),+ e+ ))+ })?;+ let mut buffer = Vec::new();+ file.read_to_end(&mut buffer).map_err(|e| {+ crate::Error::msg(format!(+ "Failed to read file {}: {}",+ fname.as_ref().display(),+ e+ ))+ })?;++ let mut cursor = Cursor::new(buffer);++ let n_entries = cursor+ .read_i32::<LittleEndian>()+ .map_err(|e| crate::Error::msg(format!("Failed to read number of entries: {e}")))?+ as usize;++ if n_entries < 1 {+ crate::bail!("No data in file {}", fname.as_ref().display());+ }++ for i in 0..n_entries {+ // Read length of the name+ let len = cursor.read_i32::<LittleEndian>().map_err(|e| {+ crate::Error::msg(format!(+ "Failed to read name length for entry {}: {}",+ i + 1,+ e+ ))+ })? as usize;++ // Read the name+ let mut name_buf = vec![0u8; len];+ cursor.read_exact(&mut name_buf).map_err(|e| {+ crate::Error::msg(format!("Failed to read name for entry {}: {}", i + 1, e))+ })?;+ let name = String::from_utf8(name_buf).map_err(|e| {+ crate::Error::msg(format!("Invalid UTF-8 name for entry {}: {}", i + 1, e))+ })?;++ // Read ncall and nval+ let ncall = cursor.read_i32::<LittleEndian>().map_err(|e| {+ crate::Error::msg(format!("Failed to read ncall for entry {}: {}", i + 1, e))+ })? as usize;++ let nval = cursor.read_i32::<LittleEndian>().map_err(|e| {+ crate::Error::msg(format!("Failed to read nval for entry {}: {}", i + 1, e))+ })? as usize;++ if nval < 1 {+ crate::bail!("Invalid nval for entry {}: {}", i + 1, nval);+ }++ let mut data = Vec::with_capacity(nval);+ for _ in 0..nval {+ let v = cursor.read_f32::<LittleEndian>().unwrap();+ if ncall == 0 {+ data.push(v);+ } else {+ data.push(v / ncall as f32);+ }+ }+ all_data.insert(name, data);+ }++ Ok(all_data)+}diff --git a/vendor/candle-core/src/quantized/k_quants.rs b/vendor/candle-core/src/quantized/k_quants.rsnew file mode 100644index 0000000..bccfedd--- /dev/null+++ b/vendor/candle-core/src/quantized/k_quants.rs@@ -0,0 +1,2953 @@+use super::utils::{+ get_scale_min_k4, group_for_dequantization, group_for_quantization, make_q3_quants,+ make_qkx1_quants, make_qx_quants, nearest_int,+};+use super::GgmlDType;+use crate::quantized::utils::{make_qkx3_quants, make_qp_quants};+use crate::Result;+use byteorder::{ByteOrder, LittleEndian};+use half::{bf16, f16, slice::HalfFloatSliceExt};++#[cfg(target_arch = "aarch64")]+use super::repack::BlockQ4Kx8;++#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+fn has_avx2_fma() -> bool {+ let features = crate::cpu::features::get();+ features.avx2 && features.fma+}++// Default to QK_K 256 rather than 64.+pub const QK_K: usize = 256;+pub const K_SCALE_SIZE: usize = 12;++pub const QK4_0: usize = 32;+pub const QK4_1: usize = 32;+pub const QK5_0: usize = 32;+pub const QK5_1: usize = 32;+pub const QK8_0: usize = 32;+pub const QK8_1: usize = 32;+pub const QK_MXFP4: usize = 32;++pub trait GgmlType: Sized + Clone + Send + Sync {+ const DTYPE: GgmlDType;+ const BLCK_SIZE: usize;+ const DIRECT_COPY: bool = false;+ type VecDotType: GgmlType;++ // This is only safe for types that include immediate values such as float/int/...+ fn zeros() -> Self {+ unsafe { std::mem::MaybeUninit::zeroed().assume_init() }+ }+ fn to_float(xs: &[Self], ys: &mut [f32]);+ fn from_float(xs: &[f32], ys: &mut [Self]);+ fn from_float_imatrix(+ _xs: &[f32],+ _ys: &mut [Self],+ _imatrix_weights: &[f32],+ _n_per_row: usize,+ ) {+ panic!(+ "`from_float_imatrix` is unimplemented for {:?}",+ Self::DTYPE+ );+ }++ fn direct_copy(_xs: &[f32], _ys: &mut [Self]) {}++ /// Dot product used as a building block for quantized matmul.+ /// n is the number of elements to be considered.+ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32;++ /// Two dot products sharing one LHS load: returns (dot(xs0,ys), dot(xs1,ys)).+ /// Can be overriden with optimized kernel+ fn vec_dot_2(n: usize, xs0: &[Self], xs1: &[Self], ys: &[Self::VecDotType]) -> (f32, f32) {+ (Self::vec_dot(n, xs0, ys), Self::vec_dot(n, xs1, ys))+ }++ /// Four dot products sharing one LHS load: returns (dot(xs0,ys),..dot(xs3,ys)).+ /// Can be overriden with optimized kernel+ fn vec_dot_4(+ n: usize,+ xs0: &[Self],+ xs1: &[Self],+ xs2: &[Self],+ xs3: &[Self],+ ys: &[Self::VecDotType],+ ) -> (f32, f32, f32, f32) {+ let (d0, d1) = Self::vec_dot_2(n, xs0, xs1, ys);+ let (d2, d3) = Self::vec_dot_2(n, xs2, xs3, ys);+ (d0, d1, d2, d3)+ }++ /// Generic implementation of the dot product without simd optimizations.+ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32;+}++#[derive(Debug, Clone, PartialEq)]+#[repr(C)]+pub struct BlockQ4_0 {+ pub(crate) d: f16,+ pub(crate) qs: [u8; QK4_0 / 2],+}+const _: () = assert!(std::mem::size_of::<BlockQ4_0>() == 18);++#[derive(Debug, Clone, PartialEq)]+#[repr(C)]+pub struct BlockQ4_1 {+ pub(crate) d: f16,+ pub(crate) m: f16,+ pub(crate) qs: [u8; QK4_1 / 2],+}+const _: () = assert!(std::mem::size_of::<BlockQ4_1>() == 20);++#[derive(Debug, Clone, PartialEq)]+#[repr(C)]+pub struct BlockQ5_0 {+ pub(crate) d: f16,+ pub(crate) qh: [u8; 4],+ pub(crate) qs: [u8; QK5_0 / 2],+}+const _: () = assert!(std::mem::size_of::<BlockQ5_0>() == 22);++#[derive(Debug, Clone, PartialEq)]+#[repr(C)]+pub struct BlockQ5_1 {+ pub(crate) d: f16,+ pub(crate) m: f16,+ pub(crate) qh: [u8; 4],+ pub(crate) qs: [u8; QK5_1 / 2],+}+const _: () = assert!(std::mem::size_of::<BlockQ5_1>() == 24);++#[derive(Debug, Clone, PartialEq)]+#[repr(C)]+pub struct BlockQ8_0 {+ pub(crate) d: f16,+ pub(crate) qs: [i8; QK8_0],+}+const _: () = assert!(std::mem::size_of::<BlockQ8_0>() == 34);++#[derive(Debug, Clone, PartialEq)]+#[repr(C)]+pub struct BlockQ8_1 {+ pub(crate) d: f16,+ pub(crate) s: f16,+ pub(crate) qs: [i8; QK8_1],+}+const _: () = assert!(std::mem::size_of::<BlockQ8_1>() == 36);++#[derive(Debug, Clone, PartialEq)]+#[repr(C)]+pub struct BlockQ2K {+ pub(crate) scales: [u8; QK_K / 16],+ pub(crate) qs: [u8; QK_K / 4],+ pub(crate) d: f16,+ pub(crate) dmin: f16,+}+const _: () = assert!(QK_K / 16 + QK_K / 4 + 2 * 2 == std::mem::size_of::<BlockQ2K>());++#[derive(Debug, Clone, PartialEq)]+#[repr(C)]+pub struct BlockQ3K {+ pub(crate) hmask: [u8; QK_K / 8],+ pub(crate) qs: [u8; QK_K / 4],+ pub(crate) scales: [u8; 12],+ pub(crate) d: f16,+}+const _: () = assert!(QK_K / 8 + QK_K / 4 + 12 + 2 == std::mem::size_of::<BlockQ3K>());++#[derive(Debug, Clone, PartialEq)]+// https://github.com/ggerganov/llama.cpp/blob/468ea24fb4633a0d681f7ac84089566c1c6190cb/k_quants.h#L82+#[repr(C)]+pub struct BlockQ4K {+ pub(crate) d: f16,+ pub(crate) dmin: f16,+ pub(crate) scales: [u8; K_SCALE_SIZE],+ pub(crate) qs: [u8; QK_K / 2],+}+const _: () = assert!(QK_K / 2 + K_SCALE_SIZE + 2 * 2 == std::mem::size_of::<BlockQ4K>());++#[derive(Debug, Clone, PartialEq)]+#[repr(C)]+pub struct BlockQ5K {+ pub(crate) d: f16,+ pub(crate) dmin: f16,+ pub(crate) scales: [u8; K_SCALE_SIZE],+ pub(crate) qh: [u8; QK_K / 8],+ pub(crate) qs: [u8; QK_K / 2],+}+const _: () =+ assert!(QK_K / 8 + QK_K / 2 + 2 * 2 + K_SCALE_SIZE == std::mem::size_of::<BlockQ5K>());++#[derive(Debug, Clone, PartialEq)]+#[repr(C)]+pub struct BlockQ6K {+ pub(crate) ql: [u8; QK_K / 2],+ pub(crate) qh: [u8; QK_K / 4],+ pub(crate) scales: [i8; QK_K / 16],+ pub(crate) d: f16,+}+const _: () = assert!(3 * QK_K / 4 + QK_K / 16 + 2 == std::mem::size_of::<BlockQ6K>());++#[derive(Debug, Clone, PartialEq)]+#[repr(C)]+pub struct BlockQ8K {+ pub(crate) d: f32,+ pub(crate) qs: [i8; QK_K],+ pub(crate) bsums: [i16; QK_K / 16],+}+const _: () = assert!(4 + QK_K + QK_K / 16 * 2 == std::mem::size_of::<BlockQ8K>());++impl GgmlType for BlockQ4_0 {+ const DTYPE: GgmlDType = GgmlDType::Q4_0;+ const BLCK_SIZE: usize = QK4_0;+ type VecDotType = BlockQ8_0;++ // https://github.com/ggerganov/llama.cpp/blob/468ea24fb4633a0d681f7ac84089566c1c6190cb/ggml.c#L1525+ fn to_float(xs: &[Self], ys: &mut [f32]) {+ let k = ys.len();+ let qk = Self::BLCK_SIZE;+ debug_assert!(+ k.is_multiple_of(qk),+ "dequantize_row_q4_0: {k} is not divisible by {qk}"+ );++ let nb = k / qk;+ for i in 0..nb {+ let d = xs[i].d.to_f32();++ for j in 0..(qk / 2) {+ let x0 = (xs[i].qs[j] & 0x0F) as i16 - 8;+ let x1 = (xs[i].qs[j] >> 4) as i16 - 8;++ ys[i * qk + j] = (x0 as f32) * d;+ ys[i * qk + j + qk / 2] = (x1 as f32) * d;+ }+ }+ }++ fn from_float(xs: &[f32], ys: &mut [Self]) {+ // quantize_row_q4_0+ let qk = Self::BLCK_SIZE;+ let k = xs.len();+ debug_assert!(k.is_multiple_of(qk), "{k} is not divisible by {qk}");+ debug_assert_eq!(+ ys.len(),+ k / qk,+ "size mismatch {} {} {}",+ xs.len(),+ ys.len(),+ qk,+ );+ for (i, ys) in ys.iter_mut().enumerate() {+ let mut amax = 0f32;+ let mut max = 0f32;++ let xs = &xs[i * qk..(i + 1) * qk];+ for &x in xs.iter() {+ if amax < x.abs() {+ amax = x.abs();+ max = x;+ }+ }+ let d = max / -8.0;+ let id = if d != 0f32 { 1. / d } else { 0. };+ ys.d = f16::from_f32(d);++ for (j, q) in ys.qs.iter_mut().enumerate() {+ let x0 = xs[j] * id;+ let x1 = xs[qk / 2 + j] * id;+ let xi0 = u8::min(15, (x0 + 8.5) as u8);+ let xi1 = u8::min(15, (x1 + 8.5) as u8);+ *q = xi0 | (xi1 << 4)+ }+ }+ }++ // https://github.com/ggerganov/llama.cpp/blob/b5ffb2849d23afe73647f68eec7b68187af09be6/ggml.c#L2361C10-L2361C122+ #[allow(unreachable_code)]+ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if has_avx2_fma() {+ return unsafe { super::avx::vec_dot_q4_0_q8_0(n, xs, ys) };+ }++ #[cfg(target_feature = "neon")]+ return super::neon::vec_dot_q4_0_q8_0(n, xs, ys);++ #[cfg(target_feature = "simd128")]+ return super::simd128::vec_dot_q4_0_q8_0(n, xs, ys);++ Self::vec_dot_unopt(n, xs, ys)+ }++ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK8_0),+ "vec_dot_q4_0_q8_0: {n} is not divisible by {QK8_0}"+ );+ // Generic implementation.+ let mut sumf = 0f32;+ for (xs, ys) in xs.iter().zip(ys.iter()) {+ let mut sum_i = 0;+ for j in 0..QK8_0 / 2 {+ let v0 = (xs.qs[j] & 0x0F) as i32 - 8;+ let v1 = (xs.qs[j] >> 4) as i32 - 8;+ sum_i += v0 * ys.qs[j] as i32 + v1 * ys.qs[j + QK8_0 / 2] as i32+ }+ sumf += sum_i as f32 * f16::to_f32(xs.d) * f16::to_f32(ys.d)+ }+ sumf+ }+}++impl GgmlType for BlockQ4_1 {+ const DTYPE: GgmlDType = GgmlDType::Q4_1;+ const BLCK_SIZE: usize = QK4_1;+ type VecDotType = BlockQ8_1;++ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ Self::vec_dot_unopt(n, xs, ys)+ }++ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ // ggml_vec_dot_q4_1_q8_1+ let qk = QK8_1;+ debug_assert!(+ n.is_multiple_of(qk),+ "vec_dot_q4_1_q8_1: {n} is not divisible by {qk}"+ );+ debug_assert!(+ (n / qk).is_multiple_of(2),+ "vec_dot_q4_1_q8_1: {n}, nb is not divisible by 2"+ );++ // Generic implementation.+ let mut sumf = 0f32;++ for (xs, ys) in xs.iter().zip(ys.iter()) {+ let mut sumi = 0i32;++ for j in 0..qk / 2 {+ let v0 = xs.qs[j] as i32 & 0x0F;+ let v1 = xs.qs[j] as i32 >> 4;+ sumi += (v0 * ys.qs[j] as i32) + (v1 * ys.qs[j + qk / 2] as i32);+ }++ sumf += sumi as f32 * f16::to_f32(xs.d) * f16::to_f32(ys.d)+ + f16::to_f32(xs.m) * f16::to_f32(ys.s)+ }+ sumf+ }++ fn from_float(xs: &[f32], ys: &mut [Self]) {+ // quantize_row_q4_1+ let qk = Self::BLCK_SIZE;++ debug_assert_eq!(+ ys.len() * qk,+ xs.len(),+ "size mismatch {} {} {}",+ xs.len(),+ ys.len(),+ qk,+ );+ for (i, ys) in ys.iter_mut().enumerate() {+ let xs = &xs[i * qk..(i + 1) * qk];++ let mut min = f32::INFINITY;+ let mut max = f32::NEG_INFINITY;+ for &x in xs.iter() {+ min = f32::min(x, min);+ max = f32::max(x, max);+ }+ let d = (max - min) / ((1 << 4) - 1) as f32;+ let id = if d != 0f32 { 1. / d } else { 0. };+ ys.d = f16::from_f32(d);+ ys.m = f16::from_f32(min);++ for (j, q) in ys.qs.iter_mut().take(qk / 2).enumerate() {+ let x0 = (xs[j] - min) * id;+ let x1 = (xs[qk / 2 + j] - min) * id;++ let xi0 = u8::min(15, (x0 + 0.5) as u8);+ let xi1 = u8::min(15, (x1 + 0.5) as u8);++ *q = xi0 | (xi1 << 4);+ }+ }+ }++ // https://github.com/ggerganov/llama.cpp/blob/468ea24fb4633a0d681f7ac84089566c1c6190cb/ggml.c#L1545+ fn to_float(xs: &[Self], ys: &mut [f32]) {+ let k = ys.len();+ debug_assert!(+ k.is_multiple_of(QK4_1),+ "dequantize_row_q4_1: {k} is not divisible by {QK4_1}"+ );++ let nb = k / QK4_1;+ for i in 0..nb {+ let d = xs[i].d.to_f32();+ let m = xs[i].m.to_f32();++ for j in 0..(QK4_1 / 2) {+ let x0 = xs[i].qs[j] & 0x0F;+ let x1 = xs[i].qs[j] >> 4;++ ys[i * QK4_1 + j] = (x0 as f32) * d + m;+ ys[i * QK4_1 + j + QK4_1 / 2] = (x1 as f32) * d + m;+ }+ }+ }+}++impl GgmlType for BlockQ5_0 {+ const DTYPE: GgmlDType = GgmlDType::Q5_0;+ const BLCK_SIZE: usize = QK5_0;+ type VecDotType = BlockQ8_0;++ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ let qk = Self::BLCK_SIZE;++ debug_assert!(+ n.is_multiple_of(qk),+ "vec_dot_q5_0_q8_0: {n} is not divisible by {qk}"+ );+ debug_assert!(+ (n / qk).is_multiple_of(2),+ "vec_dot_q5_0_q8_0: {n}, nb is not divisible by 2"+ );+ Self::vec_dot_unopt(n, xs, ys)+ }++ fn vec_dot_unopt(_n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ // Generic implementation.+ let mut sumf = 0f32;++ for (xs, ys) in xs.iter().zip(ys.iter()) {+ let qh = LittleEndian::read_u32(&xs.qh);+ let mut sumi = 0i32;++ for j in 0..Self::BLCK_SIZE / 2 {+ let xh_0 = (((qh & (1u32 << j)) >> j) << 4) as u8;+ let xh_1 = ((qh & (1u32 << (j + 16))) >> (j + 12)) as u8;++ let x0 = ((xs.qs[j] & 0x0F) as i32 | xh_0 as i32) - 16;+ let x1 = ((xs.qs[j] >> 4) as i32 | xh_1 as i32) - 16;++ sumi += (x0 * ys.qs[j] as i32) + (x1 * ys.qs[j + Self::BLCK_SIZE / 2] as i32);+ }++ sumf += sumi as f32 * f16::to_f32(xs.d) * f16::to_f32(ys.d)+ }+ sumf+ }++ fn from_float(xs: &[f32], ys: &mut [Self]) {+ // quantize_row_q5_0+ debug_assert_eq!(+ ys.len() * Self::BLCK_SIZE,+ xs.len(),+ "size mismatch {} {} {}",+ xs.len(),+ ys.len(),+ Self::BLCK_SIZE,+ );+ for (i, ys) in ys.iter_mut().enumerate() {+ let xs = &xs[i * Self::BLCK_SIZE..(i + 1) * Self::BLCK_SIZE];++ let mut amax = 0f32;+ let mut max = 0f32;+ for &x in xs.iter() {+ if amax < x.abs() {+ amax = x.abs();+ max = x;+ }+ }+ let d = max / -16.;+ let id = if d != 0f32 { 1. / d } else { 0. };+ ys.d = f16::from_f32(d);+ let mut qh = 0u32;+ for j in 0..Self::BLCK_SIZE / 2 {+ let x0 = xs[j] * id;+ let x1 = xs[j + Self::BLCK_SIZE / 2] * id;+ let xi0 = ((x0 + 16.5) as i8).min(31) as u8;+ let xi1 = ((x1 + 16.5) as i8).min(31) as u8;+ ys.qs[j] = (xi0 & 0x0F) | ((xi1 & 0x0F) << 4);+ qh |= ((xi0 as u32 & 0x10) >> 4) << j;+ qh |= ((xi1 as u32 & 0x10) >> 4) << (j + Self::BLCK_SIZE / 2);+ }+ LittleEndian::write_u32(&mut ys.qh, qh)+ }+ }++ // https://github.com/ggerganov/llama.cpp/blob/468ea24fb4633a0d681f7ac84089566c1c6190cb/ggml.c#L1566+ fn to_float(xs: &[Self], ys: &mut [f32]) {+ let k = ys.len();+ debug_assert!(+ k.is_multiple_of(QK5_0),+ "dequantize_row_q5_0: {k} is not divisible by {QK5_0}"+ );+ let nb = k / QK5_0;+ for i in 0..nb {+ let d = xs[i].d.to_f32();+ let qh: u32 = LittleEndian::read_u32(&xs[i].qh);++ for j in 0..(QK5_0 / 2) {+ let xh_0 = (((qh >> j) << 4) & 0x10) as u8;+ let xh_1 = ((qh >> (j + 12)) & 0x10) as u8;++ let x0 = ((xs[i].qs[j] & 0x0F) | xh_0) as i32 - 16;+ let x1 = ((xs[i].qs[j] >> 4) | xh_1) as i32 - 16;++ ys[i * QK5_0 + j] = (x0 as f32) * d;+ ys[i * QK5_0 + j + QK5_0 / 2] = (x1 as f32) * d;+ }+ }+ }+}++impl GgmlType for BlockQ5_1 {+ const DTYPE: GgmlDType = GgmlDType::Q5_1;+ const BLCK_SIZE: usize = QK5_1;+ type VecDotType = BlockQ8_1;++ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ Self::vec_dot_unopt(n, xs, ys)+ }++ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ let qk = Self::BLCK_SIZE;+ debug_assert!(+ n.is_multiple_of(qk),+ "vec_dot_q5_1_q8_1: {n} is not divisible by {qk}"+ );+ debug_assert!(+ (n / qk).is_multiple_of(2),+ "vec_dot_q5_1_q8_1: {n}, nb is not divisible by 2"+ );++ // Generic implementation.+ let mut sumf = 0f32;++ for (xs, ys) in xs.iter().zip(ys.iter()) {+ let qh = LittleEndian::read_u32(&xs.qh);+ let mut sumi = 0i32;++ for j in 0..Self::BLCK_SIZE / 2 {+ let xh_0 = ((qh >> j) << 4) & 0x10;+ let xh_1 = (qh >> (j + 12)) & 0x10;++ let x0 = (xs.qs[j] as i32 & 0xF) | xh_0 as i32;+ let x1 = (xs.qs[j] as i32 >> 4) | xh_1 as i32;++ sumi += (x0 * ys.qs[j] as i32) + (x1 * ys.qs[j + Self::BLCK_SIZE / 2] as i32);+ }++ sumf += sumi as f32 * f16::to_f32(xs.d) * f16::to_f32(ys.d)+ + f16::to_f32(xs.m) * f16::to_f32(ys.s)+ }+ sumf+ }++ fn from_float(xs: &[f32], ys: &mut [Self]) {+ // quantize_row_q5_1+ let qk = Self::BLCK_SIZE;+ debug_assert_eq!(+ ys.len() * qk,+ xs.len(),+ "size mismatch {} {} {}",+ xs.len(),+ ys.len(),+ qk,+ );+ for (i, ys) in ys.iter_mut().enumerate() {+ let xs = &xs[i * qk..(i + 1) * qk];++ let mut min = f32::INFINITY;+ let mut max = f32::NEG_INFINITY;+ for &x in xs.iter() {+ min = f32::min(x, min);+ max = f32::max(x, max);+ }+ let d = (max - min) / ((1 << 5) - 1) as f32;+ let id = if d != 0f32 { 1. / d } else { 0. };+ ys.d = f16::from_f32(d);+ ys.m = f16::from_f32(min);++ let mut qh = 0u32;+ for (j, q) in ys.qs.iter_mut().take(qk / 2).enumerate() {+ let x0 = (xs[j] - min) * id;+ let x1 = (xs[qk / 2 + j] - min) * id;++ let xi0 = (x0 + 0.5) as u8;+ let xi1 = (x1 + 0.5) as u8;++ *q = (xi0 & 0x0F) | ((xi1 & 0x0F) << 4);+ // get the 5-th bit and store it in qh at the right position+ qh |= ((xi0 as u32 & 0x10) >> 4) << j;+ qh |= ((xi1 as u32 & 0x10) >> 4) << (j + qk / 2);+ }+ LittleEndian::write_u32(&mut ys.qh, qh);+ }+ }++ // https://github.com/ggerganov/llama.cpp/blob/468ea24fb4633a0d681f7ac84089566c1c6190cb/ggml.c#L1592+ fn to_float(xs: &[Self], ys: &mut [f32]) {+ let k = ys.len();+ debug_assert!(+ k.is_multiple_of(QK5_1),+ "dequantize_row_q5_1: {k} is not divisible by {QK5_1}"+ );++ let nb = k / QK5_1;+ for i in 0..nb {+ let d = xs[i].d.to_f32();+ let m = xs[i].m.to_f32();+ let qh: u32 = LittleEndian::read_u32(&xs[i].qh);++ for j in 0..(QK5_1 / 2) {+ let xh_0 = (((qh >> j) << 4) & 0x10) as u8;+ let xh_1 = ((qh >> (j + 12)) & 0x10) as u8;++ let x0 = (xs[i].qs[j] & 0x0F) | xh_0;+ let x1 = (xs[i].qs[j] >> 4) | xh_1;++ ys[i * QK5_1 + j] = (x0 as f32) * d + m;+ ys[i * QK5_1 + j + QK5_1 / 2] = (x1 as f32) * d + m;+ }+ }+ }+}++// MXFP4 (microscaling FP4, OCP MX spec): a block of 32 elements stored as one+// E8M0 exponent byte plus 16 bytes of packed 4-bit E2M1 codes.+// See https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf+// and ggml's block_mxfp4 (ggml-common.h).+#[derive(Debug, Clone, PartialEq)]+#[repr(C)]+pub struct BlockMxfp4 {+ pub(crate) e: u8,+ pub(crate) qs: [u8; QK_MXFP4 / 2],+}+const _: () = assert!(std::mem::size_of::<BlockMxfp4>() == 17);++// E2M1 values doubled, shared by MXFP4 and NVFP4 (ggml-common.h kvalues_fp4).+const KVALUES_MXFP4: [i8; 16] = [0, 1, 2, 3, 4, 6, 8, 12, 0, -1, -2, -3, -4, -6, -8, -12];++// ggml_e8m0_to_fp32_half: 0.5 * 2^(e - 127) = 2^(e - 128), used with the doubled+// kvalues above so that kvalues[d] * d gives the exact E2M1 * E8M0 product.+fn e8m0_to_f32_half(e: u8) -> f32 {+ let bits = if e < 2 {+ 0x0020_0000u32 << e+ } else {+ ((e - 1) as u32) << 23+ };+ f32::from_bits(bits)+}++// best_index_mxfp4 from ggml-quants.c: nearest kvalues[i] * e to x.+fn best_index_mxfp4(x: f32, e: f32) -> u8 {+ let mut best_index = 0u8;+ let mut best_err = (KVALUES_MXFP4[0] as f32 * e - x).abs();+ for (i, &k) in KVALUES_MXFP4.iter().enumerate().skip(1) {+ let err = (k as f32 * e - x).abs();+ if err < best_err {+ best_index = i as u8;+ best_err = err;+ }+ }+ best_index+}++impl GgmlType for BlockMxfp4 {+ const DTYPE: GgmlDType = GgmlDType::Mxfp4;+ const BLCK_SIZE: usize = QK_MXFP4;+ // llama.cpp pairs MXFP4 weights with Q8_0-quantized activations+ // (ggml_vec_dot_mxfp4_q8_0).+ type VecDotType = BlockQ8_0;++ // dequantize_row_mxfp4 from ggml-quants.c+ fn to_float(xs: &[Self], ys: &mut [f32]) {+ let k = ys.len();+ debug_assert!(+ k.is_multiple_of(QK_MXFP4),+ "dequantize_row_mxfp4: {k} is not divisible by {QK_MXFP4}"+ );++ let nb = k / QK_MXFP4;++ for i in 0..nb {+ let d = e8m0_to_f32_half(xs[i].e);++ for j in 0..QK_MXFP4 / 2 {+ ys[i * QK_MXFP4 + j] = KVALUES_MXFP4[(xs[i].qs[j] & 0x0F) as usize] as f32 * d;+ ys[i * QK_MXFP4 + j + QK_MXFP4 / 2] =+ KVALUES_MXFP4[(xs[i].qs[j] >> 4) as usize] as f32 * d;+ }+ }+ }++ // quantize_row_mxfp4_ref from ggml-quants.c+ fn from_float(xs: &[f32], ys: &mut [Self]) {+ let k = xs.len();+ debug_assert!(+ k.is_multiple_of(Self::BLCK_SIZE),+ "{k} is not divisible by {}",+ Self::BLCK_SIZE+ );+ debug_assert_eq!(+ ys.len(),+ k / Self::BLCK_SIZE,+ "size mismatch {} {} {}",+ xs.len(),+ ys.len(),+ Self::BLCK_SIZE+ );+ for (i, y) in ys.iter_mut().enumerate() {+ let xs = &xs[i * Self::BLCK_SIZE..(i + 1) * Self::BLCK_SIZE];++ let mut amax = 0f32;+ for &x in xs.iter() {+ amax = amax.max(x.abs());+ }++ let e = if amax > 0.0 {+ (amax.log2().floor() - 2.0 + 127.0) as u8+ } else {+ 0+ };+ y.e = e;++ let d = e8m0_to_f32_half(e);+ for j in 0..Self::BLCK_SIZE / 2 {+ let x0 = best_index_mxfp4(xs[j], d);+ let x1 = best_index_mxfp4(xs[j + Self::BLCK_SIZE / 2], d);+ y.qs[j] = x0 | (x1 << 4);+ }+ }+ }++ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ Self::vec_dot_unopt(n, xs, ys)+ }++ // ggml_vec_dot_mxfp4_q8_0_generic from ggml-cpu/quants.c+ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_MXFP4),+ "vec_dot_mxfp4_q8_0: {n} is not divisible by {QK_MXFP4}"+ );++ let mut sumf = 0f32;+ for (xs, ys) in xs.iter().zip(ys.iter()) {+ let d = f16::to_f32(ys.d) * e8m0_to_f32_half(xs.e);++ let mut sumi1 = 0i32;+ let mut sumi2 = 0i32;+ for j in 0..QK_MXFP4 / 2 {+ sumi1 += ys.qs[j] as i32 * KVALUES_MXFP4[(xs.qs[j] & 0x0F) as usize] as i32;+ sumi2 +=+ ys.qs[j + QK_MXFP4 / 2] as i32 * KVALUES_MXFP4[(xs.qs[j] >> 4) as usize] as i32;+ }+ sumf += d * (sumi1 + sumi2) as f32;+ }+ sumf+ }+}++impl GgmlType for BlockQ8_0 {+ const DTYPE: GgmlDType = GgmlDType::Q8_0;+ const BLCK_SIZE: usize = QK8_0;+ type VecDotType = BlockQ8_0;+ // https://github.com/ggerganov/llama.cpp/blob/468ea24fb4633a0d681f7ac84089566c1c6190cb/ggml.c#L1619+ fn to_float(xs: &[Self], ys: &mut [f32]) {+ let k = ys.len();+ debug_assert!(+ k.is_multiple_of(QK8_0),+ "dequantize_row_q8_0: {k} is not divisible by {QK8_0}"+ );++ let nb = k / QK8_0;++ for i in 0..nb {+ let d = xs[i].d.to_f32();++ for j in 0..QK8_0 {+ ys[i * QK8_0 + j] = xs[i].qs[j] as f32 * d;+ }+ }+ }++ fn from_float(xs: &[f32], ys: &mut [Self]) {+ #[cfg(all(target_arch = "aarch64", target_feature = "neon"))]+ {+ super::neon::quantize_q8_0(xs, ys);+ }++ #[cfg(not(all(target_arch = "aarch64", target_feature = "neon")))]+ {+ let k = xs.len();+ debug_assert!(+ k.is_multiple_of(Self::BLCK_SIZE),+ "{k} is not divisible by {}",+ Self::BLCK_SIZE+ );+ debug_assert_eq!(+ ys.len(),+ k / Self::BLCK_SIZE,+ "size mismatch {} {} {}",+ xs.len(),+ ys.len(),+ Self::BLCK_SIZE+ );+ for (i, ys) in ys.iter_mut().enumerate() {+ let mut amax = 0f32;+ let xs = &xs[i * Self::BLCK_SIZE..(i + 1) * Self::BLCK_SIZE];+ for &x in xs.iter() {+ amax = amax.max(x.abs())+ }+ let d = amax / ((1 << 7) - 1) as f32;+ let id = if d != 0f32 { 1. / d } else { 0. };+ ys.d = f16::from_f32(d);+ for (y, &x) in ys.qs.iter_mut().zip(xs.iter()) {+ *y = f32::round(x * id) as i8+ }+ }+ }+ }++ #[allow(unreachable_code)]+ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if has_avx2_fma() {+ return unsafe { super::avx::vec_dot_q8_0_q8_0(n, xs, ys) };+ }++ #[cfg(target_feature = "neon")]+ return super::neon::vec_dot_q8_0_q8_0(n, xs, ys);++ #[cfg(target_feature = "simd128")]+ return super::simd128::vec_dot_q8_0_q8_0(n, xs, ys);++ Self::vec_dot_unopt(n, xs, ys)+ }++ #[allow(unreachable_code)]+ fn vec_dot_4(+ n: usize,+ xs0: &[Self],+ xs1: &[Self],+ xs2: &[Self],+ xs3: &[Self],+ ys: &[Self::VecDotType],+ ) -> (f32, f32, f32, f32) {+ #[cfg(target_feature = "neon")]+ return super::neon::vec_dot_4_q8_0_q8_0(n, xs0, xs1, xs2, xs3, ys);++ let (d0, d1) = Self::vec_dot_2(n, xs0, xs1, ys);+ let (d2, d3) = Self::vec_dot_2(n, xs2, xs3, ys);+ (d0, d1, d2, d3)+ }++ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK8_0),+ "vec_dot_q8_0_q8_0: {n} is not divisible by {QK8_0}"+ );++ // Generic implementation.+ let mut sumf = 0f32;+ for (xs, ys) in xs.iter().zip(ys.iter()) {+ let sum_i = xs+ .qs+ .iter()+ .zip(ys.qs.iter())+ .map(|(&x, &y)| x as i32 * y as i32)+ .sum::<i32>();+ sumf += sum_i as f32 * f16::to_f32(xs.d) * f16::to_f32(ys.d)+ }+ sumf+ }+}++impl GgmlType for BlockQ8_1 {+ const DTYPE: GgmlDType = GgmlDType::Q8_1;+ const BLCK_SIZE: usize = QK8_1;+ type VecDotType = BlockQ8_1;++ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ Self::vec_dot_unopt(n, xs, ys)+ }++ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK8_1),+ "vec_dot_q8_1_q8_1: {n} is not divisible by {QK8_1}"+ );++ // Generic implementation.+ let mut sumf = 0f32;+ for (xs, ys) in xs.iter().zip(ys.iter()) {+ let sum_i = xs+ .qs+ .iter()+ .zip(ys.qs.iter())+ .map(|(&x, &y)| x as i32 * y as i32)+ .sum::<i32>();+ sumf += sum_i as f32 * f16::to_f32(xs.d) * f16::to_f32(ys.d)+ }+ sumf+ }++ fn from_float(xs: &[f32], ys: &mut [Self]) {+ // quantize_row_q8_1+ debug_assert_eq!(+ ys.len() * Self::BLCK_SIZE,+ xs.len(),+ "size mismatch {} {} {}",+ xs.len(),+ ys.len(),+ Self::BLCK_SIZE+ );+ for (i, ys) in ys.iter_mut().enumerate() {+ let mut amax = 0f32;+ let xs = &xs[i * Self::BLCK_SIZE..(i + 1) * Self::BLCK_SIZE];+ for &x in xs.iter() {+ amax = amax.max(x.abs())+ }+ let d = amax / ((1 << 7) - 1) as f32;+ let id = if d != 0f32 { 1. / d } else { 0. };+ ys.d = f16::from_f32(d);+ let mut sum = 0i32;+ for j in 0..Self::BLCK_SIZE / 2 {+ let v0 = xs[j] * id;+ let v1 = xs[j + Self::BLCK_SIZE / 2] * id;+ ys.qs[j] = f32::round(v0) as i8;+ ys.qs[j + Self::BLCK_SIZE / 2] = f32::round(v1) as i8;+ sum += ys.qs[j] as i32 + ys.qs[j + Self::BLCK_SIZE / 2] as i32;+ }+ ys.s = f16::from_f32(sum as f32) * ys.d;+ }+ }++ fn to_float(xs: &[Self], ys: &mut [f32]) {+ debug_assert_eq!(+ xs.len() * Self::BLCK_SIZE,+ ys.len(),+ "size mismatch {} {} {}",+ xs.len(),+ ys.len(),+ Self::BLCK_SIZE+ );+ let (y_chunks, _) = ys.as_chunks_mut::<{ Self::BLCK_SIZE }>();+ for (block, ys) in xs.iter().zip(y_chunks) {+ let d = block.d.to_f32();+ for (dst, &src) in ys.iter_mut().zip(block.qs.iter()) {+ *dst = src as f32 * d;+ }+ }+ }+}++impl GgmlType for BlockQ2K {+ const DTYPE: GgmlDType = GgmlDType::Q2K;+ const BLCK_SIZE: usize = QK_K;+ type VecDotType = BlockQ8K;++ #[allow(unreachable_code)]+ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if has_avx2_fma() {+ return unsafe { super::avx::vec_dot_q2k_q8k(n, xs, ys) };+ }++ #[cfg(target_feature = "neon")]+ return super::neon::vec_dot_q2k_q8k(n, xs, ys);++ #[cfg(target_feature = "simd128")]+ return super::simd128::vec_dot_q2k_q8k(n, xs, ys);++ Self::vec_dot_unopt(n, xs, ys)+ }++ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q2k_q8k: {n} is not divisible by {QK_K}"+ );++ let mut sumf = 0.0;+ for (x, y) in xs.iter().zip(ys.iter()) {+ let mut q2: &[_] = &x.qs;+ let mut q8: &[_] = &y.qs;+ let sc = &x.scales;++ let mut summs = 0;+ for (bsum, scale) in y.bsums.iter().zip(sc) {+ summs += *bsum as i32 * ((scale >> 4) as i32);+ }++ let dall = y.d * x.d.to_f32();+ let dmin = y.d * x.dmin.to_f32();++ let mut isum = 0;+ let mut is = 0;+ for _ in 0..(QK_K / 128) {+ let mut shift = 0;+ for _ in 0..4 {+ let d = (sc[is] & 0xF) as i32;+ is += 1;+ let mut isuml = 0;+ for l in 0..16 {+ isuml += q8[l] as i32 * (((q2[l] >> shift) & 3) as i32);+ }+ isum += d * isuml;+ let d = (sc[is] & 0xF) as i32;+ is += 1;+ isuml = 0;+ for l in 16..32 {+ isuml += q8[l] as i32 * (((q2[l] >> shift) & 3) as i32);+ }+ isum += d * isuml;+ shift += 2;+ // adjust the indexing+ q8 = &q8[32..];+ }+ // adjust the indexing+ q2 = &q2[32..];+ }+ sumf += dall * isum as f32 - dmin * summs as f32;+ }++ sumf+ }++ // https://github.com/ggerganov/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/k_quants.c#L279+ fn from_float(xs: &[f32], ys: &mut [Self]) {+ const Q4SCALE: f32 = 15.0;++ for (block, x) in group_for_quantization(xs, ys) {+ //calculate scales and mins+ let mut mins: [f32; QK_K / 16] = [0.0; QK_K / 16];+ let mut scales: [f32; QK_K / 16] = [0.0; QK_K / 16];++ for (j, x_scale_slice) in x.chunks(16).enumerate() {+ (scales[j], mins[j]) = make_qkx1_quants(3, 5, x_scale_slice);+ }+ // get max scale and max min and ensure they are >= 0.0+ let max_scale = scales.iter().fold(0.0, |max, &val| val.max(max));+ let max_min = mins.iter().fold(0.0, |max, &val| val.max(max));++ if max_scale > 0.0 {+ let iscale = Q4SCALE / max_scale;+ for (j, scale) in scales.iter().enumerate().take(QK_K / 16) {+ block.scales[j] = nearest_int(iscale * scale) as u8;+ }+ block.d = f16::from_f32(max_scale / Q4SCALE);+ } else {+ for j in 0..QK_K / 16 {+ block.scales[j] = 0;+ }+ block.d = f16::from_f32(0.0);+ }++ if max_min > 0.0 {+ let iscale = Q4SCALE / max_min;+ for (j, scale) in block.scales.iter_mut().enumerate() {+ let l = nearest_int(iscale * mins[j]) as u8;+ *scale |= l << 4;+ }+ block.dmin = f16::from_f32(max_min / Q4SCALE);+ } else {+ block.dmin = f16::from_f32(0.0);+ }++ let mut big_l: [u8; QK_K] = [0; QK_K];++ for j in 0..QK_K / 16 {+ let d = block.d.to_f32() * (block.scales[j] & 0xF) as f32;+ if d == 0.0 {+ continue;+ }+ let dm = block.dmin.to_f32() * (block.scales[j] >> 4) as f32;+ for ii in 0..16 {+ let ll = nearest_int((x[16 * j + ii] + dm) / d).clamp(0, 3);+ big_l[16 * j + ii] = ll as u8;+ }+ }++ for j in (0..QK_K).step_by(128) {+ for ll in 0..32 {+ block.qs[j / 4 + ll] = big_l[j + ll]+ | (big_l[j + ll + 32] << 2)+ | (big_l[j + ll + 64] << 4)+ | (big_l[j + ll + 96] << 6);+ }+ }+ }+ }++ fn from_float_imatrix(xs: &[f32], ys: &mut [Self], imatrix_weights: &[f32], n_per_row: usize) {+ for (sblk_idx, (block, x)) in group_for_quantization(xs, ys).into_iter().enumerate() {+ let mut mins: [f32; QK_K / 16] = [0.0; QK_K / 16];+ let mut scales: [f32; QK_K / 16] = [0.0; QK_K / 16];+ let mut weights: [f32; 16] = [0.0; 16];+ let mut sw: [f32; QK_K / 16] = [0.0; QK_K / 16];+ let mut ls: [u8; QK_K / 16] = [0; QK_K / 16];+ let mut lm: [u8; QK_K / 16] = [0; QK_K / 16];++ let sum_x2 = x.iter().map(|x| x * x).sum::<f32>();+ let sigma2 = sum_x2 / QK_K as f32;+ let (x_chunks, _) = x.as_chunks::<16>();+ for (j, x_scale_slice) in x_chunks.iter().enumerate() {+ for (l, (w_elem, x_elem)) in weights.iter_mut().zip(x_scale_slice).enumerate() {+ let imatrix_row = sblk_idx % (n_per_row / QK_K);+ let imatrix_w = imatrix_weights[imatrix_row * QK_K + 16 * j + l];+ *w_elem = imatrix_w * (sigma2 + x_elem * x_elem).sqrt();+ }+ let sumw = weights.iter().sum::<f32>();+ sw[j] = sumw;+ (scales[j], mins[j]) =+ make_qkx3_quants(3, x_scale_slice, Some(&weights), -0.9, 0.05, 36, false);+ }++ let d_block = make_qp_quants(QK_K / 16, 15, &scales, &mut ls, &sw);+ let m_block = make_qp_quants(QK_K / 16, 15, &mins, &mut lm, &sw);++ block.d = f16::from_f32(d_block);+ block.dmin = f16::from_f32(m_block);++ for j in 0..QK_K / 16 {+ block.scales[j] = ls[j] | (lm[j] << 4);+ }++ let mut big_l: [u8; QK_K] = [0; QK_K];++ for j in 0..QK_K / 16 {+ let d = block.d.to_f32() * (block.scales[j] & 0xF) as f32;+ if d == 0.0 {+ continue;+ }+ let dm = block.dmin.to_f32() * (block.scales[j] >> 4) as f32;+ for ii in 0..16 {+ let ll = nearest_int((x[16 * j + ii] + dm) / d).clamp(0, 3);+ big_l[16 * j + ii] = ll as u8;+ }+ }++ for j in (0..QK_K).step_by(128) {+ for ll in 0..32 {+ block.qs[j / 4 + ll] = big_l[j + ll]+ | (big_l[j + ll + 32] << 2)+ | (big_l[j + ll + 64] << 4)+ | (big_l[j + ll + 96] << 6);+ }+ }+ }+ }+ // https://github.com/ggerganov/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/k_quants.c#L354+ fn to_float(xs: &[Self], ys: &mut [f32]) {+ for (block, y) in group_for_dequantization(xs, ys) {+ let d = block.d.to_f32();+ let min = block.dmin.to_f32();++ let mut is = 0;++ let (y_chunks, _) = y.as_chunks_mut::<128>();+ let (qs_chunks, _) = block.qs.as_chunks::<32>();+ for (y_block, qs) in y_chunks.iter_mut().zip(qs_chunks) {+ // Step by 32 over q.+ let mut shift = 0;+ let mut y_block_index = 0;+ for _j in 0..4 {+ let sc = block.scales[is];+ is += 1;+ let dl = d * (sc & 0xF) as f32;+ let ml = min * (sc >> 4) as f32;+ for q in &qs[..16] {+ let y = dl * ((q >> shift) & 3) as f32 - ml;+ y_block[y_block_index] = y;+ y_block_index += 1;+ }++ let sc = block.scales[is];+ is += 1;+ let dl = d * (sc & 0xF) as f32;+ let ml = min * (sc >> 4) as f32;+ for q in &qs[16..] {+ let y = dl * ((q >> shift) & 3) as f32 - ml;+ y_block[y_block_index] = y;+ y_block_index += 1;+ }++ shift += 2;+ }+ }+ }+ }+}++impl GgmlType for BlockQ3K {+ const DTYPE: GgmlDType = GgmlDType::Q3K;+ const BLCK_SIZE: usize = QK_K;+ type VecDotType = BlockQ8K;++ #[allow(unreachable_code)]+ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if has_avx2_fma() {+ return unsafe { super::avx::vec_dot_q3k_q8k(n, xs, ys) };+ }++ #[cfg(target_feature = "neon")]+ return super::neon::vec_dot_q3k_q8k(n, xs, ys);++ Self::vec_dot_unopt(n, xs, ys)+ }++ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q3k_q8k: {n} is not divisible by {QK_K}"+ );++ const KMASK1: u32 = 0x03030303;+ const KMASK2: u32 = 0x0f0f0f0f;++ let mut aux8: [i8; QK_K] = [0; QK_K];+ let mut aux16: [i16; 8] = [0; 8];+ let mut sums: [f32; 8] = [0.0; 8];+ let mut aux32: [i32; 8] = [0; 8];++ let mut auxs: [u32; 4] = [0; 4];++ for (x, y) in xs.iter().zip(ys.iter()) {+ let mut q3: &[u8] = &x.qs;+ let hmask: &[u8] = &x.hmask;+ let mut q8: &[i8] = &y.qs;++ aux32.fill(0);+ let mut a = &mut aux8[..];++ let mut m = 1;+ //Like the GGML original this is written this way to enable the compiler to vectorize it.+ for _ in 0..QK_K / 128 {+ a.iter_mut()+ .take(32)+ .zip(q3)+ .for_each(|(a_val, q3_val)| *a_val = (q3_val & 3) as i8);+ a.iter_mut()+ .take(32)+ .zip(hmask)+ .for_each(|(a_val, hmask_val)| {+ *a_val -= if hmask_val & m != 0 { 0 } else { 4 }+ });+ a = &mut a[32..];+ m <<= 1;++ a.iter_mut()+ .take(32)+ .zip(q3)+ .for_each(|(a_val, q3_val)| *a_val = ((q3_val >> 2) & 3) as i8);+ a.iter_mut()+ .take(32)+ .zip(hmask)+ .for_each(|(a_val, hmask_val)| {+ *a_val -= if hmask_val & m != 0 { 0 } else { 4 }+ });+ a = &mut a[32..];+ m <<= 1;++ a.iter_mut()+ .take(32)+ .zip(q3)+ .for_each(|(a_val, q3_val)| *a_val = ((q3_val >> 4) & 3) as i8);+ a.iter_mut()+ .take(32)+ .zip(hmask)+ .for_each(|(a_val, hmask_val)| {+ *a_val -= if hmask_val & m != 0 { 0 } else { 4 }+ });+ a = &mut a[32..];+ m <<= 1;++ a.iter_mut()+ .take(32)+ .zip(q3)+ .for_each(|(a_val, q3_val)| *a_val = ((q3_val >> 6) & 3) as i8);+ a.iter_mut()+ .take(32)+ .zip(hmask)+ .for_each(|(a_val, hmask_val)| {+ *a_val -= if hmask_val & m != 0 { 0 } else { 4 }+ });+ a = &mut a[32..];+ m <<= 1;+ q3 = &q3[32..];+ }++ a = &mut aux8[..];++ LittleEndian::read_u32_into(&x.scales, &mut auxs[0..3]);++ let tmp = auxs[2];+ auxs[2] = ((auxs[0] >> 4) & KMASK2) | (((tmp >> 4) & KMASK1) << 4);+ auxs[3] = ((auxs[1] >> 4) & KMASK2) | (((tmp >> 6) & KMASK1) << 4);+ auxs[0] = (auxs[0] & KMASK2) | (((tmp) & KMASK1) << 4);+ auxs[1] = (auxs[1] & KMASK2) | (((tmp >> 2) & KMASK1) << 4);++ for aux in auxs {+ for scale in aux.to_le_bytes() {+ let scale = i8::from_be_bytes([scale]);+ for l in 0..8 {+ aux16[l] = q8[l] as i16 * a[l] as i16;+ }+ for l in 0..8 {+ aux32[l] += (scale as i32 - 32) * aux16[l] as i32;+ }+ q8 = &q8[8..];+ a = &mut a[8..];++ for l in 0..8 {+ aux16[l] = q8[l] as i16 * a[l] as i16;+ }+ for l in 0..8 {+ aux32[l] += (scale as i32 - 32) * aux16[l] as i32;+ }+ q8 = &q8[8..];+ a = &mut a[8..];+ }+ }+ let d = x.d.to_f32() * y.d;+ for l in 0..8 {+ sums[l] += d * aux32[l] as f32;+ }+ }++ sums.iter().sum()+ }++ fn from_float(xs: &[f32], ys: &mut [Self]) {+ for (block, x) in group_for_quantization(xs, ys) {+ let mut scales: [f32; QK_K / 16] = [0.0; QK_K / 16];+ let (x_chunks, _) = x.as_chunks::<16>();+ for (j, x_scale_slice) in x_chunks.iter().enumerate() {+ scales[j] = make_q3_quants(x_scale_slice, 4, true);+ }++ // Get max scale by absolute value.+ let mut max_scale: f32 = 0.0;+ for &scale in scales.iter() {+ if scale.abs() > max_scale.abs() {+ max_scale = scale;+ }+ }++ block.scales.fill(0);++ if max_scale != 0.0 {+ let iscale = -32.0 / max_scale;+ for (j, scale) in scales.iter().enumerate() {+ let l_val = nearest_int(iscale * scale);+ let l_val = l_val.clamp(-32, 31) + 32;+ if j < 8 {+ block.scales[j] = (l_val & 0xF) as u8;+ } else {+ block.scales[j - 8] |= ((l_val & 0xF) << 4) as u8;+ }+ let l_val = l_val >> 4;+ block.scales[j % 4 + 8] |= (l_val << (2 * (j / 4))) as u8;+ }+ block.d = f16::from_f32(1.0 / iscale);+ } else {+ block.d = f16::from_f32(0.0);+ }++ let mut l: [i8; QK_K] = [0; QK_K];++ for j in 0..QK_K / 16 {+ let sc = if j < 8 {+ block.scales[j] & 0xF+ } else {+ block.scales[j - 8] >> 4+ };+ let sc = (sc | (((block.scales[8 + j % 4] >> (2 * (j / 4))) & 3) << 4)) as i8 - 32;+ let d = block.d.to_f32() * sc as f32;+ if d != 0.0 {+ for ii in 0..16 {+ let l_val = nearest_int(x[16 * j + ii] / d);+ l[16 * j + ii] = (l_val.clamp(-4, 3) + 4) as i8;+ }+ }+ }++ block.hmask.fill(0);+ let mut m = 0;+ let mut hm = 1;++ for ll in l.iter_mut() {+ if *ll > 3 {+ block.hmask[m] |= hm;+ *ll -= 4;+ }+ m += 1;+ if m == QK_K / 8 {+ m = 0;+ hm <<= 1;+ }+ }++ for j in (0..QK_K).step_by(128) {+ for l_val in 0..32 {+ block.qs[j / 4 + l_val] = (l[j + l_val]+ | (l[j + l_val + 32] << 2)+ | (l[j + l_val + 64] << 4)+ | (l[j + l_val + 96] << 6))+ as u8;+ }+ }+ }+ }++ fn from_float_imatrix(xs: &[f32], ys: &mut [Self], imatrix_weights: &[f32], n_per_row: usize) {+ for (sblk_idx, (block, x)) in group_for_quantization(xs, ys).into_iter().enumerate() {+ let mut scales: [f32; QK_K / 16] = [0.0; QK_K / 16];+ let mut weights: [f32; 16] = [0.0; 16];+ let mut sw: [f32; QK_K / 16] = [0.0; QK_K / 16];+ let mut ls: [i8; QK_K / 16] = [0; QK_K / 16];+ let mut l: [i8; QK_K] = [0; QK_K];++ let sum_x2 = x.iter().map(|x| x * x).sum::<f32>();+ let sigma2 = 2. * sum_x2 / QK_K as f32;++ let (x_chunks, _) = x.as_chunks::<16>();+ for (j, x_scale_slice) in x_chunks.iter().enumerate() {+ for (l_idx, (w_elem, x_elem)) in weights.iter_mut().zip(x_scale_slice).enumerate() {+ let imatrix_row = sblk_idx % (n_per_row / QK_K);+ let imatrix_w = imatrix_weights[imatrix_row * QK_K + 16 * j + l_idx];+ *w_elem = imatrix_w * (sigma2 + x_elem * x_elem).sqrt();+ }+ let sumw = weights.iter().sum::<f32>();+ sw[j] = sumw;+ scales[j] = unsafe {+ make_qx_quants(+ 16,+ 4,+ x_scale_slice.as_ptr(),+ l.as_mut_ptr().add(16 * j),+ 1,+ weights.as_ptr(),+ )+ };+ }++ block.scales.fill(0);+ let d_block = unsafe {+ make_qx_quants(+ QK_K / 16,+ 32,+ scales.as_ptr(),+ ls.as_mut_ptr(),+ 1,+ sw.as_ptr(),+ )+ };+ block.d = f16::from_f32(d_block);+ for (j, l_val) in ls.iter().enumerate().take(QK_K / 16) {+ if j < 8 {+ block.scales[j] = (l_val & 0xF) as u8;+ } else {+ block.scales[j - 8] |= ((l_val & 0xF) << 4) as u8;+ }+ let l_val = l_val >> 4;+ block.scales[j % 4 + 8] |= (l_val << (2 * (j / 4))) as u8;+ }++ for j in 0..QK_K / 16 {+ let sc = if j < 8 {+ block.scales[j] & 0xF+ } else {+ block.scales[j - 8] >> 4+ };+ let sc = (sc | (((block.scales[8 + j % 4] >> (2 * (j / 4))) & 3) << 4)) as i8 - 32;+ let d = block.d.to_f32() * sc as f32;+ if d != 0.0 {+ for ii in 0..16 {+ let l_val = nearest_int(x[16 * j + ii] / d);+ l[16 * j + ii] = (l_val.clamp(-4, 3) + 4) as i8;+ }+ }+ }++ block.hmask.fill(0);+ let mut m = 0;+ let mut hm = 1;++ for ll in l.iter_mut() {+ if *ll > 3 {+ block.hmask[m] |= hm;+ *ll -= 4;+ }+ m += 1;+ if m == QK_K / 8 {+ m = 0;+ hm <<= 1;+ }+ }++ for j in (0..QK_K).step_by(128) {+ for l_val in 0..32 {+ block.qs[j / 4 + l_val] = (l[j + l_val]+ | (l[j + l_val + 32] << 2)+ | (l[j + l_val + 64] << 4)+ | (l[j + l_val + 96] << 6))+ as u8;+ }+ }+ }+ }++ // https://github.com/ggerganov/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/k_quants.c#L533+ fn to_float(xs: &[Self], ys: &mut [f32]) {+ const KMASK1: u32 = 0x03030303;+ const KMASK2: u32 = 0x0f0f0f0f;++ for (block, y) in group_for_dequantization(xs, ys) {+ //Reconstruct the scales+ let mut aux = [0; 4];+ LittleEndian::read_u32_into(&block.scales, &mut aux[0..3]);++ let tmp = aux[2];+ aux[2] = ((aux[0] >> 4) & KMASK2) | (((tmp >> 4) & KMASK1) << 4);+ aux[3] = ((aux[1] >> 4) & KMASK2) | (((tmp >> 6) & KMASK1) << 4);+ aux[0] = (aux[0] & KMASK2) | (((tmp) & KMASK1) << 4);+ aux[1] = (aux[1] & KMASK2) | (((tmp >> 2) & KMASK1) << 4);++ //Transfer the scales into an i8 array+ let scales: &mut [i8] =+ unsafe { std::slice::from_raw_parts_mut(aux.as_mut_ptr() as *mut i8, 16) };++ let d_all = block.d.to_f32();+ let mut m = 1;+ let mut is = 0;++ // Dequantize both 128 long blocks+ // 32 qs values per 128 long block+ // Each 16 elements get a scale+ let (y_chunks, _) = y.as_chunks_mut::<128>();+ let (qs_chunks, _) = block.qs.as_chunks::<32>();+ for (y, qs) in y_chunks.iter_mut().zip(qs_chunks) {+ let mut shift = 0;++ let (scoped_y_chunks, _) = y.as_chunks_mut::<32>();+ for shift_scoped_y in scoped_y_chunks {+ let (shift_scoped_chunks, _) = shift_scoped_y.as_chunks_mut::<16>();++ for (scale_index, scale_scoped_y) in shift_scoped_chunks.iter_mut().enumerate()+ {+ let dl = d_all * (scales[is] as f32 - 32.0);+ for (i, inner_y) in scale_scoped_y.iter_mut().enumerate() {+ let new_y = dl+ * (((qs[i + 16 * scale_index] >> shift) & 3) as i8+ - if (block.hmask[i + 16 * scale_index] & m) == 0 {+ 4+ } else {+ 0+ }) as f32;+ *inner_y = new_y;+ }+ // 16 block finished => advance scale index+ is += 1;+ }+ // 32 block finished => increase shift and m+ shift += 2;+ m <<= 1;+ }+ }+ }+ }+}++impl GgmlType for BlockQ4K {+ const DTYPE: GgmlDType = GgmlDType::Q4K;+ const BLCK_SIZE: usize = QK_K;+ type VecDotType = BlockQ8K;++ #[allow(unreachable_code)]+ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if has_avx2_fma() {+ return unsafe { super::avx::vec_dot_q4k_q8k(n, xs, ys) };+ }++ #[cfg(target_feature = "neon")]+ return super::neon::vec_dot_q4k_q8k(n, xs, ys);++ #[cfg(target_feature = "simd128")]+ return super::simd128::vec_dot_q4k_q8k(n, xs, ys);++ Self::vec_dot_unopt(n, xs, ys)+ }++ #[allow(unreachable_code)]+ fn vec_dot_4(+ n: usize,+ xs0: &[Self],+ xs1: &[Self],+ xs2: &[Self],+ xs3: &[Self],+ ys: &[Self::VecDotType],+ ) -> (f32, f32, f32, f32) {+ #[cfg(target_feature = "neon")]+ return super::neon::vec_dot_4_q4k_q8k(n, xs0, xs1, xs2, xs3, ys);++ let (d0, d1) = Self::vec_dot_2(n, xs0, xs1, ys);+ let (d2, d3) = Self::vec_dot_2(n, xs2, xs3, ys);+ (d0, d1, d2, d3)+ }++ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q4k_q8k: {n} is not divisible by {QK_K}"+ );++ const KMASK1: u32 = 0x3f3f3f3f;+ const KMASK2: u32 = 0x0f0f0f0f;+ const KMASK3: u32 = 0x03030303;++ let mut utmp: [u32; 4] = [0; 4];+ let mut scales: [u8; 8] = [0; 8];+ let mut mins: [u8; 8] = [0; 8];++ let mut aux8: [i8; QK_K] = [0; QK_K];+ let mut aux16: [i16; 8] = [0; 8];+ let mut sums: [f32; 8] = [0.0; 8];+ let mut aux32: [i32; 8] = [0; 8];++ let mut sumf = 0.0;+ for (y, x) in ys.iter().zip(xs.iter()) {+ let q4 = &x.qs;+ let q8 = &y.qs;+ aux32.fill(0);++ let mut a = &mut aux8[..];+ let mut q4 = &q4[..];+ for _ in 0..QK_K / 64 {+ for l in 0..32 {+ a[l] = (q4[l] & 0xF) as i8;+ }+ a = &mut a[32..];+ for l in 0..32 {+ a[l] = (q4[l] >> 4) as i8;+ }+ a = &mut a[32..];+ q4 = &q4[32..];+ }++ LittleEndian::read_u32_into(&x.scales, &mut utmp[0..3]);++ utmp[3] = ((utmp[2] >> 4) & KMASK2) | (((utmp[1] >> 6) & KMASK3) << 4);+ let uaux = utmp[1] & KMASK1;+ utmp[1] = (utmp[2] & KMASK2) | (((utmp[0] >> 6) & KMASK3) << 4);+ utmp[2] = uaux;+ utmp[0] &= KMASK1;++ //extract scales and mins+ LittleEndian::write_u32_into(&utmp[0..2], &mut scales);+ LittleEndian::write_u32_into(&utmp[2..4], &mut mins);++ let mut sumi = 0;+ for j in 0..QK_K / 16 {+ sumi += y.bsums[j] as i32 * mins[j / 2] as i32;+ }++ let mut a = &mut aux8[..];+ let mut q8 = &q8[..];++ for scale in scales {+ let scale = scale as i32;+ for _ in 0..4 {+ for l in 0..8 {+ aux16[l] = q8[l] as i16 * a[l] as i16;+ }+ for l in 0..8 {+ aux32[l] += scale * aux16[l] as i32;+ }+ q8 = &q8[8..];+ a = &mut a[8..];+ }+ }+ let d = x.d.to_f32() * y.d;+ for l in 0..8 {+ sums[l] += d * aux32[l] as f32;+ }+ let dmin = x.dmin.to_f32() * y.d;+ sumf -= dmin * sumi as f32;+ }+ sumf + sums.iter().sum::<f32>()+ }++ fn from_float(xs: &[f32], ys: &mut [Self]) {+ for (block, x) in group_for_quantization(xs, ys) {+ let mut mins: [f32; QK_K / 32] = [0.0; QK_K / 32];+ let mut scales: [f32; QK_K / 32] = [0.0; QK_K / 32];++ let (x_chunks, _) = x.as_chunks::<32>();+ for (j, x_scale_slice) in x_chunks.iter().enumerate() {+ (scales[j], mins[j]) = make_qkx1_quants(15, 5, x_scale_slice);+ }++ // get max scale and max min and ensure they are >= 0.0+ let max_scale = scales.iter().fold(0.0, |max, &val| val.max(max));+ let max_min = mins.iter().fold(0.0, |max, &val| val.max(max));++ let inv_scale = if max_scale > 0.0 {+ 63.0 / max_scale+ } else {+ 0.0+ };+ let inv_min = if max_min > 0.0 { 63.0 / max_min } else { 0.0 };++ for j in 0..QK_K / 32 {+ let ls = nearest_int(inv_scale * scales[j]).min(63) as u8;+ let lm = nearest_int(inv_min * mins[j]).min(63) as u8;+ if j < 4 {+ block.scales[j] = ls;+ block.scales[j + 4] = lm;+ } else {+ block.scales[j + 4] = (ls & 0xF) | ((lm & 0xF) << 4);+ block.scales[j - 4] |= (ls >> 4) << 6;+ block.scales[j] |= (lm >> 4) << 6;+ }+ }++ block.d = f16::from_f32(max_scale / 63.0);+ block.dmin = f16::from_f32(max_min / 63.0);++ let mut l: [u8; QK_K] = [0; QK_K];++ for j in 0..QK_K / 32 {+ let (sc, m) = get_scale_min_k4(j, &block.scales);+ let d = block.d.to_f32() * sc as f32;+ if d != 0.0 {+ let dm = block.dmin.to_f32() * m as f32;+ for ii in 0..32 {+ let l_val = nearest_int((x[32 * j + ii] + dm) / d);+ l[32 * j + ii] = l_val.clamp(0, 15) as u8;+ }+ }+ }++ let q = &mut block.qs;+ for j in (0..QK_K).step_by(64) {+ for l_val in 0..32 {+ let offset_index = (j / 64) * 32 + l_val;+ q[offset_index] = l[j + l_val] | (l[j + l_val + 32] << 4);+ }+ }+ }+ }++ fn from_float_imatrix(xs: &[f32], ys: &mut [Self], imatrix_weights: &[f32], n_per_row: usize) {+ for (sblk_idx, (block, x)) in group_for_quantization(xs, ys).into_iter().enumerate() {+ let mut mins: [f32; QK_K / 32] = [0.0; QK_K / 32];+ let mut scales: [f32; QK_K / 32] = [0.0; QK_K / 32];+ let mut weights: [f32; 32] = [0.0; 32];+ let mut sw: [f32; QK_K / 32] = [0.0; QK_K / 32];+ let mut ls: [u8; QK_K / 32] = [0; QK_K / 32];+ let mut lm: [u8; QK_K / 32] = [0; QK_K / 32];++ let sum_x2 = x.iter().map(|x| x * x).sum::<f32>();+ let sigma2 = 2. * sum_x2 / QK_K as f32;++ let (x_chunks, _) = x.as_chunks::<32>();+ for (j, x_scale_slice) in x_chunks.iter().enumerate() {+ for (l, (w_elem, x_elem)) in weights.iter_mut().zip(x_scale_slice).enumerate() {+ let imatrix_row = sblk_idx % (n_per_row / QK_K);+ let imatrix_w = imatrix_weights[imatrix_row * QK_K + 32 * j + l];+ *w_elem = imatrix_w * (sigma2 + x_elem * x_elem).sqrt();+ }+ let sumw = weights.iter().sum::<f32>();+ sw[j] = sumw;+ (scales[j], mins[j]) =+ make_qkx3_quants(15, x_scale_slice, Some(&weights), -0.9, 0.05, 36, false);+ }++ let d_block = make_qp_quants(QK_K / 32, 63, &scales, &mut ls, &sw);+ let m_block = make_qp_quants(QK_K / 32, 63, &mins, &mut lm, &sw);+ for j in 0..QK_K / 32 {+ let ls_val = ls[j];+ let lm_val = lm[j];+ if j < 4 {+ block.scales[j] = ls_val;+ block.scales[j + 4] = lm_val;+ } else {+ block.scales[j + 4] = (ls_val & 0xF) | ((lm_val & 0xF) << 4);+ block.scales[j - 4] |= (ls_val >> 4) << 6;+ block.scales[j] |= (lm_val >> 4) << 6;+ }+ }++ block.d = f16::from_f32(d_block);+ block.dmin = f16::from_f32(m_block);++ let mut l: [u8; QK_K] = [0; QK_K];+ for j in 0..QK_K / 32 {+ let (sc, m) = get_scale_min_k4(j, &block.scales);+ let d = block.d.to_f32() * sc as f32;+ if d != 0.0 {+ let dm = block.dmin.to_f32() * m as f32;+ for ii in 0..32 {+ let l_val = nearest_int((x[32 * j + ii] + dm) / d);+ l[32 * j + ii] = l_val.clamp(0, 15) as u8;+ }+ }+ }++ let q = &mut block.qs;+ for j in (0..QK_K).step_by(64) {+ for l_val in 0..32 {+ let offset_index = (j / 64) * 32 + l_val;+ q[offset_index] = l[j + l_val] | (l[j + l_val + 32] << 4);+ }+ }+ }+ }+ // https://github.com/ggerganov/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/k_quants.c#L735+ fn to_float(xs: &[Self], ys: &mut [f32]) {+ for (block, y) in group_for_dequantization(xs, ys) {+ let d = block.d.to_f32();+ let min = block.dmin.to_f32();+ let q = &block.qs;+ let mut is = 0;+ let mut ys_index = 0;++ for j in (0..QK_K).step_by(64) {+ let q = &q[j / 2..j / 2 + 32];+ let (sc, m) = get_scale_min_k4(is, &block.scales);+ let d1 = d * sc as f32;+ let m1 = min * m as f32;+ let (sc, m) = get_scale_min_k4(is + 1, &block.scales);+ let d2 = d * sc as f32;+ let m2 = min * m as f32;+ for q in q {+ y[ys_index] = d1 * (q & 0xF) as f32 - m1;+ ys_index += 1;+ }+ for q in q {+ y[ys_index] = d2 * (q >> 4) as f32 - m2;+ ys_index += 1;+ }+ is += 2;+ }+ }+ }+}++// https://github.com/ggerganov/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/k_quants.c#L928+impl GgmlType for BlockQ5K {+ const DTYPE: GgmlDType = GgmlDType::Q5K;+ const BLCK_SIZE: usize = QK_K;+ type VecDotType = BlockQ8K;++ #[allow(unreachable_code)]+ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if has_avx2_fma() {+ return unsafe { super::avx::vec_dot_q5k_q8k(n, xs, ys) };+ }++ #[cfg(target_feature = "neon")]+ return super::neon::vec_dot_q5k_q8k(n, xs, ys);++ Self::vec_dot_unopt(n, xs, ys)+ }++ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q5k_q8k: {n} is not divisible by {QK_K}"+ );++ const KMASK1: u32 = 0x3f3f3f3f;+ const KMASK2: u32 = 0x0f0f0f0f;+ const KMASK3: u32 = 0x03030303;++ let mut utmp: [u32; 4] = [0; 4];+ let mut scales: [u8; 8] = [0; 8];+ let mut mins: [u8; 8] = [0; 8];++ let mut aux8: [i8; QK_K] = [0; QK_K];+ let mut aux16: [i16; 8] = [0; 8];+ let mut sums: [f32; 8] = [0.0; 8];+ let mut aux32: [i32; 8] = [0; 8];++ let mut sumf = 0.0;+ for (y, x) in ys.iter().zip(xs.iter()) {+ let q5 = &x.qs;+ let hm = &x.qh;+ let q8 = &y.qs;+ aux32.fill(0);++ let mut a = &mut aux8[..];+ let mut q5 = &q5[..];+ let mut m = 1u8;++ for _ in 0..QK_K / 64 {+ for l in 0..32 {+ a[l] = (q5[l] & 0xF) as i8;+ a[l] += if hm[l] & m != 0 { 16 } else { 0 };+ }+ a = &mut a[32..];+ m <<= 1;+ for l in 0..32 {+ a[l] = (q5[l] >> 4) as i8;+ a[l] += if hm[l] & m != 0 { 16 } else { 0 };+ }+ a = &mut a[32..];+ m <<= 1;+ q5 = &q5[32..];+ }++ LittleEndian::read_u32_into(&x.scales, &mut utmp[0..3]);++ utmp[3] = ((utmp[2] >> 4) & KMASK2) | (((utmp[1] >> 6) & KMASK3) << 4);+ let uaux = utmp[1] & KMASK1;+ utmp[1] = (utmp[2] & KMASK2) | (((utmp[0] >> 6) & KMASK3) << 4);+ utmp[2] = uaux;+ utmp[0] &= KMASK1;++ //extract scales and mins+ LittleEndian::write_u32_into(&utmp[0..2], &mut scales);+ LittleEndian::write_u32_into(&utmp[2..4], &mut mins);++ let mut sumi = 0;+ for j in 0..QK_K / 16 {+ sumi += y.bsums[j] as i32 * mins[j / 2] as i32;+ }++ let mut a = &mut aux8[..];+ let mut q8 = &q8[..];++ for scale in scales {+ let scale = scale as i32;+ for _ in 0..4 {+ for l in 0..8 {+ aux16[l] = q8[l] as i16 * a[l] as i16;+ }+ for l in 0..8 {+ aux32[l] += scale * aux16[l] as i32;+ }+ q8 = &q8[8..];+ a = &mut a[8..];+ }+ }+ let d = x.d.to_f32() * y.d;+ for l in 0..8 {+ sums[l] += d * aux32[l] as f32;+ }+ let dmin = x.dmin.to_f32() * y.d;+ sumf -= dmin * sumi as f32;+ }+ sumf + sums.iter().sum::<f32>()+ }++ // https://github.com/ggerganov/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/k_quants.c#L793+ fn from_float(xs: &[f32], ys: &mut [Self]) {+ for (block, x) in group_for_quantization(xs, ys) {+ let mut mins: [f32; QK_K / 32] = [0.0; QK_K / 32];+ let mut scales: [f32; QK_K / 32] = [0.0; QK_K / 32];++ let (x_chunks, _) = x.as_chunks::<32>();+ for (j, x_scale_slice) in x_chunks.iter().enumerate() {+ (scales[j], mins[j]) = make_qkx1_quants(31, 5, x_scale_slice);+ }++ // get max scale and max min and ensure they are >= 0.0+ let max_scale = scales.iter().fold(0.0, |max, &val| val.max(max));+ let max_min = mins.iter().fold(0.0, |max, &val| val.max(max));++ let inv_scale = if max_scale > 0.0 {+ 63.0 / max_scale+ } else {+ 0.0+ };+ let inv_min = if max_min > 0.0 { 63.0 / max_min } else { 0.0 };+ for j in 0..QK_K / 32 {+ let ls = nearest_int(inv_scale * scales[j]).min(63) as u8;+ let lm = nearest_int(inv_min * mins[j]).min(63) as u8;+ if j < 4 {+ block.scales[j] = ls;+ block.scales[j + 4] = lm;+ } else {+ block.scales[j + 4] = (ls & 0xF) | ((lm & 0xF) << 4);+ block.scales[j - 4] |= (ls >> 4) << 6;+ block.scales[j] |= (lm >> 4) << 6;+ }+ }+ block.d = f16::from_f32(max_scale / 63.0);+ block.dmin = f16::from_f32(max_min / 63.0);++ let mut l: [u8; QK_K] = [0; QK_K];+ for j in 0..QK_K / 32 {+ let (sc, m) = get_scale_min_k4(j, &block.scales);+ let d = block.d.to_f32() * sc as f32;+ if d == 0.0 {+ continue;+ }+ let dm = block.dmin.to_f32() * m as f32;+ for ii in 0..32 {+ let ll = nearest_int((x[32 * j + ii] + dm) / d);+ l[32 * j + ii] = ll.clamp(0, 31) as u8;+ }+ }++ let qh = &mut block.qh;+ let ql = &mut block.qs;+ qh.fill(0);++ let mut m1 = 1;+ let mut m2 = 2;+ for n in (0..QK_K).step_by(64) {+ let offset = (n / 64) * 32;+ for j in 0..32 {+ let mut l1 = l[n + j];+ if l1 > 15 {+ l1 -= 16;+ qh[j] |= m1;+ }+ let mut l2 = l[n + j + 32];+ if l2 > 15 {+ l2 -= 16;+ qh[j] |= m2;+ }+ ql[offset + j] = l1 | (l2 << 4);+ }+ m1 <<= 2;+ m2 <<= 2;+ }+ }+ }++ fn from_float_imatrix(xs: &[f32], ys: &mut [Self], imatrix_weights: &[f32], n_per_row: usize) {+ for (sblk_idx, (block, x)) in group_for_quantization(xs, ys).into_iter().enumerate() {+ let mut mins: [f32; QK_K / 32] = [0.0; QK_K / 32];+ let mut scales: [f32; QK_K / 32] = [0.0; QK_K / 32];+ let mut weights: [f32; 32] = [0.0; 32];+ let mut sw: [f32; QK_K / 32] = [0.0; QK_K / 32];+ let mut ls: [u8; QK_K / 32] = [0; QK_K / 32];+ let mut lm: [u8; QK_K / 32] = [0; QK_K / 32];++ let sum_x2 = x.iter().map(|x| x * x).sum::<f32>();+ let sigma2 = 2. * sum_x2 / QK_K as f32;++ let (x_chunks, _) = x.as_chunks::<32>();+ for (j, x_scale_slice) in x_chunks.iter().enumerate() {+ for (l, (w_elem, x_elem)) in weights.iter_mut().zip(x_scale_slice).enumerate() {+ let imatrix_row = sblk_idx % (n_per_row / QK_K);+ let imatrix_w = imatrix_weights[imatrix_row * QK_K + 32 * j + l];+ *w_elem = imatrix_w * (sigma2 + x_elem * x_elem).sqrt();+ }+ let sumw = weights.iter().sum::<f32>();+ sw[j] = sumw;+ (scales[j], mins[j]) =+ make_qkx3_quants(31, x_scale_slice, Some(&weights), -0.9, 0.05, 36, false);+ }++ let d_block = make_qp_quants(QK_K / 32, 63, &scales, &mut ls, &sw);+ let m_block = make_qp_quants(QK_K / 32, 63, &mins, &mut lm, &sw);+ for j in 0..QK_K / 32 {+ let ls_val = ls[j].min(63);+ let lm_val = lm[j].min(63);+ if j < 4 {+ block.scales[j] = ls_val;+ block.scales[j + 4] = lm_val;+ } else {+ block.scales[j + 4] = (ls_val & 0xF) | ((lm_val & 0xF) << 4);+ block.scales[j - 4] |= (ls_val >> 4) << 6;+ block.scales[j] |= (lm_val >> 4) << 6;+ }+ }++ block.d = f16::from_f32(d_block);+ block.dmin = f16::from_f32(m_block);++ let mut l: [u8; QK_K] = [0; QK_K];+ for j in 0..QK_K / 32 {+ let (sc, m) = get_scale_min_k4(j, &block.scales);+ let d = block.d.to_f32() * sc as f32;+ if d != 0.0 {+ let dm = block.dmin.to_f32() * m as f32;+ for ii in 0..32 {+ let l_val = nearest_int((x[32 * j + ii] + dm) / d);+ l[32 * j + ii] = l_val.clamp(0, 31) as u8;+ }+ }+ }++ let qh = &mut block.qh;+ let ql = &mut block.qs;+ qh.fill(0);++ let mut m1 = 1;+ let mut m2 = 2;+ for n in (0..QK_K).step_by(64) {+ let offset = (n / 64) * 32;+ for j in 0..32 {+ let mut l1 = l[n + j];+ if l1 > 15 {+ l1 -= 16;+ qh[j] |= m1;+ }+ let mut l2 = l[n + j + 32];+ if l2 > 15 {+ l2 -= 16;+ qh[j] |= m2;+ }+ ql[offset + j] = l1 | (l2 << 4);+ }+ m1 <<= 2;+ m2 <<= 2;+ }+ }+ }++ // https://github.com/ggerganov/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/k_quants.c#L928+ fn to_float(xs: &[Self], ys: &mut [f32]) {+ for (block, y) in group_for_dequantization(xs, ys) {+ let d = block.d.to_f32();+ let min = block.dmin.to_f32();+ let ql = &block.qs;+ let qh = &block.qh;+ let mut is = 0;+ let mut u1 = 1;+ let mut u2 = 2;+ let mut ys_index = 0;++ for j in (0..QK_K).step_by(64) {+ let ql = &ql[j / 2..j / 2 + 32];+ let (sc, m) = get_scale_min_k4(is, &block.scales);+ let d1 = d * sc as f32;+ let m1 = min * m as f32;+ let (sc, m) = get_scale_min_k4(is + 1, &block.scales);+ let d2 = d * sc as f32;+ let m2 = min * m as f32;+ for (ql, qh) in ql.iter().zip(qh) {+ let to_add = if qh & u1 != 0 { 16f32 } else { 0f32 };+ y[ys_index] = d1 * ((ql & 0xF) as f32 + to_add) - m1;+ ys_index += 1;+ }+ for (ql, qh) in ql.iter().zip(qh) {+ let to_add = if qh & u2 != 0 { 16f32 } else { 0f32 };+ y[ys_index] = d2 * ((ql >> 4) as f32 + to_add) - m2;+ ys_index += 1;+ }+ is += 2;+ u1 <<= 2;+ u2 <<= 2;+ }+ }+ }+}++impl GgmlType for BlockQ6K {+ const DTYPE: GgmlDType = GgmlDType::Q6K;+ const BLCK_SIZE: usize = QK_K;+ type VecDotType = BlockQ8K;++ #[allow(unreachable_code)]+ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if has_avx2_fma() {+ return unsafe { super::avx::vec_dot_q6k_q8k(n, xs, ys) };+ }++ #[cfg(target_feature = "neon")]+ return super::neon::vec_dot_q6k_q8k(n, xs, ys);++ #[cfg(target_feature = "simd128")]+ return super::simd128::vec_dot_q6k_q8k(n, xs, ys);++ Self::vec_dot_unopt(n, xs, ys)+ }++ #[allow(unreachable_code)]+ fn vec_dot_4(+ n: usize,+ xs0: &[Self],+ xs1: &[Self],+ xs2: &[Self],+ xs3: &[Self],+ ys: &[Self::VecDotType],+ ) -> (f32, f32, f32, f32) {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if has_avx2_fma() {+ // Check the runtime feature set once for the whole output quad.+ return unsafe {+ (+ super::avx::vec_dot_q6k_q8k(n, xs0, ys),+ super::avx::vec_dot_q6k_q8k(n, xs1, ys),+ super::avx::vec_dot_q6k_q8k(n, xs2, ys),+ super::avx::vec_dot_q6k_q8k(n, xs3, ys),+ )+ };+ }++ #[cfg(target_feature = "neon")]+ return super::neon::vec_dot_4_q6k_q8k(n, xs0, xs1, xs2, xs3, ys);++ let (d0, d1) = Self::vec_dot_2(n, xs0, xs1, ys);+ let (d2, d3) = Self::vec_dot_2(n, xs2, xs3, ys);+ (d0, d1, d2, d3)+ }++ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q6k_q8k: {n} is not divisible by {QK_K}"+ );++ let mut aux8 = [0i8; QK_K];+ let mut aux16 = [0i16; 8];+ let mut sums = [0f32; 8];+ let mut aux32 = [0f32; 8];++ for (x, y) in xs.iter().zip(ys.iter()) {+ let q4 = &x.ql;+ let qh = &x.qh;+ let q8 = &y.qs;+ aux32.fill(0f32);++ for j in (0..QK_K).step_by(128) {+ let aux8 = &mut aux8[j..];+ let q4 = &q4[j / 2..];+ let qh = &qh[j / 4..];+ for l in 0..32 {+ aux8[l] = (((q4[l] & 0xF) | ((qh[l] & 3) << 4)) as i32 - 32) as i8;+ aux8[l + 32] =+ (((q4[l + 32] & 0xF) | (((qh[l] >> 2) & 3) << 4)) as i32 - 32) as i8;+ aux8[l + 64] = (((q4[l] >> 4) | (((qh[l] >> 4) & 3) << 4)) as i32 - 32) as i8;+ aux8[l + 96] =+ (((q4[l + 32] >> 4) | (((qh[l] >> 6) & 3) << 4)) as i32 - 32) as i8;+ }+ }++ for (j, &scale) in x.scales.iter().enumerate() {+ let scale = scale as f32;+ let q8 = &q8[16 * j..];+ let aux8 = &aux8[16 * j..];+ for l in 0..8 {+ aux16[l] = q8[l] as i16 * aux8[l] as i16;+ }+ for l in 0..8 {+ aux32[l] += scale * aux16[l] as f32+ }+ let q8 = &q8[8..];+ let aux8 = &aux8[8..];+ for l in 0..8 {+ aux16[l] = q8[l] as i16 * aux8[l] as i16;+ }+ for l in 0..8 {+ aux32[l] += scale * aux16[l] as f32+ }+ }++ let d = x.d.to_f32() * y.d;+ for (sum, &a) in sums.iter_mut().zip(aux32.iter()) {+ *sum += a * d;+ }+ }+ sums.iter().sum()+ }++ fn from_float(xs: &[f32], ys: &mut [Self]) {+ debug_assert_eq!(+ xs.len(),+ ys.len() * Self::BLCK_SIZE,+ "quantize_row_q6k: size mismatch {} {} {}",+ xs.len(),+ ys.len(),+ Self::BLCK_SIZE+ );+ let mut l = [0i8; QK_K];+ let mut scales = [0f32; QK_K / 16];+ let mut x = xs.as_ptr();+ let l = l.as_mut_ptr();+ unsafe {+ for y in ys.iter_mut() {+ let mut max_scale = 0f32;+ let mut max_abs_scale = 0f32;+ for (ib, scale_) in scales.iter_mut().enumerate() {+ let scale =+ make_qx_quants(16, 32, x.add(16 * ib), l.add(16 * ib), 1, std::ptr::null());+ *scale_ = scale;+ let abs_scale = scale.abs();+ if abs_scale > max_abs_scale {+ max_abs_scale = abs_scale;+ max_scale = scale+ }+ }++ let iscale = -128f32 / max_scale;+ y.d = f16::from_f32(1.0 / iscale);++ for (y_scale, scale) in y.scales.iter_mut().zip(scales.iter()) {+ *y_scale = nearest_int(iscale * scale).min(127) as i8+ }++ for (j, &y_scale) in y.scales.iter().enumerate() {+ let d = y.d.to_f32() * y_scale as f32;+ if d == 0. {+ continue;+ }+ for ii in 0..16 {+ let ll = nearest_int(*x.add(16 * j + ii) / d).clamp(-32, 31);+ *l.add(16 * j + ii) = (ll + 32) as i8+ }+ }++ let mut ql = y.ql.as_mut_ptr();+ let mut qh = y.qh.as_mut_ptr();++ for j in (0..QK_K).step_by(128) {+ for l_idx in 0..32 {+ let q1 = *l.add(j + l_idx) & 0xF;+ let q2 = *l.add(j + l_idx + 32) & 0xF;+ let q3 = *l.add(j + l_idx + 64) & 0xF;+ let q4 = *l.add(j + l_idx + 96) & 0xF;+ *ql.add(l_idx) = (q1 | (q3 << 4)) as u8;+ *ql.add(l_idx + 32) = (q2 | (q4 << 4)) as u8;+ *qh.add(l_idx) = ((*l.add(j + l_idx) >> 4)+ | ((*l.add(j + l_idx + 32) >> 4) << 2)+ | ((*l.add(j + l_idx + 64) >> 4) << 4)+ | ((*l.add(j + l_idx + 96) >> 4) << 6))+ as u8;+ }+ ql = ql.add(64);+ qh = qh.add(32);+ }++ x = x.add(QK_K)+ }+ }+ }++ fn from_float_imatrix(xs: &[f32], ys: &mut [Self], imatrix_weights: &[f32], n_per_row: usize) {+ debug_assert_eq!(+ xs.len(),+ ys.len() * Self::BLCK_SIZE,+ "quantize_row_q6k imatrix: size mismatch {} {} {}",+ xs.len(),+ ys.len(),+ Self::BLCK_SIZE+ );+ let mut l = [0i8; QK_K];+ let mut scales = [0f32; QK_K / 16];+ let mut x = xs.as_ptr();+ let imatrix_weights = imatrix_weights.as_ptr();+ let l = l.as_mut_ptr();+ unsafe {+ for (sblk_idx, y) in ys.iter_mut().enumerate() {+ let mut max_scale = 0f32;+ let mut max_abs_scale = 0f32;+ for (ib, scale_) in scales.iter_mut().enumerate() {+ let imatrix_row = sblk_idx % (n_per_row / QK_K);+ let scale = make_qx_quants(+ 16,+ 32,+ x.add(16 * ib),+ l.add(16 * ib),+ 1,+ imatrix_weights.add(QK_K * imatrix_row + 16 * ib),+ );+ *scale_ = scale;+ let abs_scale = scale.abs();+ if abs_scale > max_abs_scale {+ max_abs_scale = abs_scale;+ max_scale = scale+ }+ }++ let iscale = -128f32 / max_scale;+ y.d = f16::from_f32(1.0 / iscale);++ for (y_scale, scale) in y.scales.iter_mut().zip(scales.iter()) {+ *y_scale = nearest_int(iscale * scale).min(127) as i8+ }++ for (j, &y_scale) in y.scales.iter().enumerate() {+ let d = y.d.to_f32() * y_scale as f32;+ if d == 0. {+ continue;+ }+ for ii in 0..16 {+ let ll = nearest_int(*x.add(16 * j + ii) / d).clamp(-32, 31);+ *l.add(16 * j + ii) = (ll + 32) as i8+ }+ }++ let mut ql = y.ql.as_mut_ptr();+ let mut qh = y.qh.as_mut_ptr();++ for j in (0..QK_K).step_by(128) {+ for l_idx in 0..32 {+ let q1 = *l.add(j + l_idx) & 0xF;+ let q2 = *l.add(j + l_idx + 32) & 0xF;+ let q3 = *l.add(j + l_idx + 64) & 0xF;+ let q4 = *l.add(j + l_idx + 96) & 0xF;+ *ql.add(l_idx) = (q1 | (q3 << 4)) as u8;+ *ql.add(l_idx + 32) = (q2 | (q4 << 4)) as u8;+ *qh.add(l_idx) = ((*l.add(j + l_idx) >> 4)+ | ((*l.add(j + l_idx + 32) >> 4) << 2)+ | ((*l.add(j + l_idx + 64) >> 4) << 4)+ | ((*l.add(j + l_idx + 96) >> 4) << 6))+ as u8;+ }+ ql = ql.add(64);+ qh = qh.add(32);+ }++ x = x.add(QK_K)+ }+ }+ }++ // https://github.com/ggerganov/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/k_quants.c#L1067+ fn to_float(xs: &[Self], ys: &mut [f32]) {+ let k = ys.len();+ debug_assert!(+ k.is_multiple_of(QK_K),+ "dequantize_row_q6k: {k} is not divisible by {QK_K}"+ );++ for (idx_x, x) in xs.iter().enumerate() {+ let d = x.d.to_f32();+ let ql = &x.ql;+ let qh = &x.qh;+ let sc = &x.scales;+ for n in (0..QK_K).step_by(128) {+ let idx = n / 128;+ let ys = &mut ys[idx_x * QK_K + n..];+ let sc = &sc[8 * idx..];+ let ql = &ql[64 * idx..];+ let qh = &qh[32 * idx..];+ for l in 0..32 {+ let is = l / 16;+ let q1 = ((ql[l] & 0xF) | ((qh[l] & 3) << 4)) as i8 - 32;+ let q2 = ((ql[l + 32] & 0xF) | (((qh[l] >> 2) & 3) << 4)) as i8 - 32;+ let q3 = ((ql[l] >> 4) | (((qh[l] >> 4) & 3) << 4)) as i8 - 32;+ let q4 = ((ql[l + 32] >> 4) | (((qh[l] >> 6) & 3) << 4)) as i8 - 32;+ ys[l] = d * sc[is] as f32 * q1 as f32;+ ys[l + 32] = d * sc[is + 2] as f32 * q2 as f32;+ ys[l + 64] = d * sc[is + 4] as f32 * q3 as f32;+ ys[l + 96] = d * sc[is + 6] as f32 * q4 as f32;+ }+ }+ }+ }+}++impl GgmlType for BlockQ8K {+ const DTYPE: GgmlDType = GgmlDType::Q8K;+ const BLCK_SIZE: usize = QK_K;+ type VecDotType = BlockQ8K;++ #[allow(unreachable_code)]+ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ #[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+ if has_avx2_fma() {+ return unsafe { super::avx::vec_dot_q8k_q8k(n, xs, ys) };+ }++ #[cfg(target_feature = "neon")]+ return super::neon::vec_dot_q8k_q8k(n, xs, ys);++ #[cfg(target_feature = "simd128")]+ return super::simd128::vec_dot_q8k_q8k(n, xs, ys);++ Self::vec_dot_unopt(n, xs, ys)+ }++ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q8k_q8k: {n} is not divisible by {QK_K}"+ );+ // Generic implementation.+ let mut sumf = 0f32;+ for (xs, ys) in xs.iter().zip(ys.iter()) {+ let sum_i = xs+ .qs+ .iter()+ .zip(ys.qs.iter())+ .map(|(&x, &y)| x as i32 * y as i32)+ .sum::<i32>();+ sumf += sum_i as f32 * xs.d * ys.d+ }+ sumf+ }++ #[allow(unreachable_code)]+ fn from_float(xs: &[f32], ys: &mut [Self]) {+ #[cfg(target_feature = "neon")]+ return super::neon::quantize_row_q8k(xs, ys);++ let k = xs.len();+ debug_assert!(+ k.is_multiple_of(QK_K),+ "quantize_row_q8k: {k} is not divisible by {QK_K}"+ );+ for (i, y) in ys.iter_mut().enumerate() {+ let mut max = 0f32;+ let mut amax = 0f32;+ let xs = &xs[i * QK_K..(i + 1) * QK_K];+ for &x in xs.iter() {+ if amax < x.abs() {+ amax = x.abs();+ max = x;+ }+ }+ if amax == 0f32 {+ y.d = 0f32;+ y.qs.fill(0);+ y.bsums.fill(0)+ } else {+ let iscale = -127f32 / max;+ for (j, q) in y.qs.iter_mut().enumerate() {+ let v = (iscale * xs[j]).round();+ *q = v.clamp(-128., 127.) as i8+ }+ for j in 0..QK_K / 16 {+ let mut sum = 0i32;+ for ii in 0..16 {+ sum += y.qs[j * 16 + ii] as i32+ }+ y.bsums[j] = sum as i16+ }+ y.d = 1.0 / iscale+ }+ }+ }++ fn to_float(xs: &[Self], ys: &mut [f32]) {+ let k = ys.len();+ debug_assert!(+ k.is_multiple_of(QK_K),+ "dequantize_row_q8k: {k} is not divisible by {QK_K}"+ );+ for (i, x) in xs.iter().enumerate() {+ for (j, &q) in x.qs.iter().enumerate() {+ ys[i * QK_K + j] = x.d * q as f32+ }+ }+ }+}++// https://github.com/ggml-org/llama.cpp/blob/aa3ee0eb0b80efca126cedf9bcb4fb5864b46ce3/ggml/src/ggml-cpu/ggml-cpu.c#L1205+pub fn matmul<T: GgmlType>(+ (m, k, n): (usize, usize, usize),+ lhs: &[f32],+ rhs_t: &[T],+ dst: &mut [f32],+) -> Result<()> {+ debug_assert_eq!(+ T::BLCK_SIZE,+ T::VecDotType::BLCK_SIZE,+ "Mismatched block sizes"+ );+ debug_assert_eq!(+ m * k,+ lhs.len(),+ "unexpected lhs length {} ({m},{k},{n})",+ lhs.len()+ );+ let k_in_blocks = k.div_ceil(T::BLCK_SIZE);++ // Thread-local scratch buffer reused across calls to avoid per-matmul+ // heap allocation of the quantized LHS.+ // Using u64 ensures sufficient alignment regardless of `T::VecDotType`.+ thread_local! {+ static LHS_SCRATCH: std::cell::RefCell<Vec<u64>> =+ const { std::cell::RefCell::new(Vec::new()) };+ }++ let elem_size = std::mem::size_of::<T::VecDotType>();+ // Required scratch buffer length in u64+ let required_scratch_len = (m * k_in_blocks * elem_size).div_ceil(8);++ LHS_SCRATCH.with(|cell| -> Result<()> {+ let mut scratch = cell.borrow_mut();+ if scratch.len() < required_scratch_len {+ scratch.resize(required_scratch_len, 0);+ }+ // SAFETY: u64 ensures sufficient alignment. Resize ensures sufficient size.+ // All elements written before reading.+ let lhs_b: &mut [T::VecDotType] = unsafe {+ std::slice::from_raw_parts_mut(+ scratch.as_mut_ptr() as *mut T::VecDotType,+ m * k_in_blocks,+ )+ };+ // f32, f16, and bf16 support direct copy+ if T::DIRECT_COPY {+ T::VecDotType::direct_copy(lhs, lhs_b);+ } else {+ for row_idx in 0..m {+ let lhs_b_mut = &mut lhs_b[row_idx * k_in_blocks..(row_idx + 1) * k_in_blocks];+ let lhs = &lhs[row_idx * k..(row_idx + 1) * k];+ T::VecDotType::from_float(lhs, lhs_b_mut)+ }+ }+ let n_quad = n & !3;+ let quads_total = n_quad / 4;+ let n_tail = n - n_quad; // 0..=3+ let pool = crate::utils::barrier_pool();+ // Workers 0..n_workers + calling thread as worker n_workers.+ let lhs_b: &[T::VecDotType] = lhs_b;++ for row_idx in 0..m {+ let lhs_row = &lhs_b[row_idx * k_in_blocks..(row_idx + 1) * k_in_blocks];+ let dst_row = &mut dst[row_idx * n..(row_idx + 1) * n];+ let (main, tail) = dst_row.split_at_mut(n_quad);+ let main_ptr = main.as_mut_ptr() as usize;++ let dot_range = |range: std::ops::Range<usize>| {+ let main_ptr = main_ptr as *mut f32;+ for quad_idx in range {+ let col = quad_idx * 4;+ let (d0, d1, d2, d3) = T::vec_dot_4(+ k,+ &rhs_t[col * k_in_blocks..(col + 1) * k_in_blocks],+ &rhs_t[(col + 1) * k_in_blocks..(col + 2) * k_in_blocks],+ &rhs_t[(col + 2) * k_in_blocks..(col + 3) * k_in_blocks],+ &rhs_t[(col + 3) * k_in_blocks..(col + 4) * k_in_blocks],+ lhs_row,+ );+ unsafe {+ let base = main_ptr.add(quad_idx * 4);+ *base = d0;+ *base.add(1) = d1;+ *base.add(2) = d2;+ *base.add(3) = d3;+ }+ }+ };+ if cfg!(target_arch = "x86_64") && T::DTYPE == GgmlDType::Q6K && m == 1 {+ // Compact Q6K GEMV has equal-sized dot products; static ranges+ // avoid the shared cursor overhead on this short decode path.+ pool.execute_static(quads_total, dot_range);+ } else {+ pool.execute_chunked(quads_total, dot_range);+ }+ if n_tail >= 2 {+ let col = n_quad;+ let (d0, d1) = T::vec_dot_2(+ k,+ &rhs_t[col * k_in_blocks..(col + 1) * k_in_blocks],+ &rhs_t[(col + 1) * k_in_blocks..(col + 2) * k_in_blocks],+ lhs_row,+ );+ tail[0] = d0;+ tail[1] = d1;+ }+ if n_tail & 1 == 1 {+ let col = n - 1;+ tail[n_tail - 1] = T::vec_dot(+ k,+ &rhs_t[col * k_in_blocks..(col + 1) * k_in_blocks],+ lhs_row,+ );+ }+ }+ Ok(())+ })+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "dotprod")]+pub(crate) fn matmul_q4k_x8(+ (m, k, n): (usize, usize, usize),+ lhs: &[f32],+ repacked: &[BlockQ4Kx8],+ dst: &mut [f32],+) -> crate::Result<()> {+ use crate::quantized::neon::vec_dot_8_q4k_q8k;+ debug_assert!(n.is_multiple_of(8));+ let k_in_blocks = k / QK_K;+ let n_groups = n / 8;++ thread_local! {+ static LHS_SCRATCH: std::cell::RefCell<Vec<u64>> =+ const { std::cell::RefCell::new(Vec::new()) };+ }++ let elem_size = std::mem::size_of::<BlockQ8K>();+ let required_len = (m * k_in_blocks * elem_size).div_ceil(8);++ LHS_SCRATCH.with(|cell| -> crate::Result<()> {+ let mut scratch = cell.borrow_mut();+ if scratch.len() < required_len {+ scratch.resize(required_len, 0);+ }+ let lhs_b: &mut [BlockQ8K] = unsafe {+ std::slice::from_raw_parts_mut(scratch.as_mut_ptr() as *mut BlockQ8K, m * k_in_blocks)+ };+ for row_idx in 0..m {+ let lhs_row = &lhs[row_idx * k..(row_idx + 1) * k];+ let lhs_b_mut = &mut lhs_b[row_idx * k_in_blocks..(row_idx + 1) * k_in_blocks];+ BlockQ8K::from_float(lhs_row, lhs_b_mut);+ }++ let pool = crate::utils::barrier_pool();+ let lhs_b: &[BlockQ8K] = lhs_b;+ let repacked_ptr = repacked.as_ptr() as usize;+ let x8_block_bytes = std::mem::size_of::<BlockQ4Kx8>();++ // equal-sized gemv units: a static split skips the shared cursor, which costs+ // several us per small decode matmul+ if m == 1 {+ let lhs_row_ptr = lhs_b.as_ptr() as usize;+ let dst_row_ptr = dst.as_mut_ptr() as usize;+ pool.execute_static(n_groups, |range| {+ let lhs_row: &[BlockQ8K] = unsafe {+ std::slice::from_raw_parts(lhs_row_ptr as *const BlockQ8K, k_in_blocks)+ };+ let dst_ptr = dst_row_ptr as *mut f32;+ for g in range {+ let xs = unsafe {+ std::slice::from_raw_parts(+ (repacked_ptr + g * k_in_blocks * x8_block_bytes) as *const BlockQ4Kx8,+ k_in_blocks,+ )+ };+ let results = vec_dot_8_q4k_q8k(k, xs, lhs_row);+ unsafe {+ std::ptr::copy_nonoverlapping(results.as_ptr(), dst_ptr.add(g * 8), 8);+ }+ }+ });+ return Ok(());+ }++ for row_idx in 0..m {+ let lhs_row = &lhs_b[row_idx * k_in_blocks..(row_idx + 1) * k_in_blocks];+ let lhs_row_ptr = lhs_row.as_ptr() as usize;+ let dst_row_ptr = dst[row_idx * n..(row_idx + 1) * n].as_mut_ptr() as usize;++ pool.execute_static(n_groups, |range| {+ let lhs_row: &[BlockQ8K] = unsafe {+ std::slice::from_raw_parts(lhs_row_ptr as *const BlockQ8K, k_in_blocks)+ };+ let dst_ptr = dst_row_ptr as *mut f32;+ for g in range {+ let xs = unsafe {+ std::slice::from_raw_parts(+ (repacked_ptr + g * k_in_blocks * x8_block_bytes) as *const BlockQ4Kx8,+ k_in_blocks,+ )+ };+ let results = vec_dot_8_q4k_q8k(k, xs, lhs_row);+ unsafe {+ std::ptr::copy_nonoverlapping(results.as_ptr(), dst_ptr.add(g * 8), 8);+ }+ }+ });+ }++ Ok(())+ })+}++pub fn matmul_f16<T: GgmlType>(+ mkn: (usize, usize, usize),+ lhs: &[f16],+ rhs_t: &[T],+ dst: &mut [f16],+) -> Result<()> {+ let (m, k, n) = mkn;+ if m * k != lhs.len() {+ crate::bail!("unexpected lhs length {} {mkn:?}", lhs.len());+ }++ let k_in_lhs_blocks = k.div_ceil(T::BLCK_SIZE);+ let k_in_rhs_blocks = k.div_ceil(T::VecDotType::BLCK_SIZE);+ let mut lhs_b = vec![T::VecDotType::zeros(); m * k_in_lhs_blocks];+ for row_idx in 0..m {+ let lhs_b = &mut lhs_b[row_idx * k_in_lhs_blocks..(row_idx + 1) * k_in_lhs_blocks];+ let lhs = &lhs[row_idx * k..(row_idx + 1) * k];+ let lhs_f32: Vec<_> = lhs.iter().map(|&x| x.to_f32()).collect();+ T::VecDotType::from_float(&lhs_f32, lhs_b);+ }+ let lhs_b = lhs_b.as_slice();++ for row_idx in 0..m {+ let lhs_row = &lhs_b[row_idx * k_in_lhs_blocks..(row_idx + 1) * k_in_lhs_blocks];+ let dst_row = &mut dst[row_idx * n..(row_idx + 1) * n];++ for (col_idx, dst) in dst_row.iter_mut().enumerate() {+ let rhs_col = &rhs_t[col_idx * k_in_rhs_blocks..(col_idx + 1) * k_in_rhs_blocks];+ let value = T::vec_dot(k, rhs_col, lhs_row);+ *dst = f16::from_f32(value);+ }+ }+ Ok(())+}++impl GgmlType for f32 {+ const DTYPE: GgmlDType = GgmlDType::F32;+ const BLCK_SIZE: usize = 1;+ const DIRECT_COPY: bool = true;+ type VecDotType = f32;++ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ Self::vec_dot_unopt(n, xs, ys)+ }++ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ debug_assert!(xs.len() >= n, "size mismatch xs {} < {n}", xs.len());+ debug_assert!(ys.len() >= n, "size mismatch ys {} < {n}", ys.len());+ let mut res = 0f32;+ unsafe { crate::cpu::vec_dot_f32(xs.as_ptr(), ys.as_ptr(), &mut res, n) };+ res+ }++ fn from_float(xs: &[f32], ys: &mut [Self]) {+ debug_assert_eq!(+ xs.len(),+ ys.len(),+ "size mismatch xs {} != ys {}",+ xs.len(),+ ys.len()+ );+ ys.copy_from_slice(xs);+ }++ fn to_float(xs: &[Self], ys: &mut [f32]) {+ debug_assert_eq!(+ xs.len(),+ ys.len(),+ "size mismatch xs {} != ys {}",+ xs.len(),+ ys.len()+ );+ ys.copy_from_slice(xs);+ }++ fn direct_copy(xs: &[f32], ys: &mut [Self]) {+ Self::from_float(xs, ys)+ }+}++impl GgmlType for f16 {+ const DTYPE: GgmlDType = GgmlDType::F16;+ const BLCK_SIZE: usize = 1;+ const DIRECT_COPY: bool = true;+ type VecDotType = f16;++ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ Self::vec_dot_unopt(n, xs, ys)+ }++ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ debug_assert!(xs.len() >= n, "size mismatch xs {} < {n}", xs.len());+ debug_assert!(ys.len() >= n, "size mismatch ys {} < {n}", ys.len());+ let mut res = 0f32;+ unsafe { crate::cpu::vec_dot_f16(xs.as_ptr(), ys.as_ptr(), &mut res, n) };+ res+ }++ fn from_float(xs: &[f32], ys: &mut [Self]) {+ debug_assert_eq!(+ xs.len(),+ ys.len(),+ "size mismatch xs {} != ys {}",+ xs.len(),+ ys.len()+ );+ ys.convert_from_f32_slice(xs);+ }++ fn to_float(xs: &[Self], ys: &mut [f32]) {+ debug_assert_eq!(+ xs.len(),+ ys.len(),+ "size mismatch xs {} != ys {}",+ xs.len(),+ ys.len()+ );+ xs.convert_to_f32_slice(ys);+ }++ fn direct_copy(xs: &[f32], ys: &mut [Self]) {+ Self::from_float(xs, ys)+ }+}++impl GgmlType for bf16 {+ const DTYPE: GgmlDType = GgmlDType::BF16;+ const BLCK_SIZE: usize = 1;+ const DIRECT_COPY: bool = true;+ type VecDotType = bf16;++ fn vec_dot(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ Self::vec_dot_unopt(n, xs, ys)+ }++ fn vec_dot_unopt(n: usize, xs: &[Self], ys: &[Self::VecDotType]) -> f32 {+ debug_assert!(xs.len() >= n, "size mismatch xs {} < {n}", xs.len());+ debug_assert!(ys.len() >= n, "size mismatch ys {} < {n}", ys.len());+ let mut res = 0f32;+ unsafe { crate::cpu::vec_dot_bf16(xs.as_ptr(), ys.as_ptr(), &mut res, n) };+ res+ }++ fn from_float(xs: &[f32], ys: &mut [Self]) {+ debug_assert_eq!(+ xs.len(),+ ys.len(),+ "size mismatch xs {} != ys {}",+ xs.len(),+ ys.len()+ );+ ys.convert_from_f32_slice(xs);+ }++ fn to_float(xs: &[Self], ys: &mut [f32]) {+ debug_assert_eq!(+ xs.len(),+ ys.len(),+ "size mismatch xs {} != ys {}",+ xs.len(),+ ys.len()+ );+ xs.convert_to_f32_slice(ys);+ }++ fn direct_copy(xs: &[f32], ys: &mut [Self]) {+ Self::from_float(xs, ys)+ }+}++macro_rules! verify_block_size {+ ( $block_type:ident ) => {+ const _: () =+ assert!($block_type::BLCK_SIZE == <$block_type as GgmlType>::VecDotType::BLCK_SIZE);+ };+}++macro_rules! verify_block_sizes {+ ( $( $block_type:ident ),* ) => {+ $(+ verify_block_size!($block_type);+ )*+ };+}++verify_block_sizes!(+ BlockQ4_0, BlockQ4_1, BlockQ5_0, BlockQ5_1, BlockQ8_0, BlockQ8_1, BlockQ2K, BlockQ3K, BlockQ4K,+ BlockQ5K, BlockQ6K, BlockQ8K, f32, f16, bf16+);diff --git a/vendor/candle-core/src/quantized/metal.rs b/vendor/candle-core/src/quantized/metal.rsnew file mode 100644index 0000000..b00c19b--- /dev/null+++ b/vendor/candle-core/src/quantized/metal.rs@@ -0,0 +1,509 @@+use super::{GgmlDType, QStorage};+use crate::backend::BackendStorage;+use crate::{DType, Layout, MetalDevice, MetalStorage, Result, Shape, D};+use candle_metal_kernels::metal::Buffer;+use std::sync::Arc;++pub struct QMetalStorage {+ dtype: GgmlDType,+ device: MetalDevice,+ buffer: Arc<Buffer>,+}++impl QMetalStorage {+ pub fn zeros(device: &MetalDevice, elem_count: usize, dtype: GgmlDType) -> Result<Self> {+ let size = elem_count * dtype.type_size() / dtype.block_size();+ let buffer = device+ .new_buffer_builder()+ .with_zeros(size)+ .with_label("qstorage_zeros")+ .build()?;+ Ok(Self {+ buffer,+ device: device.clone(),+ dtype,+ })+ }++ pub fn dtype(&self) -> GgmlDType {+ self.dtype+ }++ pub fn device(&self) -> &MetalDevice {+ &self.device+ }++ pub fn buffer(&self) -> &Buffer {+ &self.buffer+ }++ pub fn dequantize(&self, elem_count: usize) -> Result<MetalStorage> {+ use crate::quantized::k_quants::GgmlType;++ let buffer = self+ .device+ .new_buffer_builder()+ .with_size(self.buffer.length())+ .with_label("qstorage_dequantize_blit")+ .build()?;+ {+ let mut blit = self.device.blit_command_encoder()?;+ blit.set_label("blit_to_cpu");+ blit.copy_from_buffer(&self.buffer, 0, &buffer, 0, self.buffer.length());+ }+ self.device.flush_and_wait_current()?;+ let mut out = vec![0.0; elem_count];+ let block_len = elem_count / self.dtype.block_size();+ match self.dtype {+ GgmlDType::F32 => {+ let vec: Vec<f32> = read_to_vec(&buffer, block_len);+ f32::to_float(&vec, &mut out);+ }+ GgmlDType::F16 => {+ let vec: Vec<half::f16> = read_to_vec(&buffer, block_len);+ half::f16::to_float(&vec, &mut out);+ }+ GgmlDType::BF16 => {+ let vec: Vec<half::bf16> = read_to_vec(&buffer, block_len);+ half::bf16::to_float(&vec, &mut out);+ }+ GgmlDType::Q4_0 => {+ let vec: Vec<crate::quantized::BlockQ4_0> = read_to_vec(&buffer, block_len);+ crate::quantized::BlockQ4_0::to_float(&vec, &mut out);+ }+ GgmlDType::Q4_1 => {+ let vec: Vec<crate::quantized::BlockQ4_1> = read_to_vec(&buffer, block_len);+ crate::quantized::BlockQ4_1::to_float(&vec, &mut out);+ }+ GgmlDType::Q5_0 => {+ let vec: Vec<crate::quantized::BlockQ5_0> = read_to_vec(&buffer, block_len);+ crate::quantized::BlockQ5_0::to_float(&vec, &mut out);+ }+ GgmlDType::Q5_1 => {+ let vec: Vec<crate::quantized::BlockQ5_1> = read_to_vec(&buffer, block_len);+ crate::quantized::BlockQ5_1::to_float(&vec, &mut out);+ }+ GgmlDType::Q8_0 => {+ let vec: Vec<crate::quantized::BlockQ8_0> = read_to_vec(&buffer, block_len);+ crate::quantized::BlockQ8_0::to_float(&vec, &mut out);+ }+ GgmlDType::Q8_1 => {+ let vec: Vec<crate::quantized::BlockQ8_1> = read_to_vec(&buffer, block_len);+ crate::quantized::BlockQ8_1::to_float(&vec, &mut out);+ }+ GgmlDType::Q2K => {+ let vec: Vec<crate::quantized::BlockQ2K> = read_to_vec(&buffer, block_len);+ crate::quantized::BlockQ2K::to_float(&vec, &mut out);+ }+ GgmlDType::Q3K => {+ let vec: Vec<crate::quantized::BlockQ3K> = read_to_vec(&buffer, block_len);+ crate::quantized::BlockQ3K::to_float(&vec, &mut out);+ }+ GgmlDType::Q4K => {+ let vec: Vec<crate::quantized::BlockQ4K> = read_to_vec(&buffer, block_len);+ crate::quantized::BlockQ4K::to_float(&vec, &mut out);+ }+ GgmlDType::Q5K => {+ let vec: Vec<crate::quantized::BlockQ5K> = read_to_vec(&buffer, block_len);+ crate::quantized::BlockQ5K::to_float(&vec, &mut out);+ }+ GgmlDType::Q6K => {+ let vec: Vec<crate::quantized::BlockQ6K> = read_to_vec(&buffer, block_len);+ crate::quantized::BlockQ6K::to_float(&vec, &mut out);+ }+ GgmlDType::Q8K => {+ let vec: Vec<crate::quantized::BlockQ8K> = read_to_vec(&buffer, block_len);+ crate::quantized::BlockQ8K::to_float(&vec, &mut out);+ }+ GgmlDType::Mxfp4 => {+ let vec: Vec<crate::quantized::BlockMxfp4> = read_to_vec(&buffer, block_len);+ crate::quantized::BlockMxfp4::to_float(&vec, &mut out);+ }+ }++ let buffer = self+ .device+ .new_buffer_builder()+ .with_data(&out)+ .with_label("qstorage_dequantized")+ .build()?;+ Ok(MetalStorage::new(+ buffer,+ self.device.clone(),+ elem_count,+ DType::F32,+ ))+ }++ pub fn quantize(&mut self, src: &MetalStorage) -> Result<()> {+ // Quantization only happens on CPU for now.+ let src = src.to_cpu::<f32>()?;+ let elem_count = src.len();+ let src = crate::Storage::Cpu(crate::CpuStorage::F32(src));+ let mut qcpu_storage = crate::Device::Cpu.qzeros(elem_count, self.dtype)?;+ qcpu_storage.quantize(&src)?;+ let buffer = self+ .device+ .new_buffer_builder()+ .with_data(&qcpu_storage.data()?)+ .with_label("qstorage_quantized")+ .build()?;+ self.buffer = buffer;+ Ok(())+ }++ pub fn quantize_imatrix(+ &mut self,+ src: &MetalStorage,+ imatrix_weights: &[f32],+ n_per_row: usize,+ ) -> Result<()> {+ // Quantization only happens on CPU for now.+ let src = src.to_cpu::<f32>()?;+ let elem_count = src.len();+ let src = crate::Storage::Cpu(crate::CpuStorage::F32(src));+ let mut qcpu_storage = crate::Device::Cpu.qzeros(elem_count, self.dtype)?;+ qcpu_storage.quantize_imatrix(&src, imatrix_weights, n_per_row)?;+ let buffer = self+ .device+ .new_buffer_builder()+ .with_data(&qcpu_storage.data()?)+ .with_label("qstorage_quantize_imatrix")+ .build()?;+ self.buffer = buffer;+ Ok(())+ }++ pub fn quantize_imatrix_onto(+ &mut self,+ src: &crate::CpuStorage,+ imatrix_weights: &[f32],+ n_per_row: usize,+ ) -> Result<()> {+ // Quantization only happens on CPU for now.+ let elem_count = src.as_slice::<f32>()?.len();+ let mut qcpu_storage = crate::Device::Cpu.qzeros(elem_count, self.dtype)?;++ if let QStorage::Cpu(storage) = &mut qcpu_storage {+ storage.from_float_imatrix(src.as_slice::<f32>()?, imatrix_weights, n_per_row);+ } else {+ unreachable!()+ }++ let buffer = self+ .device+ .new_buffer_builder()+ .with_data(&qcpu_storage.data()?)+ .with_label("qstorage_quantize_imatrix_onto")+ .build()?;+ self.buffer = buffer;+ Ok(())+ }++ pub fn quantize_onto(&mut self, src: &crate::CpuStorage) -> Result<()> {+ // Quantization only happens on CPU for now.+ let elem_count = src.as_slice::<f32>()?.len();+ let mut qcpu_storage = crate::Device::Cpu.qzeros(elem_count, self.dtype)?;++ if let QStorage::Cpu(storage) = &mut qcpu_storage {+ storage.from_float(src.as_slice::<f32>()?);+ } else {+ unreachable!()+ }++ let buffer = self+ .device+ .new_buffer_builder()+ .with_data(&qcpu_storage.data()?)+ .with_label("qstorage_quantize_onto")+ .build()?;+ self.buffer = buffer;+ Ok(())+ }++ pub fn storage_size_in_bytes(&self) -> usize {+ self.buffer.length()+ }++ pub fn embedding(+ &self,+ rows: usize,+ hidden: usize,+ ids: &MetalStorage,+ ids_l: &Layout,+ ) -> Result<MetalStorage> {+ use crate::MetalError;++ if ids.dtype() != DType::U32 {+ crate::bail!("quantized embedding expects u32 ids, got {:?}", ids.dtype())+ }+ if !ids_l.is_contiguous() {+ crate::bail!("quantized embedding requires contiguous ids")+ }+ if !hidden.is_multiple_of(self.dtype.block_size()) {+ crate::bail!(+ "quantized embedding hidden size {hidden} is not divisible by block size {}",+ self.dtype.block_size()+ )+ }+ let expected_size = rows * hidden * self.dtype.type_size() / self.dtype.block_size();+ if self.storage_size_in_bytes() != expected_size {+ crate::bail!(+ "quantized tensor has {} bytes, expected {expected_size}",+ self.storage_size_in_bytes()+ )+ }+ let ids_len = ids_l.shape().elem_count();+ let device = self.device.clone();+ let dst = device+ .new_buffer_builder()+ .with_size_for(ids_len * hidden, DType::F32)+ .with_label("qembedding")+ .build()?;+ let encoder = device.command_encoder()?;+ candle_metal_kernels::call_quantized_get_rows(+ device.device(),+ &encoder,+ device.kernels(),+ candle_metal_kernels::GgmlDType::try_from(self.dtype)?,+ hidden,+ hidden * self.dtype.type_size() / self.dtype.block_size(),+ ids_len,+ &self.buffer,+ ids.buffer(),+ ids_l.start_offset() * DType::U32.size_in_bytes(),+ &dst,+ )+ .map_err(MetalError::from)?;+ Ok(MetalStorage::new(+ dst,+ device.clone(),+ ids_len * hidden,+ DType::F32,+ ))+ }++ fn fwd_mv(+ &self,+ self_shape: &Shape,+ storage: &MetalStorage,+ layout: &crate::Layout,+ ) -> Result<(MetalStorage, Shape)> {+ use crate::MetalError;++ if !layout.is_contiguous() {+ crate::bail!("input tensor is not contiguous {layout:?}")+ }+ let src_shape = layout.shape();+ // self is transposed so n is first then k.+ if src_shape.rank() < 2 {+ crate::bail!("input tensor has only one dimension {layout:?}")+ }+ let (n, k) = self_shape.dims2()?;+ let mut dst_shape = src_shape.dims().to_vec();++ // We always use a single batch dimension and stack all the tensors in the batch on the+ // second dimension as the implementation in candle-metal-kernels doesn't handle batch+ // properly.+ let m = match dst_shape.len() {+ 3 => dst_shape[0] * dst_shape[1],+ 2 => dst_shape[0],+ n => crate::bail!("Invalid rank {n} for quantized matmul metal"),+ };+ let last_k = dst_shape.pop().unwrap();+ if last_k != k {+ crate::bail!("input tensor {layout:?} incompatible with {:?}", self_shape)+ }+ dst_shape.push(n);+ let dst_shape = Shape::from(dst_shape);+ let device = storage.device().clone();+ let dst = device+ .new_buffer_builder()+ .with_size_for(dst_shape.elem_count(), DType::F32)+ .with_label("qmatmul")+ .build()?;+ let encoder = device.command_encoder()?;+ // In some cases it would be better to use the mm variant, though it has its drawbacks+ // around memory alignment.+ for batch_id in 0..m {+ candle_metal_kernels::call_quantized_matmul_mv_t(+ device.device(),+ &encoder,+ device.kernels(),+ candle_metal_kernels::GgmlDType::try_from(self.dtype)?,+ (1, 1, n, k),+ storage.buffer(),+ (layout.start_offset() + batch_id * k) * storage.dtype().size_in_bytes(),+ &self.buffer,+ batch_id * n * DType::F32.size_in_bytes(),+ &dst,+ )+ .map_err(MetalError::from)?;+ }+ let dst_storage =+ crate::MetalStorage::new(dst, device.clone(), dst_shape.elem_count(), DType::F32);+ Ok((dst_storage, dst_shape))+ }++ pub fn fwd(+ &self,+ self_shape: &Shape,+ storage: &MetalStorage,+ layout: &crate::Layout,+ ) -> Result<(MetalStorage, Shape)> {+ use crate::MetalError;++ if !layout.is_contiguous() {+ crate::bail!("input tensor is not contiguous {layout:?}")+ }+ let src_shape = layout.shape();+ // self is transposed so n is first then k.+ if src_shape.rank() < 2 {+ crate::bail!("input tensor has only one dimension {layout:?}")+ }+ let n = self_shape.dim(D::Minus2)?;+ let k = self_shape.dim(D::Minus1)?;+ let mut dst_shape = src_shape.dims().to_vec();++ if src_shape.rank() < self_shape.rank() {+ crate::bail!(+ "input rank ({}) must be >= weight rank ({})",+ src_shape.rank(),+ self_shape.rank()+ )+ }++ if src_shape.dim(D::Minus2)? == 1 || self.dtype == GgmlDType::Mxfp4 {+ return self.fwd_mv(self_shape, storage, layout);+ }++ let last_k = dst_shape.pop().unwrap();+ if last_k != k {+ crate::bail!("input tensor {layout:?} incompatible with {:?}", self_shape)+ }+ dst_shape.push(n);+ let dst_shape = Shape::from(dst_shape);+ let device = storage.device().clone();+ let dst = device+ .new_buffer_builder()+ .with_size_for(dst_shape.elem_count(), DType::F32)+ .with_label("qmatmul")+ .build()?;+ let encoder = device.command_encoder()?;++ assert_eq!(storage.dtype(), DType::F32);++ if self_shape.rank() > 4 {+ crate::bail!("weight rank ({}) must be <= 4", self_shape.rank())+ }+ let src0_l = crate::Layout::contiguous(+ [vec![1; 4 - self_shape.rank()], self_shape.dims().to_vec()].concat(),+ );+ let src0_stride = src0_l+ .stride()+ .iter()+ .map(|x| {+ (*x as f32 * (self.dtype.type_size() as f32 / self.dtype.block_size() as f32))+ as usize+ })+ .collect::<Vec<_>>();++ if src_shape.rank() > 4 {+ crate::bail!("weight rank ({}) must be <= 4", src_shape.rank())+ }+ let src1_l = crate::Layout::contiguous(+ [vec![1; 4 - src_shape.rank()], src_shape.dims().to_vec()].concat(),+ );++ candle_metal_kernels::call_quantized_matmul_mm_t(+ device.device(),+ &encoder,+ device.kernels(),+ candle_metal_kernels::GgmlDType::try_from(self.dtype)?,+ src0_l.dims(),+ &src0_stride,+ &self.buffer,+ src1_l.dims(),+ &src1_l+ .stride()+ .iter()+ .map(|x| x * DType::F32.size_in_bytes())+ .collect::<Vec<_>>(),+ storage.buffer(),+ src1_l.start_offset() * storage.dtype().size_in_bytes(),+ dst_shape.dims(),+ 0,+ &dst,+ )+ .map_err(MetalError::from)?;++ let dst_storage =+ crate::MetalStorage::new(dst, device.clone(), dst_shape.elem_count(), DType::F32);+ Ok((dst_storage, dst_shape))+ }++ pub fn data(&self) -> Result<Vec<u8>> {+ let buffer = self+ .device+ .new_buffer_builder()+ .with_size(self.buffer.length())+ .with_label("qstorage_data_blit")+ .build()?;+ {+ let mut blit = self.device.blit_command_encoder()?;+ blit.set_label("blit_to_cpu");+ blit.copy_from_buffer(&self.buffer, 0, &buffer, 0, self.buffer.length());+ }+ self.device.flush_and_wait_current()?;+ Ok(read_to_vec::<u8>(&buffer, self.storage_size_in_bytes()))+ }+}++pub fn load_quantized<T: super::GgmlType + Send + Sync + 'static>(+ device: &MetalDevice,+ data: &[T],+) -> Result<QStorage> {+ let buffer = device+ .new_buffer_builder()+ .with_data(data)+ .with_label("qstorage_load_quantized")+ .build()?;+ let device = device.clone();+ Ok(QStorage::Metal(QMetalStorage {+ dtype: T::DTYPE,+ device,+ buffer,+ }))+}++fn read_to_vec<T: Clone>(buffer: &Buffer, n: usize) -> Vec<T> {+ let ptr = buffer.contents() as *const T;+ assert!(!ptr.is_null());+ let slice = unsafe { std::slice::from_raw_parts(ptr, n) };+ slice.to_vec()+}++impl TryFrom<GgmlDType> for candle_metal_kernels::GgmlDType {+ type Error = crate::Error;++ fn try_from(value: GgmlDType) -> Result<Self> {+ match value {+ GgmlDType::Q4_0 => Ok(candle_metal_kernels::GgmlDType::Q4_0),+ GgmlDType::Q4_1 => Ok(candle_metal_kernels::GgmlDType::Q4_1),+ GgmlDType::Q5_0 => Ok(candle_metal_kernels::GgmlDType::Q5_0),+ GgmlDType::Q5_1 => Ok(candle_metal_kernels::GgmlDType::Q5_1),+ GgmlDType::Q8_0 => Ok(candle_metal_kernels::GgmlDType::Q8_0),+ GgmlDType::Q8_1 => Ok(candle_metal_kernels::GgmlDType::Q8_1),+ GgmlDType::Q2K => Ok(candle_metal_kernels::GgmlDType::Q2K),+ GgmlDType::Q3K => Ok(candle_metal_kernels::GgmlDType::Q3K),+ GgmlDType::Q4K => Ok(candle_metal_kernels::GgmlDType::Q4K),+ GgmlDType::Q5K => Ok(candle_metal_kernels::GgmlDType::Q5K),+ GgmlDType::Q6K => Ok(candle_metal_kernels::GgmlDType::Q6K),+ GgmlDType::Q8K => Ok(candle_metal_kernels::GgmlDType::Q8K),+ GgmlDType::F16 => Ok(candle_metal_kernels::GgmlDType::F16),+ GgmlDType::F32 => Ok(candle_metal_kernels::GgmlDType::F32),+ GgmlDType::BF16 => Ok(candle_metal_kernels::GgmlDType::BF16),+ GgmlDType::Mxfp4 => Ok(candle_metal_kernels::GgmlDType::Mxfp4),+ }+ }+}diff --git a/vendor/candle-core/src/quantized/mod.rs b/vendor/candle-core/src/quantized/mod.rsnew file mode 100644index 0000000..659066b--- /dev/null+++ b/vendor/candle-core/src/quantized/mod.rs@@ -0,0 +1,1322 @@+use crate::{+ backend::BackendStorage, CpuStorage, DType, Device, Result, Shape, Storage, Tensor, D,+};+use k_quants::*;+use std::borrow::Cow;++#[cfg(any(target_arch = "x86", target_arch = "x86_64"))]+pub mod avx;+mod dummy_cuda;+mod dummy_metal;+pub mod ggml_file;+pub mod gguf_file;+pub mod imatrix_file;+pub mod k_quants;+#[cfg(feature = "metal")]+pub mod metal;+pub mod repack;+#[cfg(target_arch = "x86_64")]+pub(crate) mod repack_x86;+#[cfg(not(target_arch = "wasm32"))]+pub mod tokenizer;+#[cfg(not(feature = "metal"))]+mod metal {+ pub use super::dummy_metal::*;+}+#[cfg(feature = "cuda")]+pub mod cuda;+#[cfg(feature = "cuda")]+pub mod fast_mmq;+#[cfg(feature = "cuda")]+pub mod fast_mmvq;+#[cfg(not(feature = "cuda"))]+mod cuda {+ pub use super::dummy_cuda::*;+}++#[cfg(any(+ target_arch = "aarch64",+ all(target_arch = "arm", target_feature = "neon")+))]+pub mod neon;+#[cfg(target_feature = "simd128")]+pub mod simd128;+pub mod utils;+use half::{bf16, f16};++pub use k_quants::GgmlType;++fn as_t_slice<T>(data: &[u8]) -> &[T] {+ let size = std::mem::size_of::<T>();+ assert_eq!(+ data.len() % size,+ 0,+ "Data length must be a multiple of T's size"+ );+ let ptr = data.as_ptr();+ assert_eq!(+ (ptr as usize) % std::mem::align_of::<T>(),+ 0,+ "Data pointer must be aligned to T's alignment"+ );+ unsafe { std::slice::from_raw_parts(ptr as *const T, data.len() / size) }+}++pub struct QTensor {+ storage: QStorage,+ shape: Shape,+ #[allow(dead_code)]+ repacked_qs: repack::PackedCache,+}++impl Device {+ fn qzeros(&self, elem_count: usize, dtype: GgmlDType) -> Result<QStorage> {+ match self {+ Device::Cpu => {+ let storage = dtype.cpu_zeros(elem_count);+ Ok(QStorage::Cpu(storage))+ }+ Device::Metal(metal) => {+ let storage = metal::QMetalStorage::zeros(metal, elem_count, dtype)?;+ Ok(QStorage::Metal(storage))+ }+ Device::Cuda(cuda) => {+ let storage = cuda::QCudaStorage::zeros(cuda, elem_count, dtype)?;+ Ok(QStorage::Cuda(storage))+ }+ }+ }+}++pub enum QStorage {+ Cpu(Box<dyn QuantizedType>),+ Metal(metal::QMetalStorage),+ Cuda(cuda::QCudaStorage),+}++impl QStorage {+ pub fn from_data(data: Cow<'_, [u8]>, device: &Device, dtype: GgmlDType) -> Result<Self> {+ let data: &[u8] = &data;+ match device {+ Device::Cpu => Ok(Self::Cpu(dtype.from_data(Cow::Borrowed(data)))),+ Device::Metal(d) => match dtype {+ GgmlDType::F32 => metal::load_quantized(d, as_t_slice::<f32>(data)),+ GgmlDType::F16 => metal::load_quantized(d, as_t_slice::<f16>(data)),+ GgmlDType::Q4_0 => metal::load_quantized(d, as_t_slice::<BlockQ4_0>(data)),+ GgmlDType::Q4_1 => metal::load_quantized(d, as_t_slice::<BlockQ4_1>(data)),+ GgmlDType::Q5_0 => metal::load_quantized(d, as_t_slice::<BlockQ5_0>(data)),+ GgmlDType::Q5_1 => metal::load_quantized(d, as_t_slice::<BlockQ5_1>(data)),+ GgmlDType::Q8_0 => metal::load_quantized(d, as_t_slice::<BlockQ8_0>(data)),+ GgmlDType::Q8_1 => metal::load_quantized(d, as_t_slice::<BlockQ8_1>(data)),+ GgmlDType::Q2K => metal::load_quantized(d, as_t_slice::<BlockQ2K>(data)),+ GgmlDType::Q3K => metal::load_quantized(d, as_t_slice::<BlockQ3K>(data)),+ GgmlDType::Q4K => metal::load_quantized(d, as_t_slice::<BlockQ4K>(data)),+ GgmlDType::Q5K => metal::load_quantized(d, as_t_slice::<BlockQ5K>(data)),+ GgmlDType::Q6K => metal::load_quantized(d, as_t_slice::<BlockQ6K>(data)),+ GgmlDType::Q8K => metal::load_quantized(d, as_t_slice::<BlockQ8K>(data)),+ GgmlDType::BF16 => metal::load_quantized(d, as_t_slice::<bf16>(data)),+ GgmlDType::Mxfp4 => metal::load_quantized(d, as_t_slice::<BlockMxfp4>(data)),+ },+ Device::Cuda(d) => match dtype {+ GgmlDType::F32 => cuda::load_quantized(d, as_t_slice::<f32>(data)),+ GgmlDType::F16 => cuda::load_quantized(d, as_t_slice::<f16>(data)),+ GgmlDType::Q4_0 => cuda::load_quantized(d, as_t_slice::<BlockQ4_0>(data)),+ GgmlDType::Q4_1 => cuda::load_quantized(d, as_t_slice::<BlockQ4_1>(data)),+ GgmlDType::Q5_0 => cuda::load_quantized(d, as_t_slice::<BlockQ5_0>(data)),+ GgmlDType::Q5_1 => cuda::load_quantized(d, as_t_slice::<BlockQ5_1>(data)),+ GgmlDType::Q8_0 => cuda::load_quantized(d, as_t_slice::<BlockQ8_0>(data)),+ GgmlDType::Q8_1 => cuda::load_quantized(d, as_t_slice::<BlockQ8_1>(data)),+ GgmlDType::Q2K => cuda::load_quantized(d, as_t_slice::<BlockQ2K>(data)),+ GgmlDType::Q3K => cuda::load_quantized(d, as_t_slice::<BlockQ3K>(data)),+ GgmlDType::Q4K => cuda::load_quantized(d, as_t_slice::<BlockQ4K>(data)),+ GgmlDType::Q5K => cuda::load_quantized(d, as_t_slice::<BlockQ5K>(data)),+ GgmlDType::Q6K => cuda::load_quantized(d, as_t_slice::<BlockQ6K>(data)),+ GgmlDType::Q8K => cuda::load_quantized(d, as_t_slice::<BlockQ8K>(data)),+ GgmlDType::BF16 => cuda::load_quantized(d, as_t_slice::<bf16>(data)),+ GgmlDType::Mxfp4 => cuda::load_quantized(d, as_t_slice::<BlockMxfp4>(data)),+ },+ }+ }++ fn block_size(&self) -> usize {+ match self {+ QStorage::Cpu(storage) => storage.block_size(),+ QStorage::Metal(storage) => storage.dtype().block_size(),+ QStorage::Cuda(storage) => storage.dtype().block_size(),+ }+ }++ fn dtype(&self) -> GgmlDType {+ match self {+ QStorage::Cpu(storage) => storage.dtype(),+ QStorage::Metal(storage) => storage.dtype(),+ QStorage::Cuda(storage) => storage.dtype(),+ }+ }++ fn device(&self) -> Device {+ match self {+ QStorage::Cpu(_storage) => Device::Cpu,+ QStorage::Metal(storage) => Device::Metal(storage.device().clone()),+ QStorage::Cuda(storage) => Device::Cuda(storage.device().clone()),+ }+ }++ fn size_in_bytes(&self) -> usize {+ match self {+ QStorage::Cpu(storage) => storage.storage_size_in_bytes(),+ QStorage::Metal(storage) => storage.storage_size_in_bytes(),+ QStorage::Cuda(storage) => storage.storage_size_in_bytes(),+ }+ }++ fn quantize(&mut self, src: &Storage) -> Result<()> {+ match (self, src) {+ (QStorage::Cpu(storage), Storage::Cpu(src)) => {+ storage.from_float(src.as_slice::<f32>()?);+ }+ (QStorage::Metal(storage), Storage::Metal(src)) => storage.quantize(src)?,+ (QStorage::Cuda(storage), Storage::Cuda(src)) => storage.quantize(src)?,+ _ => crate::bail!("Invalid quantize storage locations do not match"),+ }+ Ok(())+ }++ fn quantize_imatrix(+ &mut self,+ src: &Storage,+ imatrix_weights: &[f32],+ n_per_row: usize,+ ) -> Result<()> {+ match (self, src) {+ (QStorage::Cpu(storage), Storage::Cpu(src)) => {+ storage.from_float_imatrix(src.as_slice::<f32>()?, imatrix_weights, n_per_row);+ }+ (QStorage::Metal(storage), Storage::Metal(src)) => {+ storage.quantize_imatrix(src, imatrix_weights, n_per_row)?+ }+ (QStorage::Cuda(storage), Storage::Cuda(src)) => {+ storage.quantize_imatrix(src, imatrix_weights, n_per_row)?+ }+ _ => crate::bail!("Invalid quantize storage locations do not match"),+ }+ Ok(())+ }++ fn quantize_onto(&mut self, src: &Storage) -> Result<()> {+ match (self, src) {+ (QStorage::Cpu(storage), Storage::Cpu(src)) => {+ storage.from_float(src.as_slice::<f32>()?);+ }+ (QStorage::Metal(storage), Storage::Cpu(src)) => storage.quantize_onto(src)?,+ (QStorage::Cuda(storage), Storage::Cpu(src)) => storage.quantize_onto(src)?,+ _ => crate::bail!("Invalid quantize source storage locations: not on cpu"),+ }+ Ok(())+ }++ fn quantize_imatrix_onto(+ &mut self,+ src: &Storage,+ imatrix_weights: &[f32],+ n_per_row: usize,+ ) -> Result<()> {+ match (self, src) {+ (QStorage::Cpu(storage), Storage::Cpu(src)) => {+ storage.from_float_imatrix(src.as_slice::<f32>()?, imatrix_weights, n_per_row);+ }+ (QStorage::Metal(storage), Storage::Cpu(src)) => {+ storage.quantize_imatrix_onto(src, imatrix_weights, n_per_row)?+ }+ (QStorage::Cuda(storage), Storage::Cpu(src)) => {+ storage.quantize_imatrix_onto(src, imatrix_weights, n_per_row)?+ }+ _ => crate::bail!("Invalid quantize storage locations do not match"),+ }+ Ok(())+ }++ fn dequantize(&self, elem_count: usize) -> Result<Storage> {+ match self {+ QStorage::Cpu(storage) => Ok(Storage::Cpu(storage.dequantize(elem_count)?)),+ QStorage::Metal(storage) => Ok(Storage::Metal(storage.dequantize(elem_count)?)),+ QStorage::Cuda(storage) => Ok(Storage::Cuda(storage.dequantize(elem_count)?)),+ }+ }++ fn data(&self) -> Result<Cow<'_, [u8]>> {+ match self {+ QStorage::Cpu(storage) => {+ let data_ptr = storage.as_ptr();+ let size_in_bytes = storage.storage_size_in_bytes();+ let data = unsafe { std::slice::from_raw_parts(data_ptr, size_in_bytes) };+ Ok(Cow::from(data))+ }+ QStorage::Cuda(storage) => Ok(Cow::from(storage.data()?)),+ QStorage::Metal(storage) => Ok(Cow::from(storage.data()?)),+ }+ }++ pub fn device_ptr(&self) -> Result<*const u8> {+ match self {+ QStorage::Cuda(storage) => storage.device_ptr(),+ QStorage::Metal(_) | QStorage::Cpu(_) => {+ crate::bail!("not implemented");+ }+ }+ }++ #[cfg(feature = "cuda")]+ pub fn device_ptr_with_guard<'a>(+ &'a self,+ stream: &'a crate::cuda_backend::cudarc::driver::CudaStream,+ ) -> Result<(+ *const u8,+ crate::cuda_backend::cudarc::driver::SyncOnDrop<'a>,+ )> {+ match self {+ QStorage::Cuda(storage) => storage.device_ptr_with_guard(stream),+ QStorage::Metal(_) | QStorage::Cpu(_) => {+ crate::bail!("not implemented");+ }+ }+ }+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]+pub enum GgmlDType {+ F32,+ F16,+ BF16,+ Q4_0,+ Q4_1,+ Q5_0,+ Q5_1,+ Q8_0,+ Q8_1,+ Q2K,+ Q3K,+ Q4K,+ Q5K,+ Q6K,+ Q8K,+ Mxfp4,+}++impl GgmlDType {+ pub(crate) fn from_u32(u: u32) -> Result<Self> {+ let dtype = match u {+ 0 => Self::F32,+ 1 => Self::F16,+ 2 => Self::Q4_0,+ 3 => Self::Q4_1,+ 6 => Self::Q5_0,+ 7 => Self::Q5_1,+ 8 => Self::Q8_0,+ 9 => Self::Q8_1,+ 10 => Self::Q2K,+ 11 => Self::Q3K,+ 12 => Self::Q4K,+ 13 => Self::Q5K,+ 14 => Self::Q6K,+ 15 => Self::Q8K,+ // https://github.com/ggerganov/ggml/blob/29d87fc6676e7ed0cdfdec0804b06001d9c2bb44/include/ggml.h#L389+ 30 => Self::BF16,+ 39 => Self::Mxfp4,+ _ => crate::bail!("unknown dtype for tensor {u}"),+ };+ Ok(dtype)+ }++ pub(crate) fn to_u32(self) -> u32 {+ match self {+ Self::F32 => 0,+ Self::F16 => 1,+ Self::Q4_0 => 2,+ Self::Q4_1 => 3,+ Self::Q5_0 => 6,+ Self::Q5_1 => 7,+ Self::Q8_0 => 8,+ Self::Q8_1 => 9,+ Self::Q2K => 10,+ Self::Q3K => 11,+ Self::Q4K => 12,+ Self::Q5K => 13,+ Self::Q6K => 14,+ Self::Q8K => 15,+ // https://github.com/ggerganov/ggml/blob/29d87fc6676e7ed0cdfdec0804b06001d9c2bb44/include/ggml.h#L389+ Self::BF16 => 30,+ Self::Mxfp4 => 39,+ }+ }++ /// The block dtype+ pub fn cpu_zeros(&self, elem_count: usize) -> Box<dyn QuantizedType> {+ match self {+ Self::F32 => Box::new(vec![f32::zeros(); elem_count]),+ Self::F16 => Box::new(vec![f16::zeros(); elem_count]),+ Self::Q4_0 => Box::new(vec![BlockQ4_0::zeros(); elem_count / BlockQ4_0::BLCK_SIZE]),+ Self::Q4_1 => Box::new(vec![BlockQ4_1::zeros(); elem_count / BlockQ4_1::BLCK_SIZE]),+ Self::Q5_0 => Box::new(vec![BlockQ5_0::zeros(); elem_count / BlockQ5_0::BLCK_SIZE]),+ Self::Q5_1 => Box::new(vec![BlockQ5_1::zeros(); elem_count / BlockQ5_1::BLCK_SIZE]),+ Self::Q8_0 => Box::new(vec![BlockQ8_0::zeros(); elem_count / BlockQ8_0::BLCK_SIZE]),+ Self::Q8_1 => Box::new(vec![BlockQ8_1::zeros(); elem_count / BlockQ8_1::BLCK_SIZE]),+ Self::Q2K => Box::new(vec![BlockQ2K::zeros(); elem_count / BlockQ2K::BLCK_SIZE]),+ Self::Q3K => Box::new(vec![BlockQ3K::zeros(); elem_count / BlockQ3K::BLCK_SIZE]),+ Self::Q4K => Box::new(vec![BlockQ4K::zeros(); elem_count / BlockQ4K::BLCK_SIZE]),+ Self::Q5K => Box::new(vec![BlockQ5K::zeros(); elem_count / BlockQ5K::BLCK_SIZE]),+ Self::Q6K => Box::new(vec![BlockQ6K::zeros(); elem_count / BlockQ6K::BLCK_SIZE]),+ Self::Q8K => Box::new(vec![BlockQ8K::zeros(); elem_count / BlockQ8K::BLCK_SIZE]),+ Self::BF16 => Box::new(vec![bf16::zeros(); elem_count]),+ Self::Mxfp4 => Box::new(vec![+ BlockMxfp4::zeros();+ elem_count / BlockMxfp4::BLCK_SIZE+ ]),+ }+ }++ pub fn from_data(&self, data: Cow<'_, [u8]>) -> Box<dyn QuantizedType> {+ let data: &[u8] = &data;+ match self {+ Self::F32 => Box::new(as_t_slice::<f32>(data).to_vec()),+ Self::F16 => Box::new(as_t_slice::<f16>(data).to_vec()),+ Self::Q4_0 => Box::new(as_t_slice::<BlockQ4_0>(data).to_vec()),+ Self::Q4_1 => Box::new(as_t_slice::<BlockQ4_1>(data).to_vec()),+ Self::Q5_0 => Box::new(as_t_slice::<BlockQ5_0>(data).to_vec()),+ Self::Q5_1 => Box::new(as_t_slice::<BlockQ5_1>(data).to_vec()),+ Self::Q8_0 => Box::new(as_t_slice::<BlockQ8_0>(data).to_vec()),+ Self::Q8_1 => Box::new(as_t_slice::<BlockQ8_1>(data).to_vec()),+ Self::Q2K => Box::new(as_t_slice::<BlockQ2K>(data).to_vec()),+ Self::Q3K => Box::new(as_t_slice::<BlockQ3K>(data).to_vec()),+ Self::Q4K => Box::new(as_t_slice::<BlockQ4K>(data).to_vec()),+ Self::Q5K => Box::new(as_t_slice::<BlockQ5K>(data).to_vec()),+ Self::Q6K => Box::new(as_t_slice::<BlockQ6K>(data).to_vec()),+ Self::Q8K => Box::new(as_t_slice::<BlockQ8K>(data).to_vec()),+ Self::BF16 => Box::new(as_t_slice::<bf16>(data).to_vec()),+ Self::Mxfp4 => Box::new(as_t_slice::<BlockMxfp4>(data).to_vec()),+ }+ }++ /// The type size for blocks in bytes.+ pub fn type_size(&self) -> usize {+ use k_quants::*;+ match self {+ Self::F32 => 4,+ Self::F16 | Self::BF16 => 2,+ Self::Q4_0 => std::mem::size_of::<BlockQ4_0>(),+ Self::Q4_1 => std::mem::size_of::<BlockQ4_1>(),+ Self::Q5_0 => std::mem::size_of::<BlockQ5_0>(),+ Self::Q5_1 => std::mem::size_of::<BlockQ5_1>(),+ // https://github.com/ggerganov/llama.cpp/blob/468ea24fb4633a0d681f7ac84089566c1c6190cb/ggml.c#L932+ Self::Q8_0 => std::mem::size_of::<BlockQ8_0>(),+ Self::Q8_1 => std::mem::size_of::<BlockQ8_1>(),+ Self::Q2K => std::mem::size_of::<BlockQ2K>(),+ Self::Q3K => std::mem::size_of::<BlockQ3K>(),+ Self::Q4K => std::mem::size_of::<BlockQ4K>(),+ Self::Q5K => std::mem::size_of::<BlockQ5K>(),+ Self::Q6K => std::mem::size_of::<BlockQ6K>(),+ Self::Q8K => std::mem::size_of::<BlockQ8K>(),+ Self::Mxfp4 => std::mem::size_of::<BlockMxfp4>(),+ }+ }++ /// The block size, i.e. the number of elements stored in each block.+ pub fn block_size(&self) -> usize {+ match self {+ Self::F32 => 1,+ Self::F16 | Self::BF16 => 1,+ Self::Q4_0 => k_quants::QK4_0,+ Self::Q4_1 => k_quants::QK4_1,+ Self::Q5_0 => k_quants::QK5_0,+ Self::Q5_1 => k_quants::QK5_1,+ Self::Q8_0 => k_quants::QK8_0,+ Self::Q8_1 => k_quants::QK8_1,+ Self::Q2K | Self::Q3K | Self::Q4K | Self::Q5K | Self::Q6K | Self::Q8K => k_quants::QK_K,+ Self::Mxfp4 => k_quants::QK_MXFP4,+ }+ }+}++// A version of GgmlType without `vec_dot` so that it can be dyn boxed.+pub trait QuantizedType: Send + Sync {+ fn dtype(&self) -> GgmlDType;+ fn matmul_t(&self, mkn: (usize, usize, usize), lhs: &[f32], dst: &mut [f32]) -> Result<()>;+ fn matmul_t_f16(&self, mkn: (usize, usize, usize), lhs: &[f16], dst: &mut [f16]) -> Result<()>;+ fn embedding(&self, ids: &[u32], rows: usize, hidden: usize) -> Result<CpuStorage>;+ fn dequantize(&self, elem_count: usize) -> Result<CpuStorage>;+ fn storage_size_in_bytes(&self) -> usize;+ fn as_ptr(&self) -> *const u8;+ fn block_size(&self) -> usize;+ #[allow(clippy::wrong_self_convention)]+ fn from_float(&mut self, xs: &[f32]);+ #[allow(clippy::wrong_self_convention)]+ fn from_float_imatrix(&mut self, xs: &[f32], imatrix_weights: &[f32], n_per_row: usize);+ fn size(&self) -> usize;+}++impl<T: k_quants::GgmlType + Send + Sync> QuantizedType for Vec<T> {+ fn matmul_t(&self, mkn: (usize, usize, usize), lhs: &[f32], dst: &mut [f32]) -> Result<()> {+ k_quants::matmul(mkn, lhs, self.as_slice(), dst)+ }+ fn matmul_t_f16(&self, mkn: (usize, usize, usize), lhs: &[f16], dst: &mut [f16]) -> Result<()> {+ k_quants::matmul_f16(mkn, lhs, self.as_slice(), dst)+ }++ fn embedding(&self, ids: &[u32], rows: usize, hidden: usize) -> Result<CpuStorage> {+ if !hidden.is_multiple_of(T::BLCK_SIZE) {+ crate::bail!(+ "quantized embedding hidden size {hidden} is not divisible by block size {}",+ T::BLCK_SIZE+ )+ }+ let row_blocks = hidden / T::BLCK_SIZE;+ if self.len() != rows * row_blocks {+ crate::bail!(+ "quantized tensor has {} blocks, expected {}",+ self.len(),+ rows * row_blocks+ )+ }+ let mut out = vec![0f32; ids.len() * hidden];+ for (out_row, &row_id) in ids.iter().enumerate() {+ let row = row_id as usize;+ if row >= rows {+ crate::bail!("embedding id {row} is out of range for {rows} rows")+ }+ let src = &self[row * row_blocks..(row + 1) * row_blocks];+ let dst = &mut out[out_row * hidden..(out_row + 1) * hidden];+ T::to_float(src, dst);+ }+ Ok(CpuStorage::F32(out))+ }++ fn size(&self) -> usize {+ self.len() * core::mem::size_of::<T>()+ }++ fn from_float(&mut self, xs: &[f32]) {+ T::from_float(xs, self)+ }++ fn from_float_imatrix(&mut self, xs: &[f32], imatrix_weights: &[f32], n_per_row: usize) {+ T::from_float_imatrix(xs, self, imatrix_weights, n_per_row)+ }++ fn dtype(&self) -> GgmlDType {+ T::DTYPE+ }++ fn block_size(&self) -> usize {+ T::BLCK_SIZE+ }++ fn dequantize(&self, elem_count: usize) -> Result<CpuStorage> {+ let mut ys = vec![0.0f32; elem_count];+ T::to_float(self.as_slice(), &mut ys);+ Ok(CpuStorage::F32(ys))+ }++ fn storage_size_in_bytes(&self) -> usize {+ self.len() * std::mem::size_of::<T>()+ }++ fn as_ptr(&self) -> *const u8 {+ self.as_ptr() as *const u8+ }+}++impl std::fmt::Debug for QTensor {+ fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {+ write!(f, "QTensor[{:?}; {:?}]", self.shape, self.dtype())+ }+}++fn check_shape(shape: &Shape, block_size: usize) -> Result<()> {+ let dims = shape.dims();+ if dims.is_empty() {+ crate::bail!("scalar tensor cannot be quantized {shape:?}")+ }+ if !dims[dims.len() - 1].is_multiple_of(block_size) {+ crate::bail!(+ "quantized tensor must have their last dim divisible by block size {shape:?} {}",+ block_size+ )+ }+ Ok(())+}++impl QTensor {+ pub fn new<S: Into<Shape>>(storage: QStorage, shape: S) -> Result<Self> {+ let shape = shape.into();+ check_shape(&shape, storage.block_size())?;+ Ok(Self {+ storage,+ shape,+ repacked_qs: repack::PackedCache::new(),+ })+ }++ pub fn quantize(src: &Tensor, dtype: GgmlDType) -> Result<Self> {+ let shape = src.shape();+ let block_size = dtype.block_size();+ check_shape(shape, block_size)?;+ let src = src.to_dtype(crate::DType::F32)?.flatten_all()?;+ let elem_count = shape.elem_count();+ if !elem_count.is_multiple_of(block_size) {+ crate::bail!(+ "tensor size ({shape:?}) is not divisible by block size {}",+ block_size+ )+ }+ let mut storage = src.device().qzeros(elem_count, dtype)?;+ storage.quantize(&src.storage())?;+ Ok(Self {+ storage,+ shape: shape.clone(),+ repacked_qs: repack::PackedCache::new(),+ })+ }++ pub fn quantize_imatrix(+ src: &Tensor,+ imatrix_weights: &[f32],+ dtype: GgmlDType,+ ) -> Result<Self> {+ // (n_per_row/QK_K-1)*QK_K+(QK_K/32-1)*32+32=n_per_row+ // Size of imatrix == last dim of tensor+ let n_per_row = src.dim(D::Minus1)?;+ if imatrix_weights.len() != n_per_row {+ crate::bail!(+ "imatrix weights must have the same length {} as the last dim of src {}",+ imatrix_weights.len(),+ src.dim(D::Minus1)?+ );+ }++ let shape = src.shape();+ let block_size = dtype.block_size();+ check_shape(shape, block_size)?;+ let src = src.to_dtype(crate::DType::F32)?.flatten_all()?;+ let elem_count = shape.elem_count();+ if !elem_count.is_multiple_of(block_size) {+ crate::bail!(+ "tensor size ({shape:?}) is not divisible by block size {}",+ block_size+ );+ }+ let mut storage = src.device().qzeros(elem_count, dtype)?;+ storage.quantize_imatrix(&src.storage(), imatrix_weights, n_per_row)?;+ Ok(Self {+ storage,+ shape: shape.clone(),+ repacked_qs: repack::PackedCache::new(),+ })+ }++ /// Quantize `src` (currently on the CPU) to a QTensor on `dev`+ pub fn quantize_imatrix_onto(+ src: &Tensor,+ imatrix_weights: &[f32],+ dtype: GgmlDType,+ dev: &Device,+ ) -> Result<Self> {+ if !src.device().is_cpu() {+ crate::bail!(+ "`quantize_onto` expects a `src` to be on the cpu, got {:?}.",+ src.device()+ )+ }+ // (n_per_row/QK_K-1)*QK_K+(QK_K/32-1)*32+32=n_per_row+ // Size of imatrix == last dim of tensor+ let n_per_row = src.dim(D::Minus1)?;+ if imatrix_weights.len() != n_per_row {+ crate::bail!(+ "imatrix weights must have the same length {} as the last dim of src {}",+ imatrix_weights.len(),+ src.dim(D::Minus1)?+ );+ }+ let shape = src.shape();+ let block_size = dtype.block_size();+ check_shape(shape, block_size)?;+ let src = src.to_dtype(crate::DType::F32)?.flatten_all()?;+ let elem_count = shape.elem_count();+ if !elem_count.is_multiple_of(block_size) {+ crate::bail!(+ "tensor size ({shape:?}) is not divisible by block size {}",+ block_size+ )+ }+ // storage is on the `dev`, src is on `cpu`+ let mut storage = dev.qzeros(elem_count, dtype)?;+ storage.quantize_imatrix_onto(&src.storage(), imatrix_weights, n_per_row)?;+ Ok(Self {+ storage,+ shape: shape.clone(),+ repacked_qs: repack::PackedCache::new(),+ })+ }++ /// Quantize `src` (currently on the CPU) to a QTensor on `dev`+ pub fn quantize_onto(src: &Tensor, dtype: GgmlDType, dev: &Device) -> Result<Self> {+ if !src.device().is_cpu() {+ crate::bail!(+ "`quantize_onto` expects a `src` to be on the cpu, got {:?}.",+ src.device()+ )+ }+ let shape = src.shape();+ let block_size = dtype.block_size();+ check_shape(shape, block_size)?;+ let src = src.to_dtype(crate::DType::F32)?.flatten_all()?;+ let elem_count = shape.elem_count();+ if !elem_count.is_multiple_of(block_size) {+ crate::bail!(+ "tensor size ({shape:?}) is not divisible by block size {}",+ block_size+ )+ }+ // storage is on the `dev`, src is on `cpu`+ let mut storage = dev.qzeros(elem_count, dtype)?;+ storage.quantize_onto(&src.storage())?;+ Ok(Self {+ storage,+ shape: shape.clone(),+ repacked_qs: repack::PackedCache::new(),+ })+ }++ pub fn dtype(&self) -> GgmlDType {+ self.storage.dtype()+ }++ pub fn device(&self) -> Device {+ self.storage.device()+ }++ pub fn rank(&self) -> usize {+ self.shape.rank()+ }++ pub fn shape(&self) -> &Shape {+ &self.shape+ }++ pub fn dequantize(&self, device: &Device) -> Result<Tensor> {+ let storage = self.storage.dequantize(self.shape.elem_count())?;+ let none = crate::op::BackpropOp::none();+ crate::tensor::from_storage(storage, self.shape.clone(), none, false).to_device(device)+ }++ pub fn dequantize_f16(&self, device: &Device) -> Result<Tensor> {+ // In the CUDA case, we have a specialized kernel as this can be useful for volta+ // architectures. https://github.com/huggingface/candle/issues/2136+ match &self.storage {+ QStorage::Cuda(s) => {+ let s = s.dequantize_f16(self.shape.elem_count())?;+ let none = crate::op::BackpropOp::none();+ crate::tensor::from_storage(Storage::Cuda(s), self.shape.clone(), none, false)+ .to_device(device)+ }+ _ => {+ let s = self.dequantize(device)?.to_dtype(crate::DType::F16)?;+ Ok(s)+ }+ }+ }++ pub fn embedding(&self, ids: &Tensor) -> Result<Tensor> {+ let (rows, hidden) = self.shape.dims2()?;+ if !hidden.is_multiple_of(self.dtype().block_size()) {+ crate::bail!(+ "quantized embedding hidden size {hidden} is not divisible by block size {}",+ self.dtype().block_size()+ )+ }+ let mut out_shape = ids.dims().to_vec();+ out_shape.push(hidden);+ let device = self.device();+ let ids = ids+ .to_device(&device)?+ .to_dtype(DType::U32)?+ .flatten_all()?+ .contiguous()?;+ let storage = match &self.storage {+ QStorage::Cpu(storage) => {+ let ids = ids.to_vec1::<u32>()?;+ Storage::Cpu(storage.embedding(&ids, rows, hidden)?)+ }+ QStorage::Metal(storage) => match &*ids.storage() {+ Storage::Metal(ids_storage) => {+ Storage::Metal(storage.embedding(rows, hidden, ids_storage, ids.layout())?)+ }+ _ => unreachable!("ids were moved to the QTensor device"),+ },+ QStorage::Cuda(storage) => match &*ids.storage() {+ Storage::Cuda(ids_storage) => {+ Storage::Cuda(storage.embedding(rows, hidden, ids_storage, ids.layout())?)+ }+ _ => unreachable!("ids were moved to the QTensor device"),+ },+ };+ let none = crate::op::BackpropOp::none();+ Ok(crate::tensor::from_storage(storage, out_shape, none, false))+ }++ pub fn storage_size_in_bytes(&self) -> usize {+ self.storage.size_in_bytes()+ }++ pub fn data(&self) -> Result<Cow<'_, [u8]>> {+ self.storage.data()+ }++ pub fn indexed_moe_forward(&self, x: &Tensor, ids: &Tensor) -> Result<Tensor> {+ match &self.storage {+ QStorage::Cuda(s) => match (&*x.storage(), &*ids.storage()) {+ (Storage::Cuda(x_storage), Storage::Cuda(ids_storage)) => {+ let (storage, out_shape) = s.indexed_moe_forward(+ self.shape(),+ x_storage,+ x.layout(),+ ids_storage,+ ids.layout(),+ )?;+ Ok(crate::tensor::from_storage(+ Storage::Cuda(storage),+ out_shape,+ crate::op::BackpropOp::none(),+ false,+ ))+ }+ _ => {+ panic!("Non-cuda indexed_moe_forward is not implemented!");+ }+ },+ _ => {+ panic!("indexed_moe_forward is not implemented in this platform!");+ }+ }+ }++ pub fn device_ptr(&self) -> Result<*const u8> {+ match &self.storage {+ QStorage::Cuda(storage) => storage.device_ptr(),+ QStorage::Metal(_) | QStorage::Cpu(_) => {+ crate::bail!("not implemented");+ }+ }+ }++ #[cfg(feature = "cuda")]+ pub fn device_ptr_with_guard<'a>(+ &'a self,+ stream: &'a crate::cuda_backend::cudarc::driver::CudaStream,+ ) -> Result<(+ *const u8,+ crate::cuda_backend::cudarc::driver::SyncOnDrop<'a>,+ )> {+ self.storage.device_ptr_with_guard(stream)+ }+}++#[derive(Clone, Debug)]+pub enum QMatMul {+ QTensor(std::sync::Arc<QTensor>),+ Tensor(Tensor),+ TensorF16(Tensor),+}++thread_local! {+ static DEQUANTIZE_ALL: bool = {+ match std::env::var("CANDLE_DEQUANTIZE_ALL") {+ Ok(s) => {+ !s.is_empty() && s != "0"+ },+ Err(_) => false,+ }+ }+}++thread_local! {+ static DEQUANTIZE_ALL_F16: bool = {+ match std::env::var("CANDLE_DEQUANTIZE_ALL_F16") {+ Ok(s) => {+ !s.is_empty() && s != "0"+ },+ Err(_) => false,+ }+ }+}++impl QMatMul {+ pub fn from_arc(qtensor: std::sync::Arc<QTensor>) -> Result<Self> {+ let dequantize = match qtensor.dtype() {+ GgmlDType::F32 | GgmlDType::F16 | GgmlDType::BF16 => true,+ _ => DEQUANTIZE_ALL.with(|b| *b),+ };+ let t = if dequantize {+ let tensor = qtensor.dequantize(&qtensor.device())?;+ Self::Tensor(tensor)+ } else if DEQUANTIZE_ALL_F16.with(|b| *b) {+ let tensor = qtensor.dequantize_f16(&qtensor.device())?;+ Self::TensorF16(tensor)+ } else {+ Self::QTensor(qtensor)+ };+ Ok(t)+ }++ pub fn from_qtensor(qtensor: QTensor) -> Result<Self> {+ Self::from_arc(std::sync::Arc::new(qtensor))+ }++ pub fn dequantize_f16(&self) -> Result<Tensor> {+ match self {+ Self::QTensor(t) => t.dequantize_f16(&t.device()),+ Self::Tensor(t) => t.to_dtype(DType::F16),+ Self::TensorF16(t) => Ok(t.clone()),+ }+ }++ pub fn forward_via_f16(&self, xs: &Tensor) -> Result<Tensor> {+ let w = self.dequantize_f16()?;+ let in_dtype = xs.dtype();+ let w = match *xs.dims() {+ [b1, b2, _, _] => w.broadcast_left((b1, b2))?.t()?,+ [bsize, _, _] => w.broadcast_left(bsize)?.t()?,+ _ => w.t()?,+ };+ xs.to_dtype(DType::F16)?.matmul(&w)?.to_dtype(in_dtype)+ }++ pub fn indexed_moe_forward(&self, x: &Tensor, ids: &Tensor) -> Result<Tensor> {+ match self {+ Self::QTensor(t) => t.indexed_moe_forward(x, ids),+ _ => {+ panic!("Not implemented!")+ }+ }+ }++ pub fn embedding(&self, ids: &Tensor) -> Result<Tensor> {+ match self {+ Self::QTensor(t) => t.embedding(ids),+ Self::Tensor(w) | Self::TensorF16(w) => {+ let mut final_dims = ids.dims().to_vec();+ final_dims.push(w.dim(D::Minus1)?);+ let ids = ids.to_device(w.device())?.flatten_all()?;+ w.index_select(&ids, 0)?.reshape(final_dims)+ }+ }+ }+}++impl QTensor {+ /// Fused m==1 matmul over same-dtype tensors sharing one lhs (e.g. qkv or gate+up in+ /// decode): one lhs quantization and one parallel region. None when unsupported.+ pub fn gemv_fused_shared_lhs(ts: &[&Self], lhs: &Tensor) -> Result<Option<Vec<Tensor>>> {+ #[cfg(target_arch = "aarch64")]+ {+ if ts.is_empty() || !lhs.device().is_cpu() || lhs.dtype() != crate::DType::F32 {+ return Ok(None);+ }+ if !lhs.is_contiguous() {+ return Ok(None);+ }+ let dims = lhs.dims();+ let Some((&k, batch)) = dims.split_last() else {+ return Ok(None);+ };+ if batch.iter().product::<usize>() != 1 {+ return Ok(None);+ }+ let mut ns = Vec::with_capacity(ts.len());+ let mut parts: Vec<(&dyn QuantizedType, &repack::PackedCache)> =+ Vec::with_capacity(ts.len());+ for t in ts {+ let (n, tk) = t.shape.dims2()?;+ if tk != k {+ return Ok(None);+ }+ let QStorage::Cpu(s) = &t.storage else {+ return Ok(None);+ };+ ns.push(n);+ parts.push((s.as_ref(), &t.repacked_qs));+ }+ let storage = lhs.storage();+ let crate::Storage::Cpu(cpu) = &*storage else {+ return Ok(None);+ };+ let slice = cpu.as_slice::<f32>()?;+ let offset = lhs.layout().start_offset();+ let slice = &slice[offset..offset + k];+ let mut dsts: Vec<Vec<f32>> = ns.iter().map(|&n| vec![0f32; n]).collect();+ if !repack::try_gemv_fused(k, slice, &parts, &mut dsts)? {+ return Ok(None);+ }+ drop(storage);+ let outs = dsts+ .into_iter()+ .zip(&ns)+ .map(|(d, &n)| {+ let mut shape = dims.to_vec();+ *shape.last_mut().unwrap() = n;+ Tensor::from_vec(d, shape, &crate::Device::Cpu)+ })+ .collect::<Result<Vec<_>>>()?;+ Ok(Some(outs))+ }+ #[cfg(target_arch = "x86_64")]+ {+ // the fused kernel is 512-bit only; downlevel tiers fall through to+ // per-matmul calls which dispatch to the 256-bit kernels+ if repack_x86::level() != Some(repack_x86::X86Level::Avx512Vnni) {+ return Ok(None);+ }+ if ts.is_empty() || !lhs.device().is_cpu() || lhs.dtype() != crate::DType::F32 {+ return Ok(None);+ }+ if !lhs.is_contiguous() {+ return Ok(None);+ }+ let dims = lhs.dims();+ let Some((&k, batch)) = dims.split_last() else {+ return Ok(None);+ };+ let m: usize = batch.iter().product();+ if m == 0 {+ return Ok(None);+ }+ let dtype = ts[0].dtype();+ let mut shapes = Vec::with_capacity(ts.len());+ for t in ts {+ let Ok((n, tk)) = t.shape.dims2() else {+ return Ok(None);+ };+ if tk != k+ || t.dtype() != dtype+ || !matches!(&t.storage, QStorage::Cpu(_))+ || !repack_x86::select(dtype, m, n, k)+ {+ return Ok(None);+ }+ shapes.push(n);+ }+ let guard = lhs.storage();+ let crate::Storage::Cpu(cpu) = &*guard else {+ return Ok(None);+ };+ let slice = cpu.as_slice::<f32>()?;+ let offset = lhs.layout().start_offset();+ let lhs_data = &slice[offset..offset + m * k];+ let lhs_q = repack_x86::quantize_lhs(lhs_data, m, k);+ let packs: Vec<&repack_x86::PackedX86> = ts+ .iter()+ .enumerate()+ .map(|(i, t)| {+ let n = shapes[i];+ t.repacked_qs+ .x86_get_or_init(|| repack_x86::pack(dtype, t.storage_ref(), n, k))+ })+ .collect();+ let parts: Vec<(&repack_x86::PackedX86, usize)> = packs+ .iter()+ .zip(shapes.iter())+ .map(|(p, &n)| (*p, n))+ .collect();+ let mut dsts: Vec<Vec<f32>> = shapes.iter().map(|&n| vec![0f32; m * n]).collect();+ repack_x86::gemv_fused(&parts, &lhs_q, m, k, &mut dsts);+ drop(guard);+ let mut out = Vec::with_capacity(ts.len());+ let mut out_dims = dims.to_vec();+ for (dst, &n) in dsts.into_iter().zip(shapes.iter()) {+ *out_dims.last_mut().unwrap() = n;+ out.push(Tensor::from_vec(+ dst,+ out_dims.clone(),+ &crate::Device::Cpu,+ )?);+ }+ Ok(Some(out))+ }++ #[cfg(not(any(target_arch = "aarch64", target_arch = "x86_64")))]+ {+ let _ = (ts, lhs);+ Ok(None)+ }+ }+}++impl QTensor {+ #[cfg(target_arch = "x86_64")]+ pub(crate) fn storage_ref(&self) -> &dyn QuantizedType {+ match &self.storage {+ QStorage::Cpu(s) => s.as_ref(),+ _ => unreachable!("cpu-only path"),+ }+ }++ /// Indexed (MoE) matmul over stacked expert weights [n_experts, n_out, k]: each+ /// (token, expert-id) pair gemvs against its expert's rows via the repacked cache.+ /// Returns None when the layout, dtype, or device is unsupported.+ pub fn indexed_gemv(&self, x: &Tensor, ids: &Tensor) -> Result<Option<Tensor>> {+ #[cfg(target_arch = "aarch64")]+ {+ if !x.device().is_cpu()+ || !matches!(x.dtype(), crate::DType::F32 | crate::DType::BF16)+ || !x.is_contiguous()+ {+ return Ok(None);+ }+ let Ok((n_experts, n_out, k)) = self.shape.dims3() else {+ return Ok(None);+ };+ let Ok((batch, x_t, xk)) = x.dims3() else {+ return Ok(None);+ };+ let Ok((ids_b, topk)) = ids.dims2() else {+ return Ok(None);+ };+ if xk != k || ids_b != batch || (x_t != 1 && x_t != topk) {+ return Ok(None);+ }+ let QStorage::Cpu(storage) = &self.storage else {+ return Ok(None);+ };+ let ids_v: Vec<u32> = ids+ .to_dtype(crate::DType::U32)?+ .flatten_all()?+ .to_vec1::<u32>()?;+ if ids_v.iter().any(|&e| e as usize >= n_experts) {+ crate::bail!("expert index out of range");+ }+ let guard = x.storage();+ let crate::Storage::Cpu(cpu) = &*guard else {+ return Ok(None);+ };+ let offset = x.layout().start_offset();+ let n_rows = batch * x_t;+ let widened;+ let lhs: &[f32] = if x.dtype() == crate::DType::BF16 {+ let bslice = cpu.as_slice::<half::bf16>()?;+ widened = widen_bf16(&bslice[offset..offset + n_rows * k]);+ &widened+ } else {+ let slice = cpu.as_slice::<f32>()?;+ &slice[offset..offset + n_rows * k]+ };+ let mut dst = vec![0f32; batch * topk * n_out];+ let ok = repack::try_indexed_gemv(+ storage.as_ref(),+ &self.repacked_qs,+ n_experts,+ n_out,+ k,+ lhs,+ n_rows,+ &ids_v,+ topk,+ &mut dst,+ )?;+ drop(guard);+ if !ok {+ return Ok(None);+ }+ let out = Tensor::from_vec(dst, (batch, topk, n_out), &crate::Device::Cpu)?;+ if x.dtype() == crate::DType::BF16 {+ Ok(Some(out.to_dtype(crate::DType::BF16)?))+ } else {+ Ok(Some(out))+ }+ }+ #[cfg(not(target_arch = "aarch64"))]+ {+ let _ = (x, ids);+ Ok(None)+ }+ }+}++fn widen_bf16(src: &[half::bf16]) -> Vec<f32> {+ let mut out: Vec<f32> = Vec::with_capacity(src.len());+ let out_ptr = out.as_mut_ptr() as usize;+ let n_units = src.len().div_ceil(WIDEN_CHUNK);+ crate::utils::barrier_pool().execute_chunked(n_units, |range| {+ let out_ptr = out_ptr as *mut f32;+ for unit in range {+ let lo = unit * WIDEN_CHUNK;+ let hi = src.len().min(lo + WIDEN_CHUNK);+ for (i, v) in src[lo..hi].iter().enumerate() {+ unsafe { *out_ptr.add(lo + i) = v.to_f32() };+ }+ }+ });+ // SAFETY: every element written by exactly one unit.+ unsafe { out.set_len(src.len()) };+ out+}++fn narrow_bf16(src: &[f32]) -> Vec<half::bf16> {+ let mut out: Vec<half::bf16> = Vec::with_capacity(src.len());+ let out_ptr = out.as_mut_ptr() as usize;+ let n_units = src.len().div_ceil(WIDEN_CHUNK);+ crate::utils::barrier_pool().execute_chunked(n_units, |range| {+ let out_ptr = out_ptr as *mut half::bf16;+ for unit in range {+ let lo = unit * WIDEN_CHUNK;+ let hi = src.len().min(lo + WIDEN_CHUNK);+ for (i, v) in src[lo..hi].iter().enumerate() {+ unsafe { *out_ptr.add(lo + i) = half::bf16::from_f32(*v) };+ }+ }+ });+ // SAFETY: every element written by exactly one unit.+ unsafe { out.set_len(src.len()) };+ out+}++const WIDEN_CHUNK: usize = 32 * 1024;++impl crate::CustomOp1 for QTensor {+ fn name(&self) -> &'static str {+ "qmatmul"+ }++ fn cpu_fwd(+ &self,+ storage: &crate::CpuStorage,+ layout: &crate::Layout,+ ) -> Result<(crate::CpuStorage, Shape)> {+ if !layout.is_contiguous() {+ crate::bail!("input tensor is not contiguous {layout:?}")+ }+ let src_shape = layout.shape();+ // self is transposed so n is first then k.+ let (n, k) = self.shape.dims2()?;+ if src_shape.rank() < 2 {+ crate::bail!("input tensor has only one dimension {layout:?}")+ }+ let mut dst_shape = src_shape.dims().to_vec();+ let last_k = dst_shape.pop().unwrap();+ if last_k != k {+ crate::bail!("input tensor {layout:?} incompatible with {:?}", self.shape)+ }+ dst_shape.push(n);+ let dst_shape = Shape::from(dst_shape);+ #[allow(clippy::infallible_destructuring_match)]+ let self_storage = match &self.storage {+ QStorage::Cpu(storage) => storage,+ QStorage::Metal(_) | QStorage::Cuda(_) => crate::bail!("Invalid storage"),+ };+ match storage.dtype() {+ DType::F32 => {+ let slice = storage.as_slice::<f32>()?;+ let slice =+ &slice[layout.start_offset()..layout.start_offset() + src_shape.elem_count()];+ let mut dst_storage = vec![0f32; dst_shape.elem_count()];++ let mkn = (dst_shape.elem_count() / n, k, n);+ let used_packed = repack::try_matmul_f32(+ self_storage.as_ref(),+ &self.repacked_qs,+ mkn,+ slice,+ &mut dst_storage,+ )?;+ if used_packed {+ return Ok((crate::CpuStorage::F32(dst_storage), dst_shape));+ }++ self_storage.matmul_t(mkn, slice, &mut dst_storage)?;+ Ok((crate::CpuStorage::F32(dst_storage), dst_shape))+ }+ DType::F16 => {+ let slice = storage.as_slice::<f16>()?;+ let slice =+ &slice[layout.start_offset()..layout.start_offset() + src_shape.elem_count()];+ let mut dst_storage = vec![f16::ZERO; dst_shape.elem_count()];+ self_storage.matmul_t_f16(+ (dst_shape.elem_count() / n, k, n),+ slice,+ &mut dst_storage,+ )?;+ Ok((crate::CpuStorage::F16(dst_storage), dst_shape))+ }+ DType::BF16 => {+ // widen to f32 once and take the repacked path; output stays bf16+ let slice = storage.as_slice::<half::bf16>()?;+ let slice =+ &slice[layout.start_offset()..layout.start_offset() + src_shape.elem_count()];+ let lhs = widen_bf16(slice);+ let mut dst_storage = vec![0f32; dst_shape.elem_count()];++ let mkn = (dst_shape.elem_count() / n, k, n);+ let used_packed = repack::try_matmul_f32(+ self_storage.as_ref(),+ &self.repacked_qs,+ mkn,+ &lhs,+ &mut dst_storage,+ )?;+ if !used_packed {+ self_storage.matmul_t(mkn, &lhs, &mut dst_storage)?;+ }+ let dst: Vec<half::bf16> = narrow_bf16(&dst_storage);+ Ok((crate::CpuStorage::BF16(dst), dst_shape))+ }+ _ => crate::bail!("Expected f32/f16/bf16"),+ }+ }++ fn metal_fwd(+ &self,+ storage: &crate::MetalStorage,+ layout: &crate::Layout,+ ) -> Result<(crate::MetalStorage, Shape)> {+ let self_storage = match &self.storage {+ QStorage::Metal(metal) => metal,+ _ => unreachable!("Cannot call metal matmul on non metal QTensor"),+ };+ self_storage.fwd(&self.shape, storage, layout)+ }++ fn cuda_fwd(+ &self,+ storage: &crate::CudaStorage,+ layout: &crate::Layout,+ ) -> Result<(crate::CudaStorage, Shape)> {+ let self_storage = match &self.storage {+ QStorage::Cuda(cuda) => cuda,+ _ => unreachable!("Cannot call cuda matmul on non cuda QTensor"),+ };+ self_storage.fwd(&self.shape, storage, layout)+ }+}++impl crate::Module for QMatMul {+ fn forward(&self, xs: &Tensor) -> Result<Tensor> {+ match self {+ Self::QTensor(t) => xs.apply_op1_no_bwd(t.as_ref()),+ Self::Tensor(w) => {+ let w = match *xs.dims() {+ [b1, b2, _, _] => w.broadcast_left((b1, b2))?.t()?,+ [bsize, _, _] => w.broadcast_left(bsize)?.t()?,+ _ => w.t()?,+ };+ xs.matmul(&w)+ }+ Self::TensorF16(w) => {+ let in_dtype = xs.dtype();+ let w = match *xs.dims() {+ [b1, b2, _, _] => w.broadcast_left((b1, b2))?.t()?,+ [bsize, _, _] => w.broadcast_left(bsize)?.t()?,+ _ => w.t()?,+ };+ xs.to_dtype(DType::F16)?.matmul(&w)?.to_dtype(in_dtype)+ }+ }+ }+}diff --git a/vendor/candle-core/src/quantized/neon.rs b/vendor/candle-core/src/quantized/neon.rsnew file mode 100644index 0000000..d7fe760--- /dev/null+++ b/vendor/candle-core/src/quantized/neon.rs@@ -0,0 +1,4234 @@+#![allow(clippy::needless_range_loop)]++use super::k_quants::{+ BlockQ2K, BlockQ3K, BlockQ4K, BlockQ4_0, BlockQ5K, BlockQ6K, BlockQ8K, BlockQ8_0, QK8_0, QK_K,+};+#[cfg(target_arch = "aarch64")]+use super::repack::BlockQ8_0x4;+#[cfg(target_arch = "aarch64")]+use super::repack::{BlockQ4Kx8, BlockQ4_0x4};+#[cfg(target_arch = "aarch64")]+use super::repack::{BlockQ5Kx8, BlockQ6Kx8, BlockQ8Kx4};+use byteorder::{ByteOrder, LittleEndian};+use half::f16;++#[allow(unused_imports)]+#[cfg(target_arch = "arm")]+use core::arch::arm::*;++#[allow(unused_imports)]+#[cfg(target_arch = "aarch64")]+use core::arch::aarch64::*;++#[inline]+#[target_feature(enable = "dotprod")]+unsafe fn vdotq_s32(a: int8x16_t, b: int8x16_t) -> int32x4_t {+ let mut acc: int32x4_t = vdupq_n_s32(0);+ core::arch::asm!(+ "sdot {acc:v}.4s, {a:v}.16b, {b:v}.16b",+ acc = inout(vreg) acc,+ a = in(vreg) a,+ b = in(vreg) b,+ options(nostack, nomem),+ );+ acc+}++/// Two SDOT ops into one accumulator+#[inline]+#[target_feature(enable = "dotprod")]+unsafe fn vdotq_s32_pair(a0: int8x16_t, b0: int8x16_t, a1: int8x16_t, b1: int8x16_t) -> int32x4_t {+ let mut acc: int32x4_t = vdupq_n_s32(0);+ core::arch::asm!(+ "sdot {acc:v}.4s, {a0:v}.16b, {b0:v}.16b",+ "sdot {acc:v}.4s, {a1:v}.16b, {b1:v}.16b",+ acc = inout(vreg) acc,+ a0 = in(vreg) a0,+ b0 = in(vreg) b0,+ a1 = in(vreg) a1,+ b1 = in(vreg) b1,+ options(nostack, nomem),+ );+ acc+}++/// Accumulating SDOT: acc += dot4(a, b) for each lane.+#[cfg(target_arch = "aarch64")]+#[inline]+#[target_feature(enable = "dotprod")]+unsafe fn sdot_acc(acc: int32x4_t, a: int8x16_t, b: int8x16_t) -> int32x4_t {+ let mut out = acc;+ core::arch::asm!(+ "sdot {out:v}.4s, {a:v}.16b, {b:v}.16b",+ out = inout(vreg) out,+ a = in(vreg) a,+ b = in(vreg) b,+ options(nostack, nomem),+ );+ out+}++#[cfg(target_arch = "aarch64")]+#[inline(always)]+unsafe fn sdot_laneq_s32<const LANE: i32>(acc: int32x4_t, a: int8x16_t, b: int8x16_t) -> int32x4_t {+ sdot_acc(+ acc,+ a,+ vreinterpretq_s8_s32(vdupq_laneq_s32::<LANE>(vreinterpretq_s32_s8(b))),+ )+}++#[cfg(target_arch = "aarch64")]+#[inline]+#[target_feature(enable = "i8mm")]+unsafe fn smmla_s32(acc: int32x4_t, a: int8x16_t, b: int8x16_t) -> int32x4_t {+ let mut out = acc;+ core::arch::asm!(+ "smmla {out:v}.4s, {a:v}.16b, {b:v}.16b",+ out = inout(vreg) out,+ a = in(vreg) a,+ b = in(vreg) b,+ options(nostack, nomem),+ );+ out+}++#[cfg(target_arch = "aarch64")]+#[inline(always)]+unsafe fn decode_q4kx8_scales(scales_in: *const u8) -> (int16x8_t, int16x8_t) {+ const KMASK1: u32 = 0x3f3f3f3f;+ const KMASK2: u32 = 0x0f0f0f0f;+ const KMASK3: u32 = 0x03030303;+ // Direct unaligned reads avoid the copy_nonoverlapping stack round-trip.+ let sm0 = (scales_in as *const u32).read_unaligned();+ let sm1 = (scales_in.add(4) as *const u32).read_unaligned();+ let sm2 = (scales_in.add(8) as *const u32).read_unaligned();+ let mins_0_3 = sm1 & KMASK1;+ let mins_4_7 = ((sm2 >> 4) & KMASK2) | (((sm1 >> 6) & KMASK3) << 4);+ let out_mins = vreinterpretq_s16_u16(vmovl_u8(vreinterpret_u8_u32(vcreate_u32(+ (mins_0_3 as u64) | ((mins_4_7 as u64) << 32),+ ))));+ let sc_0 = sm0 & KMASK1;+ let sc_1 = (sm2 & KMASK2) | (((sm0 >> 6) & KMASK3) << 4);+ let out_scales = vmovl_s8(vreinterpret_s8_u8(vreinterpret_u8_u32(vcreate_u32(+ (sc_0 as u64) | ((sc_1 as u64) << 32),+ ))));+ (out_mins, out_scales)+}++#[inline(always)]+unsafe fn merge_signed_max(+ abs_a: float32x4_t,+ smax_a: float32x4_t,+ abs_b: float32x4_t,+ smax_b: float32x4_t,+) -> (float32x4_t, float32x4_t) {+ (+ vmaxq_f32(abs_a, abs_b),+ vbslq_f32(vcgtq_f32(abs_b, abs_a), smax_b, smax_a),+ )+}++#[inline(always)]+pub(crate) fn vec_dot_q4_0_q8_0(n: usize, xs: &[BlockQ4_0], ys: &[BlockQ8_0]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK8_0),+ "vec_dot_q4_0_q8_0: {n} is not divisible by {QK8_0}"+ );+ let nb = n / QK8_0;+ unsafe {+ let mut sumv0 = vdupq_n_f32(0.0f32);+ for i in 0..nb {+ let x0 = &xs[i];+ let y0 = &ys[i];++ let m4b = vdupq_n_u8(0x0F);+ let s8b = vdupq_n_s8(0x8);++ let v0_0 = vld1q_u8(x0.qs.as_ptr());++ // 4-bit -> 8-bit+ let v0_0l = vreinterpretq_s8_u8(vandq_u8(v0_0, m4b));+ let v0_0h = vreinterpretq_s8_u8(vshrq_n_u8(v0_0, 4));++ // sub 8+ let v0_0ls = vsubq_s8(v0_0l, s8b);+ let v0_0hs = vsubq_s8(v0_0h, s8b);++ // load y+ let v1_0l = vld1q_s8(y0.qs.as_ptr());+ let v1_0h = vld1q_s8(y0.qs.as_ptr().add(16));++ let pl0 = vdotq_s32(v0_0ls, v1_0l);+ let ph0 = vdotq_s32(v0_0hs, v1_0h);+ sumv0 = vmlaq_n_f32(+ sumv0,+ vcvtq_f32_s32(vaddq_s32(pl0, ph0)),+ x0.d.to_f32() * y0.d.to_f32(),+ );+ }+ vaddvq_f32(sumv0)+ }+}++#[inline(always)]+pub(crate) fn vec_dot_q8_0_q8_0(n: usize, xs: &[BlockQ8_0], ys: &[BlockQ8_0]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK8_0),+ "vec_dot_q8_0_q8_0: {n} is not divisible by {QK8_0}"+ );+ let nb = n / QK8_0;+ unsafe {+ let mut sumv0 = vdupq_n_f32(0.0f32);+ for i in 0..nb {+ let x0 = &xs[i];+ let y0 = &ys[i];++ let x0_0 = vld1q_s8(x0.qs.as_ptr());+ let x0_1 = vld1q_s8(x0.qs.as_ptr().add(16));++ // load y+ let y0_0 = vld1q_s8(y0.qs.as_ptr());+ let y0_1 = vld1q_s8(y0.qs.as_ptr().add(16));++ let p0 = vdotq_s32(x0_0, y0_0);+ let p1 = vdotq_s32(x0_1, y0_1);++ sumv0 = vmlaq_n_f32(+ sumv0,+ vcvtq_f32_s32(vaddq_s32(p0, p1)),+ x0.d.to_f32() * y0.d.to_f32(),+ );+ }+ vaddvq_f32(sumv0)+ }+}++#[cfg(target_arch = "aarch64")]+pub(crate) fn quantize_q8_0(xs: &[f32], ys: &mut [BlockQ8_0]) {+ debug_assert!(xs.len().is_multiple_of(QK8_0));+ debug_assert_eq!(ys.len(), xs.len() / QK8_0);++ let (x_chunks, _) = xs.as_chunks::<QK8_0>();+ for (chunk, y) in x_chunks.iter().zip(ys.iter_mut()) {+ unsafe {+ let p = chunk.as_ptr();+ let v0 = vld1q_f32(p);+ let v1 = vld1q_f32(p.add(4));+ let v2 = vld1q_f32(p.add(8));+ let v3 = vld1q_f32(p.add(12));+ let v4 = vld1q_f32(p.add(16));+ let v5 = vld1q_f32(p.add(20));+ let v6 = vld1q_f32(p.add(24));+ let v7 = vld1q_f32(p.add(28));++ let a01 = vmaxq_f32(vabsq_f32(v0), vabsq_f32(v1));+ let a23 = vmaxq_f32(vabsq_f32(v2), vabsq_f32(v3));+ let a45 = vmaxq_f32(vabsq_f32(v4), vabsq_f32(v5));+ let a67 = vmaxq_f32(vabsq_f32(v6), vabsq_f32(v7));+ let amax = vmaxvq_f32(vmaxq_f32(vmaxq_f32(a01, a23), vmaxq_f32(a45, a67)));+ let d = amax / ((1 << 7) - 1) as f32;+ let id = if d != 0.0 { 1.0 / d } else { 0.0 };+ y.d = f16::from_f32(d);+ let id = vdupq_n_f32(id);++ let s01 = vcombine_s16(+ vqmovn_s32(vcvtaq_s32_f32(vmulq_f32(v0, id))),+ vqmovn_s32(vcvtaq_s32_f32(vmulq_f32(v1, id))),+ );+ let s23 = vcombine_s16(+ vqmovn_s32(vcvtaq_s32_f32(vmulq_f32(v2, id))),+ vqmovn_s32(vcvtaq_s32_f32(vmulq_f32(v3, id))),+ );+ let q0 = vcombine_s8(vqmovn_s16(s01), vqmovn_s16(s23));+ vst1q_s8(y.qs.as_mut_ptr(), q0);++ let s45 = vcombine_s16(+ vqmovn_s32(vcvtaq_s32_f32(vmulq_f32(v4, id))),+ vqmovn_s32(vcvtaq_s32_f32(vmulq_f32(v5, id))),+ );+ let s67 = vcombine_s16(+ vqmovn_s32(vcvtaq_s32_f32(vmulq_f32(v6, id))),+ vqmovn_s32(vcvtaq_s32_f32(vmulq_f32(v7, id))),+ );+ let q1 = vcombine_s8(vqmovn_s16(s45), vqmovn_s16(s67));+ vst1q_s8(y.qs.as_mut_ptr().add(16), q1);+ }+ }+}++#[inline(always)]+pub(crate) fn vec_dot_4_q8_0_q8_0(+ n: usize,+ xs0: &[BlockQ8_0],+ xs1: &[BlockQ8_0],+ xs2: &[BlockQ8_0],+ xs3: &[BlockQ8_0],+ ys: &[BlockQ8_0],+) -> (f32, f32, f32, f32) {+ debug_assert!(+ n.is_multiple_of(QK8_0),+ "vec_dot_4_q8_0_q8_0: {n} is not divisible by {QK8_0}"+ );+ let nb = n / QK8_0;+ unsafe {+ let mut sum0 = vdupq_n_f32(0.0f32);+ let mut sum1 = vdupq_n_f32(0.0f32);+ let mut sum2 = vdupq_n_f32(0.0f32);+ let mut sum3 = vdupq_n_f32(0.0f32);+ for i in 0..nb {+ let y = &ys[i];+ let y0 = vld1q_s8(y.qs.as_ptr());+ let y1 = vld1q_s8(y.qs.as_ptr().add(16));+ let yd = y.d.to_f32();+ let x0 = &xs0[i];+ let x1 = &xs1[i];+ let x2 = &xs2[i];+ let x3 = &xs3[i];+ // Each column uses 2 SDOTs fused into one accumulator via vdotq_s32_pair.+ // All 4 columns share y0/y1 and have independent accumulators.+ let p0 = vdotq_s32_pair(+ vld1q_s8(x0.qs.as_ptr()),+ y0,+ vld1q_s8(x0.qs.as_ptr().add(16)),+ y1,+ );+ let p1 = vdotq_s32_pair(+ vld1q_s8(x1.qs.as_ptr()),+ y0,+ vld1q_s8(x1.qs.as_ptr().add(16)),+ y1,+ );+ let p2 = vdotq_s32_pair(+ vld1q_s8(x2.qs.as_ptr()),+ y0,+ vld1q_s8(x2.qs.as_ptr().add(16)),+ y1,+ );+ let p3 = vdotq_s32_pair(+ vld1q_s8(x3.qs.as_ptr()),+ y0,+ vld1q_s8(x3.qs.as_ptr().add(16)),+ y1,+ );+ sum0 = vmlaq_n_f32(sum0, vcvtq_f32_s32(p0), x0.d.to_f32() * yd);+ sum1 = vmlaq_n_f32(sum1, vcvtq_f32_s32(p1), x1.d.to_f32() * yd);+ sum2 = vmlaq_n_f32(sum2, vcvtq_f32_s32(p2), x2.d.to_f32() * yd);+ sum3 = vmlaq_n_f32(sum3, vcvtq_f32_s32(p3), x3.d.to_f32() * yd);+ }+ (+ vaddvq_f32(sum0),+ vaddvq_f32(sum1),+ vaddvq_f32(sum2),+ vaddvq_f32(sum3),+ )+ }+}++#[cfg(target_arch = "aarch64")]+fn quantize_q8_0x4_interleaved<const BLOCK_LEN: usize>(+ lhs: &[f32],+ k: usize,+ row: usize,+ out: &mut [BlockQ8_0x4],+) {+ let k_in_blocks = k / QK8_0;+ for b in 0..k_in_blocks {+ let mut p = BlockQ8_0x4 {+ d: [f16::ZERO; 4],+ qs: [0; QK8_0 * 4],+ };+ let mut ids = [0f32; 4];+ for r in 0..4 {+ let xs = &lhs[(row + r) * k + b * QK8_0..(row + r) * k + (b + 1) * QK8_0];+ let mut amax = 0f32;+ for &x in xs {+ amax = amax.max(x.abs());+ }+ let d = amax / ((1 << 7) - 1) as f32;+ ids[r] = if d != 0f32 { 1. / d } else { 0. };+ p.d[r] = f16::from_f32(d);+ }+ for r in 0..4 {+ let xs = &lhs[(row + r) * k + b * QK8_0..(row + r) * k + (b + 1) * QK8_0];+ for (j, &x) in xs.iter().enumerate() {+ p.qs[(j / BLOCK_LEN) * BLOCK_LEN * 4 + r * BLOCK_LEN + j % BLOCK_LEN] =+ (x * ids[r]).round() as i8;+ }+ }+ out[b] = p;+ }+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "dotprod")]+unsafe fn vec_dot_4_q4_0x4_q8_0_4x4(n: usize, xs: &[BlockQ4_0x4], ys: &[BlockQ8_0]) -> [f32; 4] {+ debug_assert!(n.is_multiple_of(QK8_0));+ let mut out = [0f32; 4];+ let high_mask = vdupq_n_u8(0xf0);+ let mut acc = vdupq_n_f32(0.);++ for (x, y) in xs.iter().zip(ys.iter()) {+ let bd = load_f16x4(x.d.as_ptr());+ let ad = vdupq_n_f32(y.d.to_f32());+ let a0 = vld1q_s8(y.qs.as_ptr());+ let a1 = vld1q_s8(y.qs.as_ptr().add(16));+ let b0 = vld1q_s8(x.qs.as_ptr());+ let b1 = vld1q_s8(x.qs.as_ptr().add(16));+ let b2 = vld1q_s8(x.qs.as_ptr().add(32));+ let b3 = vld1q_s8(x.qs.as_ptr().add(48));+ let mut ret = vdupq_n_s32(0);++ macro_rules! dot_lane {+ ($lane:literal, $b:expr, $a:expr) => {{+ ret = sdot_laneq_s32::<$lane>(ret, $b, $a);+ }};+ }++ dot_lane!(+ 0,+ vreinterpretq_s8_u8(vshlq_n_u8(vreinterpretq_u8_s8(b0), 4)),+ a0+ );+ dot_lane!(+ 1,+ vreinterpretq_s8_u8(vshlq_n_u8(vreinterpretq_u8_s8(b1), 4)),+ a0+ );+ dot_lane!(+ 2,+ vreinterpretq_s8_u8(vshlq_n_u8(vreinterpretq_u8_s8(b2), 4)),+ a0+ );+ dot_lane!(+ 3,+ vreinterpretq_s8_u8(vshlq_n_u8(vreinterpretq_u8_s8(b3), 4)),+ a0+ );+ dot_lane!(+ 0,+ vreinterpretq_s8_u8(vandq_u8(vreinterpretq_u8_s8(b0), high_mask)),+ a1+ );+ dot_lane!(+ 1,+ vreinterpretq_s8_u8(vandq_u8(vreinterpretq_u8_s8(b1), high_mask)),+ a1+ );+ dot_lane!(+ 2,+ vreinterpretq_s8_u8(vandq_u8(vreinterpretq_u8_s8(b2), high_mask)),+ a1+ );+ dot_lane!(+ 3,+ vreinterpretq_s8_u8(vandq_u8(vreinterpretq_u8_s8(b3), high_mask)),+ a1+ );++ acc = vmlaq_f32(+ acc,+ vmulq_n_f32(vcvtq_f32_s32(ret), 1. / 16.),+ vmulq_f32(ad, bd),+ );+ }++ vst1q_f32(out.as_mut_ptr(), acc);+ out+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "dotprod")]+unsafe fn vec_dot_4_q4_0x4_q8_0_4x8(n: usize, xs: &[BlockQ4_0x4], ys: &[BlockQ8_0]) -> [f32; 4] {+ debug_assert!(n.is_multiple_of(QK8_0));+ let mut out = [0f32; 4];+ let high_mask = vdupq_n_u8(0xf0);+ let mut acc = vdupq_n_f32(0.);++ for (x, y) in xs.iter().zip(ys.iter()) {+ let bd = load_f16x4(x.d.as_ptr());+ let ad = vdupq_n_f32(y.d.to_f32());+ let a0 = vreinterpretq_s8_s64(vld1q_dup_s64(y.qs.as_ptr() as *const i64));+ let a1 = vreinterpretq_s8_s64(vld1q_dup_s64(y.qs.as_ptr().add(8) as *const i64));+ let a2 = vreinterpretq_s8_s64(vld1q_dup_s64(y.qs.as_ptr().add(16) as *const i64));+ let a3 = vreinterpretq_s8_s64(vld1q_dup_s64(y.qs.as_ptr().add(24) as *const i64));+ let b0 = vld1q_s8(x.qs.as_ptr());+ let b1 = vld1q_s8(x.qs.as_ptr().add(16));+ let b2 = vld1q_s8(x.qs.as_ptr().add(32));+ let b3 = vld1q_s8(x.qs.as_ptr().add(48));+ let mut ret0 = vdupq_n_s32(0);+ let mut ret1 = vdupq_n_s32(0);++ ret0 = sdot_acc(+ ret0,+ vreinterpretq_s8_u8(vshlq_n_u8(vreinterpretq_u8_s8(b0), 4)),+ a0,+ );+ ret1 = sdot_acc(+ ret1,+ vreinterpretq_s8_u8(vshlq_n_u8(vreinterpretq_u8_s8(b1), 4)),+ a0,+ );+ ret0 = sdot_acc(+ ret0,+ vreinterpretq_s8_u8(vshlq_n_u8(vreinterpretq_u8_s8(b2), 4)),+ a1,+ );+ ret1 = sdot_acc(+ ret1,+ vreinterpretq_s8_u8(vshlq_n_u8(vreinterpretq_u8_s8(b3), 4)),+ a1,+ );+ ret0 = sdot_acc(+ ret0,+ vreinterpretq_s8_u8(vandq_u8(vreinterpretq_u8_s8(b0), high_mask)),+ a2,+ );+ ret1 = sdot_acc(+ ret1,+ vreinterpretq_s8_u8(vandq_u8(vreinterpretq_u8_s8(b1), high_mask)),+ a2,+ );+ ret0 = sdot_acc(+ ret0,+ vreinterpretq_s8_u8(vandq_u8(vreinterpretq_u8_s8(b2), high_mask)),+ a3,+ );+ ret1 = sdot_acc(+ ret1,+ vreinterpretq_s8_u8(vandq_u8(vreinterpretq_u8_s8(b3), high_mask)),+ a3,+ );++ let ret = vpaddq_s32(ret0, ret1);+ acc = vmlaq_f32(+ acc,+ vmulq_n_f32(vcvtq_f32_s32(ret), 1. / 16.),+ vmulq_f32(ad, bd),+ );+ }++ vst1q_f32(out.as_mut_ptr(), acc);+ out+}++#[cfg(target_arch = "aarch64")]+#[inline(always)]+fn vec_dot_4_q4_0x4_q8_0(n: usize, xs: &[BlockQ4_0x4], ys: &[BlockQ8_0]) -> [f32; 4] {+ unsafe {+ if crate::cpu::features::get().i8mm {+ vec_dot_4_q4_0x4_q8_0_4x8(n, xs, ys)+ } else {+ vec_dot_4_q4_0x4_q8_0_4x4(n, xs, ys)+ }+ }+}++#[cfg(target_arch = "aarch64")]+fn matmul_q4_0_x4_gemv(+ (_m, k, n): (usize, usize, usize),+ lhs: &[f32],+ repacked: &[BlockQ4_0x4],+ dst: &mut [f32],+) -> crate::Result<()> {+ debug_assert!(k.is_multiple_of(QK8_0));+ debug_assert!(n.is_multiple_of(4));+ let k_in_blocks = k / QK8_0;+ let n_groups = n / 4;++ thread_local! {+ static LHS_Q4_0_GEMV_SCRATCH: std::cell::RefCell<Vec<u64>> =+ const { std::cell::RefCell::new(Vec::new()) };+ }++ let elem_size = std::mem::size_of::<BlockQ8_0>();+ let required_len = (k_in_blocks * elem_size).div_ceil(8);+ LHS_Q4_0_GEMV_SCRATCH.with(|cell| -> crate::Result<()> {+ let mut scratch = cell.borrow_mut();+ if scratch.len() < required_len {+ scratch.resize(required_len, 0);+ }+ let lhs_b: &mut [BlockQ8_0] = unsafe {+ std::slice::from_raw_parts_mut(scratch.as_mut_ptr() as *mut BlockQ8_0, k_in_blocks)+ };+ <BlockQ8_0 as super::GgmlType>::from_float(lhs, lhs_b);++ let pool = crate::utils::barrier_pool();+ let lhs_ptr = lhs_b.as_ptr() as usize;+ let repacked_ptr = repacked.as_ptr() as usize;+ let dst_ptr = dst.as_mut_ptr() as usize;+ let x4_block_bytes = std::mem::size_of::<BlockQ4_0x4>();++ // equal-sized gemv units: a static split skips the shared cursor, which costs+ // several us per small decode matmul+ pool.execute_static(n_groups, |range| {+ let lhs_row =+ unsafe { std::slice::from_raw_parts(lhs_ptr as *const BlockQ8_0, k_in_blocks) };+ let dst_ptr = dst_ptr as *mut f32;+ for g in range {+ let xs = unsafe {+ std::slice::from_raw_parts(+ (repacked_ptr + g * k_in_blocks * x4_block_bytes) as *const BlockQ4_0x4,+ k_in_blocks,+ )+ };+ let results = vec_dot_4_q4_0x4_q8_0(k, xs, lhs_row);+ unsafe {+ std::ptr::copy_nonoverlapping(results.as_ptr(), dst_ptr.add(g * 4), 4);+ }+ }+ });++ Ok(())+ })+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "dotprod")]+unsafe fn store_q4_0x4_4x4(+ dst: *mut f32,+ n: usize,+ row: usize,+ col: usize,+ lhs: &[BlockQ8_0x4],+ rhs: &[BlockQ4_0x4],+ k_in_blocks: usize,+) {+ let mut sum0 = vdupq_n_f32(0.);+ let mut sum1 = vdupq_n_f32(0.);+ let mut sum2 = vdupq_n_f32(0.);+ let mut sum3 = vdupq_n_f32(0.);+ let high_mask = vdupq_n_u8(0xf0);++ for i in 0..k_in_blocks {+ let x = &rhs[i];+ let y = &lhs[i];+ let bd = [+ x.d[0].to_f32(),+ x.d[1].to_f32(),+ x.d[2].to_f32(),+ x.d[3].to_f32(),+ ];+ let bd = vld1q_f32(bd.as_ptr());+ let ad = [+ y.d[0].to_f32(),+ y.d[1].to_f32(),+ y.d[2].to_f32(),+ y.d[3].to_f32(),+ ];++ let mut ret0 = vdupq_n_s32(0);+ let mut ret1 = vdupq_n_s32(0);+ let mut ret2 = vdupq_n_s32(0);+ let mut ret3 = vdupq_n_s32(0);++ macro_rules! dot_rows {+ ($xv:expr, $yv:expr) => {{+ ret0 = sdot_laneq_s32::<0>(ret0, $xv, $yv);+ ret1 = sdot_laneq_s32::<1>(ret1, $xv, $yv);+ ret2 = sdot_laneq_s32::<2>(ret2, $xv, $yv);+ ret3 = sdot_laneq_s32::<3>(ret3, $xv, $yv);+ }};+ }++ for j in 0..QK8_0 / 8 {+ let q4 = vld1q_s8(x.qs.as_ptr().add(j * 16));+ let q4lo = vreinterpretq_s8_u8(vshlq_n_u8(vreinterpretq_u8_s8(q4), 4));+ let q4hi = vreinterpretq_s8_u8(vandq_u8(vreinterpretq_u8_s8(q4), high_mask));+ dot_rows!(q4lo, vld1q_s8(y.qs.as_ptr().add(j * 16)));+ dot_rows!(q4hi, vld1q_s8(y.qs.as_ptr().add((j + 4) * 16)));+ }++ sum0 = vmlaq_f32(+ sum0,+ vmulq_n_f32(vcvtq_f32_s32(ret0), 1. / 16.),+ vmulq_n_f32(bd, ad[0]),+ );+ sum1 = vmlaq_f32(+ sum1,+ vmulq_n_f32(vcvtq_f32_s32(ret1), 1. / 16.),+ vmulq_n_f32(bd, ad[1]),+ );+ sum2 = vmlaq_f32(+ sum2,+ vmulq_n_f32(vcvtq_f32_s32(ret2), 1. / 16.),+ vmulq_n_f32(bd, ad[2]),+ );+ sum3 = vmlaq_f32(+ sum3,+ vmulq_n_f32(vcvtq_f32_s32(ret3), 1. / 16.),+ vmulq_n_f32(bd, ad[3]),+ );+ }++ vst1q_f32(dst.add(row * n + col), sum0);+ vst1q_f32(dst.add((row + 1) * n + col), sum1);+ vst1q_f32(dst.add((row + 2) * n + col), sum2);+ vst1q_f32(dst.add((row + 3) * n + col), sum3);+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "dotprod")]+pub(crate) fn matmul_q4_0_x4(+ (m, k, n): (usize, usize, usize),+ lhs: &[f32],+ repacked: &[BlockQ4_0x4],+ dst: &mut [f32],+) -> crate::Result<()> {+ debug_assert!(k.is_multiple_of(QK8_0));+ debug_assert!(n.is_multiple_of(4));+ if m == 1 {+ return matmul_q4_0_x4_gemv((m, k, n), lhs, repacked, dst);+ }+ debug_assert!(m.is_multiple_of(4));+ let k_in_blocks = k / QK8_0;+ let n_groups = n / 4;+ let row_groups = m / 4;++ thread_local! {+ static LHS_Q4_0_X4_SCRATCH: std::cell::RefCell<Vec<u64>> =+ const { std::cell::RefCell::new(Vec::new()) };+ }++ let elem_size = std::mem::size_of::<BlockQ8_0x4>();+ let required_len = (row_groups * k_in_blocks * elem_size).div_ceil(8);+ LHS_Q4_0_X4_SCRATCH.with(|cell| -> crate::Result<()> {+ let mut scratch = cell.borrow_mut();+ if scratch.len() < required_len {+ scratch.resize(required_len, 0);+ }+ let lhs_x4: &mut [BlockQ8_0x4] = unsafe {+ std::slice::from_raw_parts_mut(+ scratch.as_mut_ptr() as *mut BlockQ8_0x4,+ row_groups * k_in_blocks,+ )+ };+ for group in 0..row_groups {+ quantize_q8_0x4_interleaved::<4>(+ lhs,+ k,+ group * 4,+ &mut lhs_x4[group * k_in_blocks..(group + 1) * k_in_blocks],+ );+ }++ let tiles_total = row_groups * n_groups;+ let pool = crate::utils::barrier_pool();+ let lhs_ptr = lhs_x4.as_ptr() as usize;+ let repacked_ptr = repacked.as_ptr() as usize;+ let dst_ptr = dst.as_mut_ptr() as usize;++ pool.execute_chunked(tiles_total, |range| {+ let lhs_ptr = lhs_ptr as *const BlockQ8_0x4;+ let repacked_ptr = repacked_ptr as *const BlockQ4_0x4;+ let dst_ptr = dst_ptr as *mut f32;+ for tile in range {+ let row_group = tile / n_groups;+ let col_group = tile - row_group * n_groups;+ let row = row_group * 4;+ let col = col_group * 4;+ let lhs_tile = unsafe {+ std::slice::from_raw_parts(lhs_ptr.add(row_group * k_in_blocks), k_in_blocks)+ };+ let rhs_tile = unsafe {+ std::slice::from_raw_parts(+ repacked_ptr.add(col_group * k_in_blocks),+ k_in_blocks,+ )+ };+ unsafe { store_q4_0x4_4x4(dst_ptr, n, row, col, lhs_tile, rhs_tile, k_in_blocks) };+ }+ });++ Ok(())+ })+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "i8mm")]+unsafe fn store_q4_0x4_4x4_i8mm(+ dst: *mut f32,+ n: usize,+ row: usize,+ col: usize,+ lhs: &[BlockQ8_0x4],+ rhs: &[BlockQ4_0x4],+ k_in_blocks: usize,+) {+ let high_mask = vdupq_n_u8(0xf0);+ let mut sum0 = vdupq_n_f32(0.);+ let mut sum1 = vdupq_n_f32(0.);+ let mut sum2 = vdupq_n_f32(0.);+ let mut sum3 = vdupq_n_f32(0.);++ for i in 0..k_in_blocks {+ let x = &rhs[i];+ let y = &lhs[i];+ let bd = [+ x.d[0].to_f32(),+ x.d[1].to_f32(),+ x.d[2].to_f32(),+ x.d[3].to_f32(),+ ];+ let bd = vld1q_f32(bd.as_ptr());+ let ad = [+ y.d[0].to_f32(),+ y.d[1].to_f32(),+ y.d[2].to_f32(),+ y.d[3].to_f32(),+ ];++ let mut acc00 = vdupq_n_s32(0);+ let mut acc01 = vdupq_n_s32(0);+ let mut acc10 = vdupq_n_s32(0);+ let mut acc11 = vdupq_n_s32(0);++ for chunk in 0..2 {+ let q4_offset = chunk * 32;+ let q8_lo_offset = chunk * 32;+ let q8_hi_offset = (chunk + 2) * 32;++ let q4_01 = vld1q_s8(x.qs.as_ptr().add(q4_offset));+ let q4_23 = vld1q_s8(x.qs.as_ptr().add(q4_offset + 16));+ let q4lo_01 = vreinterpretq_s8_u8(vshlq_n_u8(vreinterpretq_u8_s8(q4_01), 4));+ let q4lo_23 = vreinterpretq_s8_u8(vshlq_n_u8(vreinterpretq_u8_s8(q4_23), 4));+ let q4hi_01 = vreinterpretq_s8_u8(vandq_u8(vreinterpretq_u8_s8(q4_01), high_mask));+ let q4hi_23 = vreinterpretq_s8_u8(vandq_u8(vreinterpretq_u8_s8(q4_23), high_mask));++ let a01 = vld1q_s8(y.qs.as_ptr().add(q8_lo_offset));+ let a23 = vld1q_s8(y.qs.as_ptr().add(q8_lo_offset + 16));+ acc00 = smmla_s32(acc00, a01, q4lo_01);+ acc01 = smmla_s32(acc01, a01, q4lo_23);+ acc10 = smmla_s32(acc10, a23, q4lo_01);+ acc11 = smmla_s32(acc11, a23, q4lo_23);++ let a01 = vld1q_s8(y.qs.as_ptr().add(q8_hi_offset));+ let a23 = vld1q_s8(y.qs.as_ptr().add(q8_hi_offset + 16));+ acc00 = smmla_s32(acc00, a01, q4hi_01);+ acc01 = smmla_s32(acc01, a01, q4hi_23);+ acc10 = smmla_s32(acc10, a23, q4hi_01);+ acc11 = smmla_s32(acc11, a23, q4hi_23);+ }++ let row0 = vcombine_s32(vget_low_s32(acc00), vget_low_s32(acc01));+ let row1 = vcombine_s32(vget_high_s32(acc00), vget_high_s32(acc01));+ let row2 = vcombine_s32(vget_low_s32(acc10), vget_low_s32(acc11));+ let row3 = vcombine_s32(vget_high_s32(acc10), vget_high_s32(acc11));++ sum0 = vmlaq_f32(+ sum0,+ vmulq_n_f32(vcvtq_f32_s32(row0), 1. / 16.),+ vmulq_n_f32(bd, ad[0]),+ );+ sum1 = vmlaq_f32(+ sum1,+ vmulq_n_f32(vcvtq_f32_s32(row1), 1. / 16.),+ vmulq_n_f32(bd, ad[1]),+ );+ sum2 = vmlaq_f32(+ sum2,+ vmulq_n_f32(vcvtq_f32_s32(row2), 1. / 16.),+ vmulq_n_f32(bd, ad[2]),+ );+ sum3 = vmlaq_f32(+ sum3,+ vmulq_n_f32(vcvtq_f32_s32(row3), 1. / 16.),+ vmulq_n_f32(bd, ad[3]),+ );+ }++ vst1q_f32(dst.add(row * n + col), sum0);+ vst1q_f32(dst.add((row + 1) * n + col), sum1);+ vst1q_f32(dst.add((row + 2) * n + col), sum2);+ vst1q_f32(dst.add((row + 3) * n + col), sum3);+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "i8mm")]+pub(crate) fn matmul_q4_0_x4_i8mm(+ (m, k, n): (usize, usize, usize),+ lhs: &[f32],+ repacked: &[BlockQ4_0x4],+ dst: &mut [f32],+) -> crate::Result<()> {+ debug_assert!(k.is_multiple_of(QK8_0));+ debug_assert!(n.is_multiple_of(4));+ debug_assert!(m.is_multiple_of(4));+ let k_in_blocks = k / QK8_0;+ let n_groups = n / 4;+ let row_groups = m / 4;++ thread_local! {+ static LHS_Q4_0_X4_I8MM_SCRATCH: std::cell::RefCell<Vec<u64>> =+ const { std::cell::RefCell::new(Vec::new()) };+ }++ let elem_size = std::mem::size_of::<BlockQ8_0x4>();+ let required_len = (row_groups * k_in_blocks * elem_size).div_ceil(8);+ LHS_Q4_0_X4_I8MM_SCRATCH.with(|cell| -> crate::Result<()> {+ let mut scratch = cell.borrow_mut();+ if scratch.len() < required_len {+ scratch.resize(required_len, 0);+ }+ let lhs_x4: &mut [BlockQ8_0x4] = unsafe {+ std::slice::from_raw_parts_mut(+ scratch.as_mut_ptr() as *mut BlockQ8_0x4,+ row_groups * k_in_blocks,+ )+ };+ for group in 0..row_groups {+ quantize_q8_0x4_interleaved::<8>(+ lhs,+ k,+ group * 4,+ &mut lhs_x4[group * k_in_blocks..(group + 1) * k_in_blocks],+ );+ }++ let tiles_total = row_groups * n_groups;+ let pool = crate::utils::barrier_pool();+ let lhs_ptr = lhs_x4.as_ptr() as usize;+ let repacked_ptr = repacked.as_ptr() as usize;+ let dst_ptr = dst.as_mut_ptr() as usize;++ pool.execute_chunked(tiles_total, |range| {+ let lhs_ptr = lhs_ptr as *const BlockQ8_0x4;+ let repacked_ptr = repacked_ptr as *const BlockQ4_0x4;+ let dst_ptr = dst_ptr as *mut f32;+ for tile in range {+ let row_group = tile / n_groups;+ let col_group = tile - row_group * n_groups;+ let row = row_group * 4;+ let col = col_group * 4;+ let lhs_tile = unsafe {+ std::slice::from_raw_parts(lhs_ptr.add(row_group * k_in_blocks), k_in_blocks)+ };+ let rhs_tile = unsafe {+ std::slice::from_raw_parts(+ repacked_ptr.add(col_group * k_in_blocks),+ k_in_blocks,+ )+ };+ unsafe {+ store_q4_0x4_4x4_i8mm(dst_ptr, n, row, col, lhs_tile, rhs_tile, k_in_blocks)+ };+ }+ });++ Ok(())+ })+}++#[cfg(target_arch = "aarch64")]+fn quantize_q8k_x4_interleaved<const BLOCK_LEN: usize>(+ lhs: &[f32],+ k: usize,+ row: usize,+ out: &mut [BlockQ8Kx4],+) {+ let k_in_blocks = k / QK_K;+ for b in 0..k_in_blocks {+ let mut p = BlockQ8Kx4 {+ d: [0.; 4],+ qs: [0; QK_K * 4],+ bsums: [0; QK_K / 4],+ };+ let mut src = [[0f32; QK_K]; 4];+ let mut iscale = [0f32; 4];+ for r in 0..4 {+ let xs = &lhs[(row + r) * k + b * QK_K..(row + r) * k + (b + 1) * QK_K];+ let mut max = 0f32;+ let mut amax = 0f32;+ for (j, &x) in xs.iter().enumerate() {+ src[r][j] = x;+ if amax < x.abs() {+ amax = x.abs();+ max = x;+ }+ }+ iscale[r] = if amax != 0. { -127. / max } else { 0. };+ p.d[r] = if amax != 0. { 1. / iscale[r] } else { 0. };+ }+ for j in 0..QK_K * 4 {+ let src_offset = (j / (4 * BLOCK_LEN)) * BLOCK_LEN + j % BLOCK_LEN;+ let src_id = (j % (4 * BLOCK_LEN)) / BLOCK_LEN;+ let q = (src[src_id][src_offset] * iscale[src_id]).round() as i8;+ p.qs[j] = q;+ let index = if BLOCK_LEN == 4 {+ (((j & 15) >> 2) << 2) + ((j >> 8) << 4) + ((j >> 6) & 3)+ } else {+ (((j & 31) >> 3) << 2) + ((j >> 8) << 4) + ((j >> 6) & 3)+ };+ p.bsums[index] += q as i16;+ }+ out[b] = p;+ }+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "i8mm")]+unsafe fn store_q4kx8_4x8_i8mm(+ dst: *mut f32,+ n: usize,+ row: usize,+ col: usize,+ lhs: &[BlockQ8Kx4],+ rhs: &[BlockQ4Kx8],+ k_in_blocks: usize,+) {+ let m4b = vdupq_n_u8(0x0f);+ let mut acc_f32 = [vdupq_n_f32(0.); 8];++ for b in 0..k_in_blocks {+ let x = &rhs[b];+ let y = &lhs[b];+ let mut bsums_arr = [[0i16; 8]; 4];+ for (r, bsums_row) in bsums_arr.iter_mut().enumerate() {+ for (i, bsums) in bsums_row.iter_mut().enumerate() {+ // quarter-major interleaved layout: [16 * quarter + 4 * row + half-pair]+ let base = 16 * (i / 2) + 4 * r + (2 * i) % 4;+ *bsums = y.bsums[base] + y.bsums[base + 1];+ }+ }++ let mut acc = [vdupq_n_s32(0); 8];+ let mut bias_acc = [vdupq_n_s32(0); 8];++ for sb in 0..QK_K / 64 {+ let (mins_0, scales_0) = decode_q4kx8_scales(x.scales.as_ptr().add(sb * 24));+ let (mins_1, scales_1) = decode_q4kx8_scales(x.scales.as_ptr().add(sb * 24 + 12));+ let q4sb_mins = [mins_0, mins_1];+ let mut q4sb_scales = [[0i16; 8]; 2];+ vst1q_s16(q4sb_scales[0].as_mut_ptr(), scales_0);+ vst1q_s16(q4sb_scales[1].as_mut_ptr(), scales_1);++ let q8_base = y.qs.as_ptr().add(sb * 256);+ let mut q8s = [[vdupq_n_s8(0); 8]; 2];+ let (q8s_0, q8s_1) = q8s.split_at_mut(1);+ for (i, (q8_0, q8_1)) in q8s_0[0].iter_mut().zip(q8s_1[0].iter_mut()).enumerate() {+ *q8_0 = vld1q_s8(q8_base.add(i * 32));+ *q8_1 = vld1q_s8(q8_base.add(i * 32 + 16));+ }++ for cp in 0..4 {+ let qs = [+ vld1q_u8(x.qs.as_ptr().add(sb * QK_K + 16 * cp)),+ vld1q_u8(x.qs.as_ptr().add(sb * QK_K + 16 * cp + 64)),+ vld1q_u8(x.qs.as_ptr().add(sb * QK_K + 16 * cp + 128)),+ vld1q_u8(x.qs.as_ptr().add(sb * QK_K + 16 * cp + 192)),+ ];++ let mut acc_0 = vdupq_n_s32(0);+ let mut acc_1 = vdupq_n_s32(0);+ let mut acc_2 = vdupq_n_s32(0);+ let mut acc_3 = vdupq_n_s32(0);++ for i in 0..4 {+ let qs_lo = vreinterpretq_s8_u8(vandq_u8(qs[i], m4b));+ let qs_hi = vreinterpretq_s8_u8(vshrq_n_u8(qs[i], 4));+ acc_0 = smmla_s32(acc_0, qs_lo, q8s[0][i]);+ acc_2 = smmla_s32(acc_2, qs_lo, q8s[1][i]);+ acc_1 = smmla_s32(acc_1, qs_hi, q8s[0][i + 4]);+ acc_3 = smmla_s32(acc_3, qs_hi, q8s[1][i + 4]);+ }++ let scale_offset = cp * 2;+ let scale_0 = vcombine_s32(+ vdup_n_s32(q4sb_scales[0][scale_offset] as i32),+ vdup_n_s32(q4sb_scales[0][scale_offset + 1] as i32),+ );+ let scale_1 = vcombine_s32(+ vdup_n_s32(q4sb_scales[1][scale_offset] as i32),+ vdup_n_s32(q4sb_scales[1][scale_offset + 1] as i32),+ );+ acc[cp] = vmlaq_s32(acc[cp], acc_0, scale_0);+ acc[cp] = vmlaq_s32(acc[cp], acc_1, scale_1);+ acc[cp + 4] = vmlaq_s32(acc[cp + 4], acc_2, scale_0);+ acc[cp + 4] = vmlaq_s32(acc[cp + 4], acc_3, scale_1);+ }++ for q8_row in 0..4 {+ let bsums_vec_lo = vdup_n_s16(bsums_arr[q8_row][sb * 2]);+ let bsums_vec_hi = vdup_n_s16(bsums_arr[q8_row][sb * 2 + 1]);+ bias_acc[2 * q8_row] = vmlal_s16(+ bias_acc[2 * q8_row],+ bsums_vec_lo,+ vget_low_s16(q4sb_mins[0]),+ );+ bias_acc[2 * q8_row] = vmlal_s16(+ bias_acc[2 * q8_row],+ bsums_vec_hi,+ vget_low_s16(q4sb_mins[1]),+ );+ bias_acc[2 * q8_row + 1] = vmlal_s16(+ bias_acc[2 * q8_row + 1],+ bsums_vec_lo,+ vget_high_s16(q4sb_mins[0]),+ );+ bias_acc[2 * q8_row + 1] = vmlal_s16(+ bias_acc[2 * q8_row + 1],+ bsums_vec_hi,+ vget_high_s16(q4sb_mins[1]),+ );+ }+ }++ for item in &mut acc {+ let aux = vzip_s32(vget_low_s32(*item), vget_high_s32(*item));+ *item = vcombine_s32(aux.0, aux.1);+ }+ let reorder_acc = [+ vcombine_s32(vget_low_s32(acc[0]), vget_low_s32(acc[1])),+ vcombine_s32(vget_low_s32(acc[2]), vget_low_s32(acc[3])),+ vcombine_s32(vget_high_s32(acc[0]), vget_high_s32(acc[1])),+ vcombine_s32(vget_high_s32(acc[2]), vget_high_s32(acc[3])),+ vcombine_s32(vget_low_s32(acc[4]), vget_low_s32(acc[5])),+ vcombine_s32(vget_low_s32(acc[6]), vget_low_s32(acc[7])),+ vcombine_s32(vget_high_s32(acc[4]), vget_high_s32(acc[5])),+ vcombine_s32(vget_high_s32(acc[6]), vget_high_s32(acc[7])),+ ];++ for r in 0..4 {+ for g in 0..2 {+ let q4_d = [+ x.d[g * 4].to_f32(),+ x.d[g * 4 + 1].to_f32(),+ x.d[g * 4 + 2].to_f32(),+ x.d[g * 4 + 3].to_f32(),+ ];+ let q4_dmin = [+ x.dmin[g * 4].to_f32(),+ x.dmin[g * 4 + 1].to_f32(),+ x.dmin[g * 4 + 2].to_f32(),+ x.dmin[g * 4 + 3].to_f32(),+ ];+ let q8_d = y.d[r];+ let scale = vmulq_n_f32(vld1q_f32(q4_d.as_ptr()), q8_d);+ let dmins = vmulq_n_f32(vld1q_f32(q4_dmin.as_ptr()), q8_d);+ let idx = 2 * r + g;+ acc_f32[idx] =+ vsubq_f32(acc_f32[idx], vmulq_f32(vcvtq_f32_s32(bias_acc[idx]), dmins));+ acc_f32[idx] = vmlaq_f32(acc_f32[idx], vcvtq_f32_s32(reorder_acc[idx]), scale);+ }+ }+ }++ for r in 0..4 {+ vst1q_f32(dst.add((row + r) * n + col), acc_f32[2 * r]);+ vst1q_f32(dst.add((row + r) * n + col + 4), acc_f32[2 * r + 1]);+ }+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "i8mm")]+pub(crate) fn matmul_q4k_x8_i8mm(+ (m, k, n): (usize, usize, usize),+ lhs: &[f32],+ repacked: &[BlockQ4Kx8],+ dst: &mut [f32],+) -> crate::Result<()> {+ debug_assert!(k.is_multiple_of(QK_K));+ debug_assert!(n.is_multiple_of(8));+ debug_assert!(m.is_multiple_of(4));+ let k_in_blocks = k / QK_K;+ let n_groups = n / 8;+ let row_groups = m / 4;++ thread_local! {+ static LHS_Q4K_X8_I8MM_SCRATCH: std::cell::RefCell<Vec<u64>> =+ const { std::cell::RefCell::new(Vec::new()) };+ }++ let elem_size = std::mem::size_of::<BlockQ8Kx4>();+ let required_len = (row_groups * k_in_blocks * elem_size).div_ceil(8);+ LHS_Q4K_X8_I8MM_SCRATCH.with(|cell| -> crate::Result<()> {+ let mut scratch = cell.borrow_mut();+ if scratch.len() < required_len {+ scratch.resize(required_len, 0);+ }+ let lhs_x4: &mut [BlockQ8Kx4] = unsafe {+ std::slice::from_raw_parts_mut(+ scratch.as_mut_ptr() as *mut BlockQ8Kx4,+ row_groups * k_in_blocks,+ )+ };+ for group in 0..row_groups {+ quantize_q8k_x4_interleaved::<8>(+ lhs,+ k,+ group * 4,+ &mut lhs_x4[group * k_in_blocks..(group + 1) * k_in_blocks],+ );+ }++ let tiles_total = row_groups * n_groups;+ let pool = crate::utils::barrier_pool();+ let lhs_ptr = lhs_x4.as_ptr() as usize;+ let repacked_ptr = repacked.as_ptr() as usize;+ let dst_ptr = dst.as_mut_ptr() as usize;++ pool.execute_chunked(tiles_total, |range| {+ let lhs_ptr = lhs_ptr as *const BlockQ8Kx4;+ let repacked_ptr = repacked_ptr as *const BlockQ4Kx8;+ let dst_ptr = dst_ptr as *mut f32;+ for tile in range {+ let row_group = tile / n_groups;+ let col_group = tile - row_group * n_groups;+ let row = row_group * 4;+ let col = col_group * 8;+ let lhs_tile = unsafe {+ std::slice::from_raw_parts(lhs_ptr.add(row_group * k_in_blocks), k_in_blocks)+ };+ let rhs_tile = unsafe {+ std::slice::from_raw_parts(+ repacked_ptr.add(col_group * k_in_blocks),+ k_in_blocks,+ )+ };+ unsafe {+ store_q4kx8_4x8_i8mm(dst_ptr, n, row, col, lhs_tile, rhs_tile, k_in_blocks)+ };+ }+ });++ Ok(())+ })+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "dotprod")]+unsafe fn store_q5kx8_4x8(+ dst: *mut f32,+ n: usize,+ row: usize,+ col: usize,+ lhs: &[BlockQ8Kx4],+ rhs: &[BlockQ5Kx8],+ k_in_blocks: usize,+) {+ let m4b = vdupq_n_u8(0x0f);+ let mone = vdupq_n_u8(1);+ let mtwo = vdupq_n_u8(2);+ let mut acc_f32 = [vdupq_n_f32(0.); 8];++ for b in 0..k_in_blocks {+ let x = &rhs[b];+ let y = &lhs[b];+ let q5_d_0123 = vld1q_f32(+ [+ x.d[0].to_f32(),+ x.d[1].to_f32(),+ x.d[2].to_f32(),+ x.d[3].to_f32(),+ ]+ .as_ptr(),+ );+ let q5_d_4567 = vld1q_f32(+ [+ x.d[4].to_f32(),+ x.d[5].to_f32(),+ x.d[6].to_f32(),+ x.d[7].to_f32(),+ ]+ .as_ptr(),+ );+ let q5_dmin_0123 = vld1q_f32(+ [+ x.dmin[0].to_f32(),+ x.dmin[1].to_f32(),+ x.dmin[2].to_f32(),+ x.dmin[3].to_f32(),+ ]+ .as_ptr(),+ );+ let q5_dmin_4567 = vld1q_f32(+ [+ x.dmin[4].to_f32(),+ x.dmin[5].to_f32(),+ x.dmin[6].to_f32(),+ x.dmin[7].to_f32(),+ ]+ .as_ptr(),+ );++ let sbd_scale_0123 = [+ vmulq_n_f32(q5_d_0123, y.d[0]),+ vmulq_n_f32(q5_d_0123, y.d[1]),+ vmulq_n_f32(q5_d_0123, y.d[2]),+ vmulq_n_f32(q5_d_0123, y.d[3]),+ ];+ let sbd_scale_4567 = [+ vmulq_n_f32(q5_d_4567, y.d[0]),+ vmulq_n_f32(q5_d_4567, y.d[1]),+ vmulq_n_f32(q5_d_4567, y.d[2]),+ vmulq_n_f32(q5_d_4567, y.d[3]),+ ];+ let sbd_min_0123 = [+ vmulq_n_f32(q5_dmin_0123, y.d[0]),+ vmulq_n_f32(q5_dmin_0123, y.d[1]),+ vmulq_n_f32(q5_dmin_0123, y.d[2]),+ vmulq_n_f32(q5_dmin_0123, y.d[3]),+ ];+ let sbd_min_4567 = [+ vmulq_n_f32(q5_dmin_4567, y.d[0]),+ vmulq_n_f32(q5_dmin_4567, y.d[1]),+ vmulq_n_f32(q5_dmin_4567, y.d[2]),+ vmulq_n_f32(q5_dmin_4567, y.d[3]),+ ];++ let mut bsums_arr = [[0i16; 8]; 4];+ for r in 0..4 {+ for i in 0..8 {+ // quarter-major interleaved layout: [16 * quarter + 4 * row + half-pair]+ let base = 16 * (i / 2) + 4 * r + (2 * i) % 4;+ bsums_arr[r][i] = y.bsums[base] + y.bsums[base + 1];+ }+ }++ let mut bias_acc = [vdupq_n_s32(0); 8];+ let mut qh = [[vdupq_n_u8(0); 8]; 2];+ for c in 0..2 {+ for i in 0..8 {+ qh[c][i] = vld1q_u8(x.qh.as_ptr().add(i * 32 + 16 * c));+ }+ }++ for sb in 0..QK_K / 64 {+ let mut acc_lo = [vdupq_n_s32(0); 8];+ let mut acc_hi = [vdupq_n_s32(0); 8];+ let (q5sb_mins_0, q5sb_scales_0) = decode_q4kx8_scales(x.scales.as_ptr().add(sb * 24));+ let (q5sb_mins_1, q5sb_scales_1) =+ decode_q4kx8_scales(x.scales.as_ptr().add(sb * 24 + 12));+ let q5sb_mins = [q5sb_mins_0, q5sb_mins_1];+ let q5sb_scales = [q5sb_scales_0, q5sb_scales_1];++ for k_idx in 0..8 {+ let q8_blk0 = vld1q_s8(y.qs.as_ptr().add(sb * 256 + 16 * k_idx));+ let q8_blk1 = vld1q_s8(y.qs.as_ptr().add(sb * 256 + 16 * k_idx + 128));+ let q5_0123 = vld1q_u8(x.qs.as_ptr().add(sb * QK_K + 32 * k_idx));+ let q5_4567 = vld1q_u8(x.qs.as_ptr().add(sb * QK_K + 32 * k_idx + 16));++ let hbit_lo_0123 = vandq_u8(qh[0][k_idx], mone);+ let hbit_hi_0123 = vshlq_n_u8(vandq_u8(qh[0][k_idx], mtwo), 3);+ qh[0][k_idx] = vshrq_n_u8(qh[0][k_idx], 2);+ let hbit_lo_4567 = vandq_u8(qh[1][k_idx], mone);+ let hbit_hi_4567 = vshlq_n_u8(vandq_u8(qh[1][k_idx], mtwo), 3);+ qh[1][k_idx] = vshrq_n_u8(qh[1][k_idx], 2);++ let q5_0123_lo = vreinterpretq_s8_u8(vorrq_u8(+ vandq_u8(q5_0123, m4b),+ vshlq_n_u8(hbit_lo_0123, 4),+ ));+ let q5_0123_hi =+ vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q5_0123, 4), hbit_hi_0123));++ acc_lo[0] = sdot_laneq_s32::<0>(acc_lo[0], q5_0123_lo, q8_blk0);+ acc_lo[1] = sdot_laneq_s32::<1>(acc_lo[1], q5_0123_lo, q8_blk0);+ acc_lo[2] = sdot_laneq_s32::<2>(acc_lo[2], q5_0123_lo, q8_blk0);+ acc_lo[3] = sdot_laneq_s32::<3>(acc_lo[3], q5_0123_lo, q8_blk0);++ acc_hi[0] = sdot_laneq_s32::<0>(acc_hi[0], q5_0123_hi, q8_blk1);+ acc_hi[1] = sdot_laneq_s32::<1>(acc_hi[1], q5_0123_hi, q8_blk1);+ acc_hi[2] = sdot_laneq_s32::<2>(acc_hi[2], q5_0123_hi, q8_blk1);+ acc_hi[3] = sdot_laneq_s32::<3>(acc_hi[3], q5_0123_hi, q8_blk1);++ let q5_4567_lo = vreinterpretq_s8_u8(vorrq_u8(+ vandq_u8(q5_4567, m4b),+ vshlq_n_u8(hbit_lo_4567, 4),+ ));+ let q5_4567_hi =+ vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q5_4567, 4), hbit_hi_4567));++ acc_lo[4] = sdot_laneq_s32::<0>(acc_lo[4], q5_4567_lo, q8_blk0);+ acc_lo[5] = sdot_laneq_s32::<1>(acc_lo[5], q5_4567_lo, q8_blk0);+ acc_lo[6] = sdot_laneq_s32::<2>(acc_lo[6], q5_4567_lo, q8_blk0);+ acc_lo[7] = sdot_laneq_s32::<3>(acc_lo[7], q5_4567_lo, q8_blk0);++ acc_hi[4] = sdot_laneq_s32::<0>(acc_hi[4], q5_4567_hi, q8_blk1);+ acc_hi[5] = sdot_laneq_s32::<1>(acc_hi[5], q5_4567_hi, q8_blk1);+ acc_hi[6] = sdot_laneq_s32::<2>(acc_hi[6], q5_4567_hi, q8_blk1);+ acc_hi[7] = sdot_laneq_s32::<3>(acc_hi[7], q5_4567_hi, q8_blk1);+ }++ let sc_0123_lo = vget_low_s16(q5sb_scales[0]);+ let sc_4567_lo = vget_high_s16(q5sb_scales[0]);+ let sc_0123_hi = vget_low_s16(q5sb_scales[1]);+ let sc_4567_hi = vget_high_s16(q5sb_scales[1]);+ for r in 0..4 {+ let sumf_0123 = vcvtq_f32_s32(vaddq_s32(+ vmulq_s32(vmovl_s16(sc_0123_lo), acc_lo[r]),+ vmulq_s32(vmovl_s16(sc_0123_hi), acc_hi[r]),+ ));+ acc_f32[2 * r] = vmlaq_f32(acc_f32[2 * r], sbd_scale_0123[r], sumf_0123);++ let sumf_4567 = vcvtq_f32_s32(vaddq_s32(+ vmulq_s32(vmovl_s16(sc_4567_lo), acc_lo[r + 4]),+ vmulq_s32(vmovl_s16(sc_4567_hi), acc_hi[r + 4]),+ ));+ acc_f32[2 * r + 1] = vmlaq_f32(acc_f32[2 * r + 1], sbd_scale_4567[r], sumf_4567);++ let bsums_vec_lo = vdup_n_s16(bsums_arr[r][sb * 2]);+ let bsums_vec_hi = vdup_n_s16(bsums_arr[r][sb * 2 + 1]);+ bias_acc[2 * r] =+ vmlal_s16(bias_acc[2 * r], bsums_vec_lo, vget_low_s16(q5sb_mins[0]));+ bias_acc[2 * r] =+ vmlal_s16(bias_acc[2 * r], bsums_vec_hi, vget_low_s16(q5sb_mins[1]));+ bias_acc[2 * r + 1] = vmlal_s16(+ bias_acc[2 * r + 1],+ bsums_vec_lo,+ vget_high_s16(q5sb_mins[0]),+ );+ bias_acc[2 * r + 1] = vmlal_s16(+ bias_acc[2 * r + 1],+ bsums_vec_hi,+ vget_high_s16(q5sb_mins[1]),+ );+ }+ }++ for r in 0..4 {+ acc_f32[2 * r] = vsubq_f32(+ acc_f32[2 * r],+ vmulq_f32(vcvtq_f32_s32(bias_acc[2 * r]), sbd_min_0123[r]),+ );+ acc_f32[2 * r + 1] = vsubq_f32(+ acc_f32[2 * r + 1],+ vmulq_f32(vcvtq_f32_s32(bias_acc[2 * r + 1]), sbd_min_4567[r]),+ );+ }+ }++ for r in 0..4 {+ vst1q_f32(dst.add((row + r) * n + col), acc_f32[2 * r]);+ vst1q_f32(dst.add((row + r) * n + col + 4), acc_f32[2 * r + 1]);+ }+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "i8mm")]+unsafe fn store_q5kx8_4x8_i8mm(+ dst: *mut f32,+ n: usize,+ row: usize,+ col: usize,+ lhs: &[BlockQ8Kx4],+ rhs: &[BlockQ5Kx8],+ k_in_blocks: usize,+) {+ let m4b = vdupq_n_u8(0x0f);+ let mone = vdupq_n_u8(1);+ let mtwo = vdupq_n_u8(2);+ let mut acc_f32 = [vdupq_n_f32(0.); 8];++ for b in 0..k_in_blocks {+ let x = &rhs[b];+ let y = &lhs[b];+ let mut bsums_arr = [[0i16; 8]; 4];+ for (r, bsums_row) in bsums_arr.iter_mut().enumerate() {+ for (i, bsums) in bsums_row.iter_mut().enumerate() {+ // quarter-major interleaved layout: [16 * quarter + 4 * row + half-pair]+ let base = 16 * (i / 2) + 4 * r + (2 * i) % 4;+ *bsums = y.bsums[base] + y.bsums[base + 1];+ }+ }++ let mut acc = [vdupq_n_s32(0); 8];+ let mut bias_acc = [vdupq_n_s32(0); 8];+ let mut qh = [[vdupq_n_u8(0); 4]; 4];+ for (cp, qh_cp) in qh.iter_mut().enumerate() {+ qh_cp[0] = vld1q_u8(x.qh.as_ptr().add(16 * cp));+ qh_cp[1] = vld1q_u8(x.qh.as_ptr().add(16 * cp + 64));+ qh_cp[2] = vld1q_u8(x.qh.as_ptr().add(16 * cp + 128));+ qh_cp[3] = vld1q_u8(x.qh.as_ptr().add(16 * cp + 192));+ }++ for sb in 0..QK_K / 64 {+ let (mins_0, scales_0) = decode_q4kx8_scales(x.scales.as_ptr().add(sb * 24));+ let (mins_1, scales_1) = decode_q4kx8_scales(x.scales.as_ptr().add(sb * 24 + 12));+ let q5sb_mins = [mins_0, mins_1];+ let mut q5sb_scales = [[0i16; 8]; 2];+ vst1q_s16(q5sb_scales[0].as_mut_ptr(), scales_0);+ vst1q_s16(q5sb_scales[1].as_mut_ptr(), scales_1);++ let q8_base = y.qs.as_ptr().add(sb * 256);+ let mut q8s = [[vdupq_n_s8(0); 8]; 2];+ let (q8s_0, q8s_1) = q8s.split_at_mut(1);+ for (i, (q8_0, q8_1)) in q8s_0[0].iter_mut().zip(q8s_1[0].iter_mut()).enumerate() {+ *q8_0 = vld1q_s8(q8_base.add(i * 32));+ *q8_1 = vld1q_s8(q8_base.add(i * 32 + 16));+ }++ for cp in 0..4 {+ let qs = [+ vld1q_u8(x.qs.as_ptr().add(sb * QK_K + 16 * cp)),+ vld1q_u8(x.qs.as_ptr().add(sb * QK_K + 16 * cp + 64)),+ vld1q_u8(x.qs.as_ptr().add(sb * QK_K + 16 * cp + 128)),+ vld1q_u8(x.qs.as_ptr().add(sb * QK_K + 16 * cp + 192)),+ ];++ let mut acc_0 = vdupq_n_s32(0);+ let mut acc_1 = vdupq_n_s32(0);+ let mut acc_2 = vdupq_n_s32(0);+ let mut acc_3 = vdupq_n_s32(0);++ for i in 0..4 {+ let hbit_lo = vandq_u8(qh[cp][i], mone);+ let hbit_hi = vshlq_n_u8(vandq_u8(qh[cp][i], mtwo), 3);+ qh[cp][i] = vshrq_n_u8(qh[cp][i], 2);++ let qs_lo =+ vreinterpretq_s8_u8(vorrq_u8(vandq_u8(qs[i], m4b), vshlq_n_u8(hbit_lo, 4)));+ let qs_hi = vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(qs[i], 4), hbit_hi));++ acc_0 = smmla_s32(acc_0, qs_lo, q8s[0][i]);+ acc_2 = smmla_s32(acc_2, qs_lo, q8s[1][i]);+ acc_1 = smmla_s32(acc_1, qs_hi, q8s[0][i + 4]);+ acc_3 = smmla_s32(acc_3, qs_hi, q8s[1][i + 4]);+ }++ let scale_offset = cp * 2;+ let scale_0 = vcombine_s32(+ vdup_n_s32(q5sb_scales[0][scale_offset] as i32),+ vdup_n_s32(q5sb_scales[0][scale_offset + 1] as i32),+ );+ let scale_1 = vcombine_s32(+ vdup_n_s32(q5sb_scales[1][scale_offset] as i32),+ vdup_n_s32(q5sb_scales[1][scale_offset + 1] as i32),+ );+ acc[cp] = vmlaq_s32(acc[cp], acc_0, scale_0);+ acc[cp] = vmlaq_s32(acc[cp], acc_1, scale_1);+ acc[cp + 4] = vmlaq_s32(acc[cp + 4], acc_2, scale_0);+ acc[cp + 4] = vmlaq_s32(acc[cp + 4], acc_3, scale_1);+ }++ for q8_row in 0..4 {+ let bsums_vec_lo = vdup_n_s16(bsums_arr[q8_row][sb * 2]);+ let bsums_vec_hi = vdup_n_s16(bsums_arr[q8_row][sb * 2 + 1]);+ bias_acc[2 * q8_row] = vmlal_s16(+ bias_acc[2 * q8_row],+ bsums_vec_lo,+ vget_low_s16(q5sb_mins[0]),+ );+ bias_acc[2 * q8_row] = vmlal_s16(+ bias_acc[2 * q8_row],+ bsums_vec_hi,+ vget_low_s16(q5sb_mins[1]),+ );+ bias_acc[2 * q8_row + 1] = vmlal_s16(+ bias_acc[2 * q8_row + 1],+ bsums_vec_lo,+ vget_high_s16(q5sb_mins[0]),+ );+ bias_acc[2 * q8_row + 1] = vmlal_s16(+ bias_acc[2 * q8_row + 1],+ bsums_vec_hi,+ vget_high_s16(q5sb_mins[1]),+ );+ }+ }++ for item in &mut acc {+ let aux = vzip_s32(vget_low_s32(*item), vget_high_s32(*item));+ *item = vcombine_s32(aux.0, aux.1);+ }+ let reorder_acc = [+ vcombine_s32(vget_low_s32(acc[0]), vget_low_s32(acc[1])),+ vcombine_s32(vget_low_s32(acc[2]), vget_low_s32(acc[3])),+ vcombine_s32(vget_high_s32(acc[0]), vget_high_s32(acc[1])),+ vcombine_s32(vget_high_s32(acc[2]), vget_high_s32(acc[3])),+ vcombine_s32(vget_low_s32(acc[4]), vget_low_s32(acc[5])),+ vcombine_s32(vget_low_s32(acc[6]), vget_low_s32(acc[7])),+ vcombine_s32(vget_high_s32(acc[4]), vget_high_s32(acc[5])),+ vcombine_s32(vget_high_s32(acc[6]), vget_high_s32(acc[7])),+ ];++ for r in 0..4 {+ for g in 0..2 {+ let q5_d = [+ x.d[g * 4].to_f32(),+ x.d[g * 4 + 1].to_f32(),+ x.d[g * 4 + 2].to_f32(),+ x.d[g * 4 + 3].to_f32(),+ ];+ let q5_dmin = [+ x.dmin[g * 4].to_f32(),+ x.dmin[g * 4 + 1].to_f32(),+ x.dmin[g * 4 + 2].to_f32(),+ x.dmin[g * 4 + 3].to_f32(),+ ];+ let q8_d = y.d[r];+ let scale = vmulq_n_f32(vld1q_f32(q5_d.as_ptr()), q8_d);+ let dmins = vmulq_n_f32(vld1q_f32(q5_dmin.as_ptr()), q8_d);+ let idx = 2 * r + g;+ acc_f32[idx] =+ vsubq_f32(acc_f32[idx], vmulq_f32(vcvtq_f32_s32(bias_acc[idx]), dmins));+ acc_f32[idx] = vmlaq_f32(acc_f32[idx], vcvtq_f32_s32(reorder_acc[idx]), scale);+ }+ }+ }++ for r in 0..4 {+ vst1q_f32(dst.add((row + r) * n + col), acc_f32[2 * r]);+ vst1q_f32(dst.add((row + r) * n + col + 4), acc_f32[2 * r + 1]);+ }+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "dotprod")]+unsafe fn vec_dot_8_q5kx8_q8k_8x4(n: usize, xs: &[BlockQ5Kx8], ys: &[BlockQ8K]) -> [f32; 8] {+ debug_assert!(n.is_multiple_of(QK_K));+ let mut out = [0f32; 8];+ let m4b = vdupq_n_u8(0x0f);+ let mone = vdupq_n_u8(1);+ let mtwo = vdupq_n_u8(2);+ let mut acc_f32_0 = vdupq_n_f32(0.);+ let mut acc_f32_1 = vdupq_n_f32(0.);++ for (q5, q8) in xs.iter().zip(ys.iter()) {+ let sb_scale_0 = vmulq_f32(load_f16x4(q5.d.as_ptr()), vdupq_n_f32(q8.d));+ let sb_scale_1 = vmulq_f32(load_f16x4(q5.d.as_ptr().add(4)), vdupq_n_f32(q8.d));+ let sb_min_0 = vmulq_f32(load_f16x4(q5.dmin.as_ptr()), vdupq_n_f32(q8.d));+ let sb_min_1 = vmulq_f32(load_f16x4(q5.dmin.as_ptr().add(4)), vdupq_n_f32(q8.d));+ let bsums = vpaddq_s16(+ vld1q_s16(q8.bsums.as_ptr()),+ vld1q_s16(q8.bsums.as_ptr().add(8)),+ );+ let mut bsums_arr = [0i16; 8];+ vst1q_s16(bsums_arr.as_mut_ptr(), bsums);+ let mut qh = [[vdupq_n_u8(0); 8]; 2];+ for c in 0..2 {+ for i in 0..8 {+ qh[c][i] = vld1q_u8(q5.qh.as_ptr().add(i * 32 + 16 * c));+ }+ }+ let mut bias_acc = [vdupq_n_s32(0); 2];++ for sb in 0..QK_K / 64 {+ let mut acc_lo = [vdupq_n_s32(0); 2];+ let mut acc_hi = [vdupq_n_s32(0); 2];+ let (mins_0, scales_0) = decode_q4kx8_scales(q5.scales.as_ptr().add(sb * 24));+ let (mins_1, scales_1) = decode_q4kx8_scales(q5.scales.as_ptr().add(sb * 24 + 12));+ let q5sb_mins = [mins_0, mins_1];+ let q5sb_scales = [scales_0, scales_1];+ let q8_qs = [+ vld1q_s8(q8.qs.as_ptr().add(sb * 64)),+ vld1q_s8(q8.qs.as_ptr().add(sb * 64 + 16)),+ vld1q_s8(q8.qs.as_ptr().add(sb * 64 + 32)),+ vld1q_s8(q8.qs.as_ptr().add(sb * 64 + 48)),+ ];++ for c in 0..2 {+ for i in 0..8 {+ let q5_col = vld1q_u8(q5.qs.as_ptr().add(sb * QK_K + i * 32 + 16 * c));+ let hbit_lo = vandq_u8(qh[c][i], mone);+ let hbit_hi = vshlq_n_u8(vandq_u8(qh[c][i], mtwo), 3);+ qh[c][i] = vshrq_n_u8(qh[c][i], 2);+ let q5_lo = vreinterpretq_s8_u8(vsliq_n_u8(vandq_u8(q5_col, m4b), hbit_lo, 4));+ let q5_hi = vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q5_col, 4), hbit_hi));+ match i {+ 0 => {+ acc_lo[c] = sdot_laneq_s32::<0>(acc_lo[c], q5_lo, q8_qs[0]);+ acc_hi[c] = sdot_laneq_s32::<0>(acc_hi[c], q5_hi, q8_qs[2]);+ }+ 1 => {+ acc_lo[c] = sdot_laneq_s32::<1>(acc_lo[c], q5_lo, q8_qs[0]);+ acc_hi[c] = sdot_laneq_s32::<1>(acc_hi[c], q5_hi, q8_qs[2]);+ }+ 2 => {+ acc_lo[c] = sdot_laneq_s32::<2>(acc_lo[c], q5_lo, q8_qs[0]);+ acc_hi[c] = sdot_laneq_s32::<2>(acc_hi[c], q5_hi, q8_qs[2]);+ }+ 3 => {+ acc_lo[c] = sdot_laneq_s32::<3>(acc_lo[c], q5_lo, q8_qs[0]);+ acc_hi[c] = sdot_laneq_s32::<3>(acc_hi[c], q5_hi, q8_qs[2]);+ }+ 4 => {+ acc_lo[c] = sdot_laneq_s32::<0>(acc_lo[c], q5_lo, q8_qs[1]);+ acc_hi[c] = sdot_laneq_s32::<0>(acc_hi[c], q5_hi, q8_qs[3]);+ }+ 5 => {+ acc_lo[c] = sdot_laneq_s32::<1>(acc_lo[c], q5_lo, q8_qs[1]);+ acc_hi[c] = sdot_laneq_s32::<1>(acc_hi[c], q5_hi, q8_qs[3]);+ }+ 6 => {+ acc_lo[c] = sdot_laneq_s32::<2>(acc_lo[c], q5_lo, q8_qs[1]);+ acc_hi[c] = sdot_laneq_s32::<2>(acc_hi[c], q5_hi, q8_qs[3]);+ }+ _ => {+ acc_lo[c] = sdot_laneq_s32::<3>(acc_lo[c], q5_lo, q8_qs[1]);+ acc_hi[c] = sdot_laneq_s32::<3>(acc_hi[c], q5_hi, q8_qs[3]);+ }+ }+ }+ }++ let sumf_0 = vcvtq_f32_s32(vaddq_s32(+ vmulq_s32(vmovl_s16(vget_low_s16(q5sb_scales[0])), acc_lo[0]),+ vmulq_s32(vmovl_s16(vget_low_s16(q5sb_scales[1])), acc_hi[0]),+ ));+ acc_f32_0 = vfmaq_f32(acc_f32_0, sb_scale_0, sumf_0);+ let sumf_1 = vcvtq_f32_s32(vaddq_s32(+ vmulq_s32(vmovl_s16(vget_high_s16(q5sb_scales[0])), acc_lo[1]),+ vmulq_s32(vmovl_s16(vget_high_s16(q5sb_scales[1])), acc_hi[1]),+ ));+ acc_f32_1 = vfmaq_f32(acc_f32_1, sb_scale_1, sumf_1);++ let bsums_vec_lo = vdup_n_s16(bsums_arr[2 * sb]);+ let bsums_vec_hi = vdup_n_s16(bsums_arr[2 * sb + 1]);+ bias_acc[0] = vmlal_s16(bias_acc[0], bsums_vec_lo, vget_low_s16(q5sb_mins[0]));+ bias_acc[0] = vmlal_s16(bias_acc[0], bsums_vec_hi, vget_low_s16(q5sb_mins[1]));+ bias_acc[1] = vmlal_s16(bias_acc[1], bsums_vec_lo, vget_high_s16(q5sb_mins[0]));+ bias_acc[1] = vmlal_s16(bias_acc[1], bsums_vec_hi, vget_high_s16(q5sb_mins[1]));+ }++ acc_f32_0 = vmlsq_f32(acc_f32_0, vcvtq_f32_s32(bias_acc[0]), sb_min_0);+ acc_f32_1 = vmlsq_f32(acc_f32_1, vcvtq_f32_s32(bias_acc[1]), sb_min_1);+ }++ vst1q_f32(out.as_mut_ptr(), acc_f32_0);+ vst1q_f32(out.as_mut_ptr().add(4), acc_f32_1);+ out+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "i8mm")]+unsafe fn vec_dot_8_q5kx8_q8k_8x8(n: usize, xs: &[BlockQ5Kx8], ys: &[BlockQ8K]) -> [f32; 8] {+ debug_assert!(n.is_multiple_of(QK_K));+ let mut out = [0f32; 8];+ let m4b = vdupq_n_u8(0x0f);+ let mone = vdupq_n_u8(1);+ let mtwo = vdupq_n_u8(2);+ let mut acc_f32_0 = vdupq_n_f32(0.);+ let mut acc_f32_1 = vdupq_n_f32(0.);++ for (q5, q8) in xs.iter().zip(ys.iter()) {+ let sb_scale_0 = vmulq_f32(load_f16x4(q5.d.as_ptr()), vdupq_n_f32(q8.d));+ let sb_scale_1 = vmulq_f32(load_f16x4(q5.d.as_ptr().add(4)), vdupq_n_f32(q8.d));+ let sb_min_0 = vmulq_f32(load_f16x4(q5.dmin.as_ptr()), vdupq_n_f32(q8.d));+ let sb_min_1 = vmulq_f32(load_f16x4(q5.dmin.as_ptr().add(4)), vdupq_n_f32(q8.d));+ let bsums = vpaddq_s16(+ vld1q_s16(q8.bsums.as_ptr()),+ vld1q_s16(q8.bsums.as_ptr().add(8)),+ );+ let mut bsums_arr = [0i16; 8];+ vst1q_s16(bsums_arr.as_mut_ptr(), bsums);+ let mut qh = [[vdupq_n_u8(0); 4]; 4];+ for (cp, qh_cp) in qh.iter_mut().enumerate() {+ qh_cp[0] = vld1q_u8(q5.qh.as_ptr().add(16 * cp));+ qh_cp[1] = vld1q_u8(q5.qh.as_ptr().add(16 * cp + 64));+ qh_cp[2] = vld1q_u8(q5.qh.as_ptr().add(16 * cp + 128));+ qh_cp[3] = vld1q_u8(q5.qh.as_ptr().add(16 * cp + 192));+ }+ let mut bias_acc = [vdupq_n_s32(0); 2];++ for sb in 0..QK_K / 64 {+ let mut acc_lo = [vdupq_n_s32(0); 4];+ let mut acc_hi = [vdupq_n_s32(0); 4];+ let (mins_0, scales_0) = decode_q4kx8_scales(q5.scales.as_ptr().add(sb * 24));+ let (mins_1, scales_1) = decode_q4kx8_scales(q5.scales.as_ptr().add(sb * 24 + 12));+ let q5sb_mins = [mins_0, mins_1];+ let q5sb_scales = [scales_0, scales_1];+ let q8_base = q8.qs.as_ptr().add(sb * 64);+ let q8_qs = [+ vreinterpretq_s8_s64(vld1q_dup_s64(q8_base as *const i64)),+ vreinterpretq_s8_s64(vld1q_dup_s64(q8_base.add(8) as *const i64)),+ vreinterpretq_s8_s64(vld1q_dup_s64(q8_base.add(16) as *const i64)),+ vreinterpretq_s8_s64(vld1q_dup_s64(q8_base.add(24) as *const i64)),+ vreinterpretq_s8_s64(vld1q_dup_s64(q8_base.add(32) as *const i64)),+ vreinterpretq_s8_s64(vld1q_dup_s64(q8_base.add(40) as *const i64)),+ vreinterpretq_s8_s64(vld1q_dup_s64(q8_base.add(48) as *const i64)),+ vreinterpretq_s8_s64(vld1q_dup_s64(q8_base.add(56) as *const i64)),+ ];++ for cp in 0..4 {+ for i in 0..4 {+ let q5_col = vld1q_u8(q5.qs.as_ptr().add(sb * QK_K + i * 64 + 16 * cp));+ let hbit_lo = vandq_u8(qh[cp][i], mone);+ let hbit_hi = vshlq_n_u8(vandq_u8(qh[cp][i], mtwo), 3);+ qh[cp][i] = vshrq_n_u8(qh[cp][i], 2);+ let q5_lo = vreinterpretq_s8_u8(vsliq_n_u8(vandq_u8(q5_col, m4b), hbit_lo, 4));+ let q5_hi = vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q5_col, 4), hbit_hi));+ acc_lo[cp] = sdot_acc(acc_lo[cp], q5_lo, q8_qs[i]);+ acc_hi[cp] = sdot_acc(acc_hi[cp], q5_hi, q8_qs[i + 4]);+ }+ }++ let sumf_0 = vcvtq_f32_s32(vaddq_s32(+ vmulq_s32(+ vmovl_s16(vget_low_s16(q5sb_scales[0])),+ vpaddq_s32(acc_lo[0], acc_lo[1]),+ ),+ vmulq_s32(+ vmovl_s16(vget_low_s16(q5sb_scales[1])),+ vpaddq_s32(acc_hi[0], acc_hi[1]),+ ),+ ));+ acc_f32_0 = vfmaq_f32(acc_f32_0, sb_scale_0, sumf_0);+ let sumf_1 = vcvtq_f32_s32(vaddq_s32(+ vmulq_s32(+ vmovl_s16(vget_high_s16(q5sb_scales[0])),+ vpaddq_s32(acc_lo[2], acc_lo[3]),+ ),+ vmulq_s32(+ vmovl_s16(vget_high_s16(q5sb_scales[1])),+ vpaddq_s32(acc_hi[2], acc_hi[3]),+ ),+ ));+ acc_f32_1 = vfmaq_f32(acc_f32_1, sb_scale_1, sumf_1);++ let bsums_vec_lo = vdup_n_s16(bsums_arr[2 * sb]);+ let bsums_vec_hi = vdup_n_s16(bsums_arr[2 * sb + 1]);+ bias_acc[0] = vmlal_s16(bias_acc[0], bsums_vec_lo, vget_low_s16(q5sb_mins[0]));+ bias_acc[0] = vmlal_s16(bias_acc[0], bsums_vec_hi, vget_low_s16(q5sb_mins[1]));+ bias_acc[1] = vmlal_s16(bias_acc[1], bsums_vec_lo, vget_high_s16(q5sb_mins[0]));+ bias_acc[1] = vmlal_s16(bias_acc[1], bsums_vec_hi, vget_high_s16(q5sb_mins[1]));+ }++ acc_f32_0 = vmlsq_f32(acc_f32_0, vcvtq_f32_s32(bias_acc[0]), sb_min_0);+ acc_f32_1 = vmlsq_f32(acc_f32_1, vcvtq_f32_s32(bias_acc[1]), sb_min_1);+ }++ vst1q_f32(out.as_mut_ptr(), acc_f32_0);+ vst1q_f32(out.as_mut_ptr().add(4), acc_f32_1);+ out+}++#[cfg(target_arch = "aarch64")]+#[inline(always)]+fn vec_dot_8_q5kx8_q8k(n: usize, xs: &[BlockQ5Kx8], ys: &[BlockQ8K]) -> [f32; 8] {+ unsafe {+ if crate::cpu::features::get().i8mm {+ vec_dot_8_q5kx8_q8k_8x8(n, xs, ys)+ } else {+ vec_dot_8_q5kx8_q8k_8x4(n, xs, ys)+ }+ }+}++#[cfg(target_arch = "aarch64")]+fn matmul_q5k_x8_gemv(+ (_m, k, n): (usize, usize, usize),+ lhs: &[f32],+ repacked: &[BlockQ5Kx8],+ dst: &mut [f32],+) -> crate::Result<()> {+ debug_assert!(k.is_multiple_of(QK_K));+ debug_assert!(n.is_multiple_of(8));+ let k_in_blocks = k / QK_K;+ let n_groups = n / 8;++ thread_local! {+ static LHS_Q5K_GEMV_SCRATCH: std::cell::RefCell<Vec<u64>> =+ const { std::cell::RefCell::new(Vec::new()) };+ }++ let elem_size = std::mem::size_of::<BlockQ8K>();+ let required_len = (k_in_blocks * elem_size).div_ceil(8);+ LHS_Q5K_GEMV_SCRATCH.with(|cell| -> crate::Result<()> {+ let mut scratch = cell.borrow_mut();+ if scratch.len() < required_len {+ scratch.resize(required_len, 0);+ }+ let lhs_b: &mut [BlockQ8K] = unsafe {+ std::slice::from_raw_parts_mut(scratch.as_mut_ptr() as *mut BlockQ8K, k_in_blocks)+ };+ <BlockQ8K as super::GgmlType>::from_float(lhs, lhs_b);++ let pool = crate::utils::barrier_pool();+ let lhs_ptr = lhs_b.as_ptr() as usize;+ let repacked_ptr = repacked.as_ptr() as usize;+ let dst_ptr = dst.as_mut_ptr() as usize;+ let x8_block_bytes = std::mem::size_of::<BlockQ5Kx8>();++ // equal-sized gemv units: a static split skips the shared cursor, which costs+ // several us per small decode matmul+ pool.execute_static(n_groups, |range| {+ let lhs_row =+ unsafe { std::slice::from_raw_parts(lhs_ptr as *const BlockQ8K, k_in_blocks) };+ let dst_ptr = dst_ptr as *mut f32;+ for g in range {+ let xs = unsafe {+ std::slice::from_raw_parts(+ (repacked_ptr + g * k_in_blocks * x8_block_bytes) as *const BlockQ5Kx8,+ k_in_blocks,+ )+ };+ let results = vec_dot_8_q5kx8_q8k(k, xs, lhs_row);+ unsafe {+ std::ptr::copy_nonoverlapping(results.as_ptr(), dst_ptr.add(g * 8), 8);+ }+ }+ });++ Ok(())+ })+}++#[cfg(target_arch = "aarch64")]+pub(crate) fn matmul_q5k_x8(+ (m, k, n): (usize, usize, usize),+ lhs: &[f32],+ repacked: &[BlockQ5Kx8],+ dst: &mut [f32],+) -> crate::Result<()> {+ debug_assert!(k.is_multiple_of(QK_K));+ debug_assert!(n.is_multiple_of(8));+ let features = crate::cpu::features::get();+ if features.dotprod && m == 1 {+ return matmul_q5k_x8_gemv((m, k, n), lhs, repacked, dst);+ }+ let use_i8mm = features.i8mm;+ debug_assert!(m.is_multiple_of(4));+ let k_in_blocks = k / QK_K;+ let n_groups = n / 8;+ let row_groups = m / 4;++ thread_local! {+ static LHS_Q5K_X8_SCRATCH: std::cell::RefCell<Vec<u64>> =+ const { std::cell::RefCell::new(Vec::new()) };+ }++ let elem_size = std::mem::size_of::<BlockQ8Kx4>();+ let required_len = (row_groups * k_in_blocks * elem_size).div_ceil(8);+ LHS_Q5K_X8_SCRATCH.with(|cell| -> crate::Result<()> {+ let mut scratch = cell.borrow_mut();+ if scratch.len() < required_len {+ scratch.resize(required_len, 0);+ }+ let lhs_x4: &mut [BlockQ8Kx4] = unsafe {+ std::slice::from_raw_parts_mut(+ scratch.as_mut_ptr() as *mut BlockQ8Kx4,+ row_groups * k_in_blocks,+ )+ };+ for group in 0..row_groups {+ if use_i8mm {+ quantize_q8k_x4_interleaved::<8>(+ lhs,+ k,+ group * 4,+ &mut lhs_x4[group * k_in_blocks..(group + 1) * k_in_blocks],+ );+ } else {+ quantize_q8k_x4_interleaved::<4>(+ lhs,+ k,+ group * 4,+ &mut lhs_x4[group * k_in_blocks..(group + 1) * k_in_blocks],+ );+ }+ }++ let tiles_total = row_groups * n_groups;+ let pool = crate::utils::barrier_pool();+ let lhs_ptr = lhs_x4.as_ptr() as usize;+ let repacked_ptr = repacked.as_ptr() as usize;+ let dst_ptr = dst.as_mut_ptr() as usize;++ pool.execute_chunked(tiles_total, |range| {+ let lhs_ptr = lhs_ptr as *const BlockQ8Kx4;+ let repacked_ptr = repacked_ptr as *const BlockQ5Kx8;+ let dst_ptr = dst_ptr as *mut f32;+ for tile in range {+ let row_group = tile / n_groups;+ let col_group = tile - row_group * n_groups;+ let row = row_group * 4;+ let col = col_group * 8;+ let lhs_tile = unsafe {+ std::slice::from_raw_parts(lhs_ptr.add(row_group * k_in_blocks), k_in_blocks)+ };+ let rhs_tile = unsafe {+ std::slice::from_raw_parts(+ repacked_ptr.add(col_group * k_in_blocks),+ k_in_blocks,+ )+ };+ if use_i8mm {+ unsafe {+ store_q5kx8_4x8_i8mm(dst_ptr, n, row, col, lhs_tile, rhs_tile, k_in_blocks)+ };+ } else {+ unsafe {+ store_q5kx8_4x8(dst_ptr, n, row, col, lhs_tile, rhs_tile, k_in_blocks)+ };+ }+ }+ });++ Ok(())+ })+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "dotprod")]+unsafe fn store_q6kx8_4x8(+ dst: *mut f32,+ n: usize,+ row: usize,+ col: usize,+ lhs: &[BlockQ8Kx4],+ rhs: &[BlockQ6Kx8],+ k_in_blocks: usize,+) {+ let m4b = vdupq_n_u8(0x0f);+ let mask_lo = vdupq_n_u8(0x03);+ let mask_hi = vdupq_n_u8(0x30);+ let m32s = vdupq_n_s8(32);+ let mut acc_f32 = [vdupq_n_f32(0.); 8];++ for b in 0..k_in_blocks {+ let x = &rhs[b];+ let y = &lhs[b];+ let q6_d_0123 = vld1q_f32(+ [+ x.d[0].to_f32(),+ x.d[1].to_f32(),+ x.d[2].to_f32(),+ x.d[3].to_f32(),+ ]+ .as_ptr(),+ );+ let q6_d_4567 = vld1q_f32(+ [+ x.d[4].to_f32(),+ x.d[5].to_f32(),+ x.d[6].to_f32(),+ x.d[7].to_f32(),+ ]+ .as_ptr(),+ );++ let sbd_scale_0123 = [+ vmulq_n_f32(q6_d_0123, y.d[0]),+ vmulq_n_f32(q6_d_0123, y.d[1]),+ vmulq_n_f32(q6_d_0123, y.d[2]),+ vmulq_n_f32(q6_d_0123, y.d[3]),+ ];+ let sbd_scale_4567 = [+ vmulq_n_f32(q6_d_4567, y.d[0]),+ vmulq_n_f32(q6_d_4567, y.d[1]),+ vmulq_n_f32(q6_d_4567, y.d[2]),+ vmulq_n_f32(q6_d_4567, y.d[3]),+ ];++ let mut acc_s32 = [vdupq_n_s32(0); 8];+ for half in 0..2 {+ let ql_base = x.ql.as_ptr().add(half * 512);+ let qh_base = x.qh.as_ptr().add(half * 256);+ for sb in 0..QK_K / 64 {+ let mut acc_lo = [vdupq_n_s32(0); 8];+ let mut acc_hi = [vdupq_n_s32(0); 8];+ let q8_base_l = y.qs.as_ptr().add(half * 512 + sb * 64);+ let q8_base_h = y.qs.as_ptr().add(half * 512 + 256 + sb * 64);+ let ql_off_base = sb * QK_K / 2;+ let qh_off_base = ql_off_base & 255;++ for k_idx in 0..4 {+ let q8_l = vld1q_s8(q8_base_l.add(16 * k_idx));+ let q8_h = vld1q_s8(q8_base_h.add(16 * k_idx));+ let q6_ql_0123 = vld1q_u8(ql_base.add(ql_off_base + k_idx * 32));+ let q6_ql_4567 = vld1q_u8(ql_base.add(ql_off_base + k_idx * 32 + 16));+ let mut q6_qh_0123 = vld1q_u8(qh_base.add(qh_off_base + k_idx * 32));+ let mut q6_qh_4567 = vld1q_u8(qh_base.add(qh_off_base + k_idx * 32 + 16));+ if sb > 1 {+ q6_qh_0123 = vshrq_n_u8(q6_qh_0123, 2);+ q6_qh_4567 = vshrq_n_u8(q6_qh_4567, 2);+ }++ let hbit_lo_0123 = vandq_u8(q6_qh_0123, mask_lo);+ let hbit_hi_0123 = vandq_u8(q6_qh_0123, mask_hi);+ let hbit_lo_4567 = vandq_u8(q6_qh_4567, mask_lo);+ let hbit_hi_4567 = vandq_u8(q6_qh_4567, mask_hi);++ let q6_0123_lo = vsubq_s8(+ vreinterpretq_s8_u8(vorrq_u8(+ vandq_u8(q6_ql_0123, m4b),+ vshlq_n_u8(hbit_lo_0123, 4),+ )),+ m32s,+ );+ let q6_0123_hi = vsubq_s8(+ vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q6_ql_0123, 4), hbit_hi_0123)),+ m32s,+ );++ acc_lo[0] = sdot_laneq_s32::<0>(acc_lo[0], q6_0123_lo, q8_l);+ acc_lo[1] = sdot_laneq_s32::<1>(acc_lo[1], q6_0123_lo, q8_l);+ acc_lo[2] = sdot_laneq_s32::<2>(acc_lo[2], q6_0123_lo, q8_l);+ acc_lo[3] = sdot_laneq_s32::<3>(acc_lo[3], q6_0123_lo, q8_l);++ acc_hi[0] = sdot_laneq_s32::<0>(acc_hi[0], q6_0123_hi, q8_h);+ acc_hi[1] = sdot_laneq_s32::<1>(acc_hi[1], q6_0123_hi, q8_h);+ acc_hi[2] = sdot_laneq_s32::<2>(acc_hi[2], q6_0123_hi, q8_h);+ acc_hi[3] = sdot_laneq_s32::<3>(acc_hi[3], q6_0123_hi, q8_h);++ let q6_4567_lo = vsubq_s8(+ vreinterpretq_s8_u8(vorrq_u8(+ vandq_u8(q6_ql_4567, m4b),+ vshlq_n_u8(hbit_lo_4567, 4),+ )),+ m32s,+ );+ let q6_4567_hi = vsubq_s8(+ vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q6_ql_4567, 4), hbit_hi_4567)),+ m32s,+ );++ acc_lo[4] = sdot_laneq_s32::<0>(acc_lo[4], q6_4567_lo, q8_l);+ acc_lo[5] = sdot_laneq_s32::<1>(acc_lo[5], q6_4567_lo, q8_l);+ acc_lo[6] = sdot_laneq_s32::<2>(acc_lo[6], q6_4567_lo, q8_l);+ acc_lo[7] = sdot_laneq_s32::<3>(acc_lo[7], q6_4567_lo, q8_l);++ acc_hi[4] = sdot_laneq_s32::<0>(acc_hi[4], q6_4567_hi, q8_h);+ acc_hi[5] = sdot_laneq_s32::<1>(acc_hi[5], q6_4567_hi, q8_h);+ acc_hi[6] = sdot_laneq_s32::<2>(acc_hi[6], q6_4567_hi, q8_h);+ acc_hi[7] = sdot_laneq_s32::<3>(acc_hi[7], q6_4567_hi, q8_h);+ }++ let scale_idx_l = half * 8 + sb;+ let scale_idx_h = half * 8 + sb + 4;+ for g in 0..2 {+ let base_l = scale_idx_l * 8 + g * 4;+ let base_h = scale_idx_h * 8 + g * 4;+ let scale_vec_l = vld1q_s32(+ [+ x.scales[base_l] as i32,+ x.scales[base_l + 1] as i32,+ x.scales[base_l + 2] as i32,+ x.scales[base_l + 3] as i32,+ ]+ .as_ptr(),+ );+ let scale_vec_h = vld1q_s32(+ [+ x.scales[base_h] as i32,+ x.scales[base_h + 1] as i32,+ x.scales[base_h + 2] as i32,+ x.scales[base_h + 3] as i32,+ ]+ .as_ptr(),+ );+ let acc_offset = g * 4;+ for row in 0..4 {+ let idx = row * 2 + g;+ acc_s32[idx] =+ vmlaq_s32(acc_s32[idx], acc_lo[acc_offset + row], scale_vec_l);+ acc_s32[idx] =+ vmlaq_s32(acc_s32[idx], acc_hi[acc_offset + row], scale_vec_h);+ }+ }+ }+ }++ for r in 0..4 {+ let idx0 = 2 * r;+ let idx1 = 2 * r + 1;+ acc_f32[idx0] = vmlaq_f32(+ acc_f32[idx0],+ vcvtq_f32_s32(acc_s32[idx0]),+ sbd_scale_0123[r],+ );+ acc_f32[idx1] = vmlaq_f32(+ acc_f32[idx1],+ vcvtq_f32_s32(acc_s32[idx1]),+ sbd_scale_4567[r],+ );+ }+ }++ for r in 0..4 {+ vst1q_f32(dst.add((row + r) * n + col), acc_f32[2 * r]);+ vst1q_f32(dst.add((row + r) * n + col + 4), acc_f32[2 * r + 1]);+ }+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "i8mm")]+unsafe fn store_q6kx8_4x8_i8mm(+ dst: *mut f32,+ n: usize,+ row: usize,+ col: usize,+ lhs: &[BlockQ8Kx4],+ rhs: &[BlockQ6Kx8],+ k_in_blocks: usize,+) {+ let m4b = vdupq_n_u8(0x0f);+ let mask_lo = vdupq_n_u8(0x03);+ let mask_hi = vdupq_n_u8(0x30);+ let m32s = vdupq_n_s8(32);+ let mut acc_f32 = [vdupq_n_f32(0.); 8];++ for b in 0..k_in_blocks {+ let x = &rhs[b];+ let y = &lhs[b];+ let mut acc = [vdupq_n_s32(0); 8];++ for half in 0..2 {+ let ql_base = x.ql.as_ptr().add(half * 512);+ let qh_base = x.qh.as_ptr().add(half * 256);+ for sb in 0..QK_K / 64 {+ let q8_base_l = y.qs.as_ptr().add(half * 512 + sb * 64);+ let q8_base_h = y.qs.as_ptr().add(half * 512 + 256 + sb * 64);+ let q8_l_01 = [vld1q_s8(q8_base_l), vld1q_s8(q8_base_l.add(32))];+ let q8_l_23 = [vld1q_s8(q8_base_l.add(16)), vld1q_s8(q8_base_l.add(48))];+ let q8_h_01 = [vld1q_s8(q8_base_h), vld1q_s8(q8_base_h.add(32))];+ let q8_h_23 = [vld1q_s8(q8_base_h.add(16)), vld1q_s8(q8_base_h.add(48))];++ let ql_off_base = sb * QK_K / 2;+ let qh_off_base = ql_off_base & 255;++ for cp in 0..4 {+ let mut q6_qh_0 = vld1q_u8(qh_base.add(qh_off_base + 16 * cp));+ let mut q6_qh_1 = vld1q_u8(qh_base.add(qh_off_base + 64 + 16 * cp));+ if sb > 1 {+ q6_qh_0 = vshrq_n_u8(q6_qh_0, 2);+ q6_qh_1 = vshrq_n_u8(q6_qh_1, 2);+ }++ let q6_ql_0 = vld1q_u8(ql_base.add(ql_off_base + 16 * cp));+ let q6_ql_1 = vld1q_u8(ql_base.add(ql_off_base + 64 + 16 * cp));++ let q6_l0 = vsubq_s8(+ vreinterpretq_s8_u8(vorrq_u8(+ vandq_u8(q6_ql_0, m4b),+ vshlq_n_u8(vandq_u8(q6_qh_0, mask_lo), 4),+ )),+ m32s,+ );+ let q6_l1 = vsubq_s8(+ vreinterpretq_s8_u8(vorrq_u8(+ vandq_u8(q6_ql_1, m4b),+ vshlq_n_u8(vandq_u8(q6_qh_1, mask_lo), 4),+ )),+ m32s,+ );+ let q6_h0 = vsubq_s8(+ vreinterpretq_s8_u8(vorrq_u8(+ vshrq_n_u8(q6_ql_0, 4),+ vandq_u8(q6_qh_0, mask_hi),+ )),+ m32s,+ );+ let q6_h1 = vsubq_s8(+ vreinterpretq_s8_u8(vorrq_u8(+ vshrq_n_u8(q6_ql_1, 4),+ vandq_u8(q6_qh_1, mask_hi),+ )),+ m32s,+ );++ let mut sb_acc_0l = smmla_s32(vdupq_n_s32(0), q6_l0, q8_l_01[0]);+ sb_acc_0l = smmla_s32(sb_acc_0l, q6_l1, q8_l_01[1]);+ let mut sb_acc_0h = smmla_s32(vdupq_n_s32(0), q6_h0, q8_h_01[0]);+ sb_acc_0h = smmla_s32(sb_acc_0h, q6_h1, q8_h_01[1]);+ let mut sb_acc_1l = smmla_s32(vdupq_n_s32(0), q6_l0, q8_l_23[0]);+ sb_acc_1l = smmla_s32(sb_acc_1l, q6_l1, q8_l_23[1]);+ let mut sb_acc_1h = smmla_s32(vdupq_n_s32(0), q6_h0, q8_h_23[0]);+ sb_acc_1h = smmla_s32(sb_acc_1h, q6_h1, q8_h_23[1]);++ let scale_idx_l = half * 8 + sb;+ let scale_idx_h = half * 8 + sb + 4;+ let s0 = x.scales[scale_idx_l * 8 + cp * 2] as i32;+ let s1 = x.scales[scale_idx_l * 8 + cp * 2 + 1] as i32;+ let s2 = x.scales[scale_idx_h * 8 + cp * 2] as i32;+ let s3 = x.scales[scale_idx_h * 8 + cp * 2 + 1] as i32;+ let scale_vec_l = vcombine_s32(vdup_n_s32(s0), vdup_n_s32(s1));+ let scale_vec_h = vcombine_s32(vdup_n_s32(s2), vdup_n_s32(s3));+ acc[cp] = vmlaq_s32(acc[cp], sb_acc_0l, scale_vec_l);+ acc[cp] = vmlaq_s32(acc[cp], sb_acc_0h, scale_vec_h);+ acc[cp + 4] = vmlaq_s32(acc[cp + 4], sb_acc_1l, scale_vec_l);+ acc[cp + 4] = vmlaq_s32(acc[cp + 4], sb_acc_1h, scale_vec_h);+ }+ }+ }++ for item in &mut acc {+ let aux = vzip_s32(vget_low_s32(*item), vget_high_s32(*item));+ *item = vcombine_s32(aux.0, aux.1);+ }+ let reorder_acc = [+ vcombine_s32(vget_low_s32(acc[0]), vget_low_s32(acc[1])),+ vcombine_s32(vget_low_s32(acc[2]), vget_low_s32(acc[3])),+ vcombine_s32(vget_high_s32(acc[0]), vget_high_s32(acc[1])),+ vcombine_s32(vget_high_s32(acc[2]), vget_high_s32(acc[3])),+ vcombine_s32(vget_low_s32(acc[4]), vget_low_s32(acc[5])),+ vcombine_s32(vget_low_s32(acc[6]), vget_low_s32(acc[7])),+ vcombine_s32(vget_high_s32(acc[4]), vget_high_s32(acc[5])),+ vcombine_s32(vget_high_s32(acc[6]), vget_high_s32(acc[7])),+ ];++ for r in 0..4 {+ for g in 0..2 {+ let q6_d = [+ x.d[g * 4].to_f32(),+ x.d[g * 4 + 1].to_f32(),+ x.d[g * 4 + 2].to_f32(),+ x.d[g * 4 + 3].to_f32(),+ ];+ let scale = vmulq_n_f32(vld1q_f32(q6_d.as_ptr()), y.d[r]);+ let idx = 2 * r + g;+ acc_f32[idx] = vmlaq_f32(acc_f32[idx], vcvtq_f32_s32(reorder_acc[idx]), scale);+ }+ }+ }++ for r in 0..4 {+ vst1q_f32(dst.add((row + r) * n + col), acc_f32[2 * r]);+ vst1q_f32(dst.add((row + r) * n + col + 4), acc_f32[2 * r + 1]);+ }+}++#[cfg(target_arch = "aarch64")]+#[inline(always)]+unsafe fn load_f16x4(ptr: *const f16) -> float32x4_t {+ let raw = vld1_u64(ptr as *const u64);+ let mut result: float32x4_t;+ core::arch::asm!(+ "fcvtl {out:v}.4s, {inp:v}.4h",+ inp = in(vreg) raw,+ out = out(vreg) result,+ options(nostack, nomem),+ );+ result+}++#[cfg(target_arch = "aarch64")]+#[inline(always)]+unsafe fn load_i32x2(a: i32, b: i32) -> int32x2_t {+ vld1_s32([a, b].as_ptr())+}++#[cfg(target_arch = "aarch64")]+#[inline(always)]+unsafe fn load_i32x4(a: i32, b: i32, c: i32, d: i32) -> int32x4_t {+ vld1q_s32([a, b, c, d].as_ptr())+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "i8mm")]+unsafe fn vec_dot_8_q6kx8_q8k_8x8(n: usize, xs: &[BlockQ6Kx8], ys: &[BlockQ8K]) -> [f32; 8] {+ debug_assert!(n.is_multiple_of(QK_K));+ let mut out = [0f32; 8];+ let m4b = vdupq_n_u8(0x0f);+ let mask_lo = vdupq_n_u8(0x03);+ let mask_hi = vdupq_n_u8(0x30);+ let mut acc_f32_0 = vdupq_n_f32(0.0);+ let mut acc_f32_1 = vdupq_n_f32(0.0);++ for (q6, q8) in xs.iter().zip(ys.iter()) {+ let sb_scale_0 = vmulq_f32(load_f16x4(q6.d.as_ptr()), vdupq_n_f32(q8.d));+ let sb_scale_1 = vmulq_f32(load_f16x4(q6.d.as_ptr().add(4)), vdupq_n_f32(q8.d));+ let mut acc = [vdup_n_s32(0); 4];+ let mut q6_scales = [0i16; 16 * 8];++ for i in 0..16 {+ let scales = vmovl_s8(vld1_s8(q6.scales.as_ptr().add(i * 8)));+ vst1q_s16(q6_scales.as_mut_ptr().add(i * 8), scales);+ }++ let mut bias = [0i32; 8];+ for i in 0..16 {+ let bsum = q8.bsums[i] as i32;+ for c in 0..8 {+ bias[c] += bsum * q6_scales[i * 8 + c] as i32;+ }+ }++ for half in 0..2 {+ let ql_base = q6.ql.as_ptr().add(half * 512);+ let qh_base = q6.qh.as_ptr().add(half * 256);+ for sb in 0..QK_K / 64 {+ let q8_base_l = q8.qs.as_ptr().add(half * 128 + sb * 16);+ let q8_base_h = q8_base_l.add(64);+ let q8_l = [+ vreinterpretq_s8_s64(vld1q_dup_s64(q8_base_l as *const i64)),+ vreinterpretq_s8_s64(vld1q_dup_s64(q8_base_l.add(8) as *const i64)),+ ];+ let q8_h = [+ vreinterpretq_s8_s64(vld1q_dup_s64(q8_base_h as *const i64)),+ vreinterpretq_s8_s64(vld1q_dup_s64(q8_base_h.add(8) as *const i64)),+ ];+ let ql_off_base = sb * QK_K / 2;+ let qh_off_base = ql_off_base & 255;+ let q6_ql_0 = vld1q_u8_x4(ql_base.add(ql_off_base));+ let q6_ql_1 = vld1q_u8_x4(ql_base.add(ql_off_base + 64));+ let q6_qh_0_raw = vld1q_u8_x4(qh_base.add(qh_off_base));+ let q6_qh_1_raw = vld1q_u8_x4(qh_base.add(qh_off_base + 64));+ let q6_qh_0 = if sb > 1 {+ (+ vshrq_n_u8(q6_qh_0_raw.0, 2),+ vshrq_n_u8(q6_qh_0_raw.1, 2),+ vshrq_n_u8(q6_qh_0_raw.2, 2),+ vshrq_n_u8(q6_qh_0_raw.3, 2),+ )+ } else {+ (q6_qh_0_raw.0, q6_qh_0_raw.1, q6_qh_0_raw.2, q6_qh_0_raw.3)+ };+ let q6_qh_1 = if sb > 1 {+ (+ vshrq_n_u8(q6_qh_1_raw.0, 2),+ vshrq_n_u8(q6_qh_1_raw.1, 2),+ vshrq_n_u8(q6_qh_1_raw.2, 2),+ vshrq_n_u8(q6_qh_1_raw.3, 2),+ )+ } else {+ (q6_qh_1_raw.0, q6_qh_1_raw.1, q6_qh_1_raw.2, q6_qh_1_raw.3)+ };+ let q6_ql = [+ q6_ql_0.0, q6_ql_0.1, q6_ql_0.2, q6_ql_0.3, q6_ql_1.0, q6_ql_1.1, q6_ql_1.2,+ q6_ql_1.3,+ ];+ let q6_qh = [+ q6_qh_0.0, q6_qh_0.1, q6_qh_0.2, q6_qh_0.3, q6_qh_1.0, q6_qh_1.1, q6_qh_1.2,+ q6_qh_1.3,+ ];++ for cp in 0..4 {+ let q6_l0 = vreinterpretq_s8_u8(vorrq_u8(+ vandq_u8(q6_ql[cp], m4b),+ vshlq_n_u8(vandq_u8(q6_qh[cp], mask_lo), 4),+ ));+ let q6_l1 = vreinterpretq_s8_u8(vorrq_u8(+ vandq_u8(q6_ql[cp + 4], m4b),+ vshlq_n_u8(vandq_u8(q6_qh[cp + 4], mask_lo), 4),+ ));+ let q6_h0 = vreinterpretq_s8_u8(vorrq_u8(+ vshrq_n_u8(q6_ql[cp], 4),+ vandq_u8(q6_qh[cp], mask_hi),+ ));+ let q6_h1 = vreinterpretq_s8_u8(vorrq_u8(+ vshrq_n_u8(q6_ql[cp + 4], 4),+ vandq_u8(q6_qh[cp + 4], mask_hi),+ ));+ let mut sb_acc_l = vdupq_n_s32(0);+ sb_acc_l = sdot_acc(sb_acc_l, q6_l0, q8_l[0]);+ sb_acc_l = sdot_acc(sb_acc_l, q6_l1, q8_l[1]);+ let mut sb_acc_h = vdupq_n_s32(0);+ sb_acc_h = sdot_acc(sb_acc_h, q6_h0, q8_h[0]);+ sb_acc_h = sdot_acc(sb_acc_h, q6_h1, q8_h[1]);+ let sum_l = vpadd_s32(vget_low_s32(sb_acc_l), vget_high_s32(sb_acc_l));+ let sum_h = vpadd_s32(vget_low_s32(sb_acc_h), vget_high_s32(sb_acc_h));+ let scale_idx_l = half * 8 + sb;+ let scale_idx_h = half * 8 + sb + 4;+ let scale_l = load_i32x2(+ q6_scales[scale_idx_l * 8 + cp * 2] as i32,+ q6_scales[scale_idx_l * 8 + cp * 2 + 1] as i32,+ );+ let scale_h = load_i32x2(+ q6_scales[scale_idx_h * 8 + cp * 2] as i32,+ q6_scales[scale_idx_h * 8 + cp * 2 + 1] as i32,+ );+ acc[cp] = vmla_s32(acc[cp], sum_l, scale_l);+ acc[cp] = vmla_s32(acc[cp], sum_h, scale_h);+ }+ }+ }++ acc[0] = vsub_s32(acc[0], load_i32x2(bias[0] << 5, bias[1] << 5));+ acc[1] = vsub_s32(acc[1], load_i32x2(bias[2] << 5, bias[3] << 5));+ acc[2] = vsub_s32(acc[2], load_i32x2(bias[4] << 5, bias[5] << 5));+ acc[3] = vsub_s32(acc[3], load_i32x2(bias[6] << 5, bias[7] << 5));+ let w_01 = vmul_f32(vcvt_f32_s32(acc[0]), vget_low_f32(sb_scale_0));+ let w_23 = vmul_f32(vcvt_f32_s32(acc[1]), vget_high_f32(sb_scale_0));+ let w_45 = vmul_f32(vcvt_f32_s32(acc[2]), vget_low_f32(sb_scale_1));+ let w_67 = vmul_f32(vcvt_f32_s32(acc[3]), vget_high_f32(sb_scale_1));+ acc_f32_0 = vaddq_f32(acc_f32_0, vcombine_f32(w_01, w_23));+ acc_f32_1 = vaddq_f32(acc_f32_1, vcombine_f32(w_45, w_67));+ }++ vst1q_f32(out.as_mut_ptr(), acc_f32_0);+ vst1q_f32(out.as_mut_ptr().add(4), acc_f32_1);+ out+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "dotprod")]+unsafe fn vec_dot_8_q6kx8_q8k_8x4(n: usize, xs: &[BlockQ6Kx8], ys: &[BlockQ8K]) -> [f32; 8] {+ debug_assert!(n.is_multiple_of(QK_K));+ let mut out = [0f32; 8];+ let m4b = vdupq_n_u8(0x0f);+ let mask_lo = vdupq_n_u8(0x03);+ let mask_hi = vdupq_n_u8(0x30);+ let mut acc_f32_0 = vdupq_n_f32(0.0);+ let mut acc_f32_1 = vdupq_n_f32(0.0);++ for (q6, q8) in xs.iter().zip(ys.iter()) {+ let sb_scale_0 = vmulq_f32(load_f16x4(q6.d.as_ptr()), vdupq_n_f32(q8.d));+ let sb_scale_1 = vmulq_f32(load_f16x4(q6.d.as_ptr().add(4)), vdupq_n_f32(q8.d));+ let mut acc = [vdupq_n_s32(0); 2];+ let mut q6_scales = [0i16; 16 * 8];++ for i in 0..16 {+ let scales = vmovl_s8(vld1_s8(q6.scales.as_ptr().add(i * 8)));+ vst1q_s16(q6_scales.as_mut_ptr().add(i * 8), scales);+ }++ let mut bias = [0i32; 8];+ for i in 0..16 {+ let bsum = q8.bsums[i] as i32;+ for c in 0..8 {+ bias[c] += bsum * q6_scales[i * 8 + c] as i32;+ }+ }++ for half in 0..2 {+ let ql_base = q6.ql.as_ptr().add(half * 512);+ let qh_base = q6.qh.as_ptr().add(half * 256);+ for sb in 0..QK_K / 64 {+ let q8_base_l = q8.qs.as_ptr().add(half * 128 + sb * 16);+ let q8_base_h = q8_base_l.add(64);+ let q8_l = [+ vreinterpretq_s8_s32(vld1q_dup_s32(q8_base_l as *const i32)),+ vreinterpretq_s8_s32(vld1q_dup_s32(q8_base_l.add(4) as *const i32)),+ vreinterpretq_s8_s32(vld1q_dup_s32(q8_base_l.add(8) as *const i32)),+ vreinterpretq_s8_s32(vld1q_dup_s32(q8_base_l.add(12) as *const i32)),+ ];+ let q8_h = [+ vreinterpretq_s8_s32(vld1q_dup_s32(q8_base_h as *const i32)),+ vreinterpretq_s8_s32(vld1q_dup_s32(q8_base_h.add(4) as *const i32)),+ vreinterpretq_s8_s32(vld1q_dup_s32(q8_base_h.add(8) as *const i32)),+ vreinterpretq_s8_s32(vld1q_dup_s32(q8_base_h.add(12) as *const i32)),+ ];+ let ql_off_base = sb * QK_K / 2;+ let qh_off_base = ql_off_base & 255;+ let q6_ql_0 = vld1q_u8_x4(ql_base.add(ql_off_base));+ let q6_ql_1 = vld1q_u8_x4(ql_base.add(ql_off_base + 64));+ let q6_qh_0_raw = vld1q_u8_x4(qh_base.add(qh_off_base));+ let q6_qh_1_raw = vld1q_u8_x4(qh_base.add(qh_off_base + 64));+ let q6_qh_0 = if sb > 1 {+ (+ vshrq_n_u8(q6_qh_0_raw.0, 2),+ vshrq_n_u8(q6_qh_0_raw.1, 2),+ vshrq_n_u8(q6_qh_0_raw.2, 2),+ vshrq_n_u8(q6_qh_0_raw.3, 2),+ )+ } else {+ (q6_qh_0_raw.0, q6_qh_0_raw.1, q6_qh_0_raw.2, q6_qh_0_raw.3)+ };+ let q6_qh_1 = if sb > 1 {+ (+ vshrq_n_u8(q6_qh_1_raw.0, 2),+ vshrq_n_u8(q6_qh_1_raw.1, 2),+ vshrq_n_u8(q6_qh_1_raw.2, 2),+ vshrq_n_u8(q6_qh_1_raw.3, 2),+ )+ } else {+ (q6_qh_1_raw.0, q6_qh_1_raw.1, q6_qh_1_raw.2, q6_qh_1_raw.3)+ };+ let q6_ql = [+ q6_ql_0.0, q6_ql_0.1, q6_ql_0.2, q6_ql_0.3, q6_ql_1.0, q6_ql_1.1, q6_ql_1.2,+ q6_ql_1.3,+ ];+ let q6_qh = [+ q6_qh_0.0, q6_qh_0.1, q6_qh_0.2, q6_qh_0.3, q6_qh_1.0, q6_qh_1.1, q6_qh_1.2,+ q6_qh_1.3,+ ];++ for g in 0..2 {+ let mut sb_acc_l = vdupq_n_s32(0);+ let mut sb_acc_h = vdupq_n_s32(0);+ for chunk in 0..4 {+ let idx = chunk * 2 + g;+ let q6_l = vreinterpretq_s8_u8(vorrq_u8(+ vandq_u8(q6_ql[idx], m4b),+ vshlq_n_u8(vandq_u8(q6_qh[idx], mask_lo), 4),+ ));+ let q6_h = vreinterpretq_s8_u8(vorrq_u8(+ vshrq_n_u8(q6_ql[idx], 4),+ vandq_u8(q6_qh[idx], mask_hi),+ ));+ sb_acc_l = sdot_acc(sb_acc_l, q6_l, q8_l[chunk]);+ sb_acc_h = sdot_acc(sb_acc_h, q6_h, q8_h[chunk]);+ }++ let scale_idx_l = half * 8 + sb;+ let scale_idx_h = half * 8 + sb + 4;+ let scale_l = load_i32x4(+ q6_scales[scale_idx_l * 8 + g * 4] as i32,+ q6_scales[scale_idx_l * 8 + g * 4 + 1] as i32,+ q6_scales[scale_idx_l * 8 + g * 4 + 2] as i32,+ q6_scales[scale_idx_l * 8 + g * 4 + 3] as i32,+ );+ let scale_h = load_i32x4(+ q6_scales[scale_idx_h * 8 + g * 4] as i32,+ q6_scales[scale_idx_h * 8 + g * 4 + 1] as i32,+ q6_scales[scale_idx_h * 8 + g * 4 + 2] as i32,+ q6_scales[scale_idx_h * 8 + g * 4 + 3] as i32,+ );+ acc[g] = vmlaq_s32(acc[g], sb_acc_l, scale_l);+ acc[g] = vmlaq_s32(acc[g], sb_acc_h, scale_h);+ }+ }+ }++ acc[0] = vsubq_s32(+ acc[0],+ load_i32x4(bias[0] << 5, bias[1] << 5, bias[2] << 5, bias[3] << 5),+ );+ acc[1] = vsubq_s32(+ acc[1],+ load_i32x4(bias[4] << 5, bias[5] << 5, bias[6] << 5, bias[7] << 5),+ );+ acc_f32_0 = vfmaq_f32(acc_f32_0, vcvtq_f32_s32(acc[0]), sb_scale_0);+ acc_f32_1 = vfmaq_f32(acc_f32_1, vcvtq_f32_s32(acc[1]), sb_scale_1);+ }++ vst1q_f32(out.as_mut_ptr(), acc_f32_0);+ vst1q_f32(out.as_mut_ptr().add(4), acc_f32_1);+ out+}++#[cfg(target_arch = "aarch64")]+#[inline(always)]+pub(crate) fn vec_dot_8_q6kx8_q8k(n: usize, xs: &[BlockQ6Kx8], ys: &[BlockQ8K]) -> [f32; 8] {+ unsafe {+ if crate::cpu::features::get().i8mm {+ vec_dot_8_q6kx8_q8k_8x8(n, xs, ys)+ } else {+ vec_dot_8_q6kx8_q8k_8x4(n, xs, ys)+ }+ }+}++#[cfg(target_arch = "aarch64")]+fn matmul_q6k_x8_gemv(+ (m, k, n): (usize, usize, usize),+ lhs: &[f32],+ repacked: &[BlockQ6Kx8],+ dst: &mut [f32],+) -> crate::Result<()> {+ debug_assert_eq!(m, 1);+ debug_assert!(k.is_multiple_of(QK_K));+ debug_assert!(n.is_multiple_of(8));+ let k_in_blocks = k / QK_K;+ let n_groups = n / 8;++ thread_local! {+ static LHS_Q6K_GEMV_SCRATCH: std::cell::RefCell<Vec<u64>> =+ const { std::cell::RefCell::new(Vec::new()) };+ }++ let elem_size = std::mem::size_of::<BlockQ8K>();+ let required_len = (k_in_blocks * elem_size).div_ceil(8);+ LHS_Q6K_GEMV_SCRATCH.with(|cell| -> crate::Result<()> {+ let mut scratch = cell.borrow_mut();+ if scratch.len() < required_len {+ scratch.resize(required_len, 0);+ }+ let lhs_b: &mut [BlockQ8K] = unsafe {+ std::slice::from_raw_parts_mut(scratch.as_mut_ptr() as *mut BlockQ8K, k_in_blocks)+ };+ <BlockQ8K as super::GgmlType>::from_float(lhs, lhs_b);++ let pool = crate::utils::barrier_pool();+ let lhs_ptr = lhs_b.as_ptr() as usize;+ let repacked_ptr = repacked.as_ptr() as usize;+ let dst_ptr = dst.as_mut_ptr() as usize;+ let x8_block_bytes = std::mem::size_of::<BlockQ6Kx8>();++ // equal-sized gemv units: a static split skips the shared cursor, which costs+ // several us per small decode matmul+ pool.execute_static(n_groups, |range| {+ let lhs_row =+ unsafe { std::slice::from_raw_parts(lhs_ptr as *const BlockQ8K, k_in_blocks) };+ let dst_ptr = dst_ptr as *mut f32;+ for g in range {+ let xs = unsafe {+ std::slice::from_raw_parts(+ (repacked_ptr + g * k_in_blocks * x8_block_bytes) as *const BlockQ6Kx8,+ k_in_blocks,+ )+ };+ let results = vec_dot_8_q6kx8_q8k(k, xs, lhs_row);+ unsafe {+ std::ptr::copy_nonoverlapping(results.as_ptr(), dst_ptr.add(g * 8), 8);+ }+ }+ });++ Ok(())+ })+}++#[cfg(target_arch = "aarch64")]+pub(crate) fn matmul_q6k_x8(+ (m, k, n): (usize, usize, usize),+ lhs: &[f32],+ repacked: &[BlockQ6Kx8],+ dst: &mut [f32],+) -> crate::Result<()> {+ debug_assert!(k.is_multiple_of(QK_K));+ debug_assert!(n.is_multiple_of(8));+ let features = crate::cpu::features::get();+ if features.dotprod && m == 1 {+ return matmul_q6k_x8_gemv((m, k, n), lhs, repacked, dst);+ }+ let use_i8mm = features.i8mm;+ debug_assert!(m.is_multiple_of(4));+ let k_in_blocks = k / QK_K;+ let n_groups = n / 8;+ let row_groups = m / 4;++ thread_local! {+ static LHS_Q6K_X8_SCRATCH: std::cell::RefCell<Vec<u64>> =+ const { std::cell::RefCell::new(Vec::new()) };+ }++ let elem_size = std::mem::size_of::<BlockQ8Kx4>();+ let required_len = (row_groups * k_in_blocks * elem_size).div_ceil(8);+ LHS_Q6K_X8_SCRATCH.with(|cell| -> crate::Result<()> {+ let mut scratch = cell.borrow_mut();+ if scratch.len() < required_len {+ scratch.resize(required_len, 0);+ }+ let lhs_x4: &mut [BlockQ8Kx4] = unsafe {+ std::slice::from_raw_parts_mut(+ scratch.as_mut_ptr() as *mut BlockQ8Kx4,+ row_groups * k_in_blocks,+ )+ };+ for group in 0..row_groups {+ if use_i8mm {+ quantize_q8k_x4_interleaved::<8>(+ lhs,+ k,+ group * 4,+ &mut lhs_x4[group * k_in_blocks..(group + 1) * k_in_blocks],+ );+ } else {+ quantize_q8k_x4_interleaved::<4>(+ lhs,+ k,+ group * 4,+ &mut lhs_x4[group * k_in_blocks..(group + 1) * k_in_blocks],+ );+ }+ }++ let tiles_total = row_groups * n_groups;+ let pool = crate::utils::barrier_pool();+ let lhs_ptr = lhs_x4.as_ptr() as usize;+ let repacked_ptr = repacked.as_ptr() as usize;+ let dst_ptr = dst.as_mut_ptr() as usize;++ pool.execute_chunked(tiles_total, |range| {+ let lhs_ptr = lhs_ptr as *const BlockQ8Kx4;+ let repacked_ptr = repacked_ptr as *const BlockQ6Kx8;+ let dst_ptr = dst_ptr as *mut f32;+ for tile in range {+ let row_group = tile / n_groups;+ let col_group = tile - row_group * n_groups;+ let row = row_group * 4;+ let col = col_group * 8;+ let lhs_tile = unsafe {+ std::slice::from_raw_parts(lhs_ptr.add(row_group * k_in_blocks), k_in_blocks)+ };+ let rhs_tile = unsafe {+ std::slice::from_raw_parts(+ repacked_ptr.add(col_group * k_in_blocks),+ k_in_blocks,+ )+ };+ if use_i8mm {+ unsafe {+ store_q6kx8_4x8_i8mm(dst_ptr, n, row, col, lhs_tile, rhs_tile, k_in_blocks)+ };+ } else {+ unsafe {+ store_q6kx8_4x8(dst_ptr, n, row, col, lhs_tile, rhs_tile, k_in_blocks)+ };+ }+ }+ });++ Ok(())+ })+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "dotprod")]+unsafe fn store_q8_0x4_4x4(+ dst: *mut f32,+ n: usize,+ row: usize,+ col: usize,+ lhs: &[BlockQ8_0x4],+ rhs: &[BlockQ8_0x4],+ k_in_blocks: usize,+) {+ let mut sum0 = vdupq_n_f32(0.);+ let mut sum1 = vdupq_n_f32(0.);+ let mut sum2 = vdupq_n_f32(0.);+ let mut sum3 = vdupq_n_f32(0.);++ for i in 0..k_in_blocks {+ let x = &rhs[i];+ let y = &lhs[i];+ let bd = [+ x.d[0].to_f32(),+ x.d[1].to_f32(),+ x.d[2].to_f32(),+ x.d[3].to_f32(),+ ];+ let bd = vld1q_f32(bd.as_ptr());+ let ad = [+ y.d[0].to_f32(),+ y.d[1].to_f32(),+ y.d[2].to_f32(),+ y.d[3].to_f32(),+ ];++ let mut ret0 = vdupq_n_s32(0);+ let mut ret1 = vdupq_n_s32(0);+ let mut ret2 = vdupq_n_s32(0);+ let mut ret3 = vdupq_n_s32(0);++ macro_rules! dot_rows {+ ($xv:expr, $yv:expr) => {{+ ret0 = sdot_laneq_s32::<0>(ret0, $xv, $yv);+ ret1 = sdot_laneq_s32::<1>(ret1, $xv, $yv);+ ret2 = sdot_laneq_s32::<2>(ret2, $xv, $yv);+ ret3 = sdot_laneq_s32::<3>(ret3, $xv, $yv);+ }};+ }++ let y0 = vld1q_s8_x4(y.qs.as_ptr());+ let x0 = vld1q_s8_x4(x.qs.as_ptr());+ dot_rows!(x0.0, y0.0);+ dot_rows!(x0.1, y0.1);+ dot_rows!(x0.2, y0.2);+ dot_rows!(x0.3, y0.3);++ let y1 = vld1q_s8_x4(y.qs.as_ptr().add(64));+ let x1 = vld1q_s8_x4(x.qs.as_ptr().add(64));+ dot_rows!(x1.0, y1.0);+ dot_rows!(x1.1, y1.1);+ dot_rows!(x1.2, y1.2);+ dot_rows!(x1.3, y1.3);++ sum0 = vmlaq_f32(sum0, vcvtq_f32_s32(ret0), vmulq_n_f32(bd, ad[0]));+ sum1 = vmlaq_f32(sum1, vcvtq_f32_s32(ret1), vmulq_n_f32(bd, ad[1]));+ sum2 = vmlaq_f32(sum2, vcvtq_f32_s32(ret2), vmulq_n_f32(bd, ad[2]));+ sum3 = vmlaq_f32(sum3, vcvtq_f32_s32(ret3), vmulq_n_f32(bd, ad[3]));+ }++ vst1q_f32(dst.add(row * n + col), sum0);+ vst1q_f32(dst.add((row + 1) * n + col), sum1);+ vst1q_f32(dst.add((row + 2) * n + col), sum2);+ vst1q_f32(dst.add((row + 3) * n + col), sum3);+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "dotprod")]+unsafe fn vec_dot_4_q8_0x4_q8_0_4x4(n: usize, xs: &[BlockQ8_0x4], ys: &[BlockQ8_0]) -> [f32; 4] {+ debug_assert!(n.is_multiple_of(QK8_0));+ let mut out = [0f32; 4];+ let mut acc = vdupq_n_f32(0.);++ for (x, y) in xs.iter().zip(ys.iter()) {+ let bd = load_f16x4(x.d.as_ptr());+ let ad = vdupq_n_f32(y.d.to_f32());+ let y0 = vld1q_s8(y.qs.as_ptr());+ let y1 = vld1q_s8(y.qs.as_ptr().add(16));+ let x0 = vld1q_s8_x4(x.qs.as_ptr());+ let x1 = vld1q_s8_x4(x.qs.as_ptr().add(64));+ let mut ret = vdupq_n_s32(0);++ ret = sdot_laneq_s32::<0>(ret, x0.0, y0);+ ret = sdot_laneq_s32::<1>(ret, x0.1, y0);+ ret = sdot_laneq_s32::<2>(ret, x0.2, y0);+ ret = sdot_laneq_s32::<3>(ret, x0.3, y0);+ ret = sdot_laneq_s32::<0>(ret, x1.0, y1);+ ret = sdot_laneq_s32::<1>(ret, x1.1, y1);+ ret = sdot_laneq_s32::<2>(ret, x1.2, y1);+ ret = sdot_laneq_s32::<3>(ret, x1.3, y1);++ acc = vmlaq_f32(acc, vcvtq_f32_s32(ret), vmulq_f32(ad, bd));+ }++ vst1q_f32(out.as_mut_ptr(), acc);+ out+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "dotprod")]+unsafe fn vec_dot_4_q8_0x4_q8_0_4x8(n: usize, xs: &[BlockQ8_0x4], ys: &[BlockQ8_0]) -> [f32; 4] {+ debug_assert!(n.is_multiple_of(QK8_0));+ let mut out = [0f32; 4];+ let mut acc = vdupq_n_f32(0.);++ for (x, y) in xs.iter().zip(ys.iter()) {+ let bd = load_f16x4(x.d.as_ptr());+ let ad = vdupq_n_f32(y.d.to_f32());+ let a0 = vreinterpretq_s8_s64(vld1q_dup_s64(y.qs.as_ptr() as *const i64));+ let a1 = vreinterpretq_s8_s64(vld1q_dup_s64(y.qs.as_ptr().add(8) as *const i64));+ let a2 = vreinterpretq_s8_s64(vld1q_dup_s64(y.qs.as_ptr().add(16) as *const i64));+ let a3 = vreinterpretq_s8_s64(vld1q_dup_s64(y.qs.as_ptr().add(24) as *const i64));+ let x0 = vld1q_s8_x4(x.qs.as_ptr());+ let x1 = vld1q_s8_x4(x.qs.as_ptr().add(64));+ let mut ret0 = vdupq_n_s32(0);+ let mut ret1 = vdupq_n_s32(0);++ ret0 = sdot_acc(ret0, x0.0, a0);+ ret1 = sdot_acc(ret1, x0.1, a0);+ ret0 = sdot_acc(ret0, x0.2, a1);+ ret1 = sdot_acc(ret1, x0.3, a1);+ ret0 = sdot_acc(ret0, x1.0, a2);+ ret1 = sdot_acc(ret1, x1.1, a2);+ ret0 = sdot_acc(ret0, x1.2, a3);+ ret1 = sdot_acc(ret1, x1.3, a3);++ let ret = vpaddq_s32(ret0, ret1);+ acc = vmlaq_f32(acc, vcvtq_f32_s32(ret), vmulq_f32(ad, bd));+ }++ vst1q_f32(out.as_mut_ptr(), acc);+ out+}++#[cfg(target_arch = "aarch64")]+#[inline(always)]+pub(crate) fn vec_dot_4_q8_0x4_q8_0(n: usize, xs: &[BlockQ8_0x4], ys: &[BlockQ8_0]) -> [f32; 4] {+ unsafe {+ if crate::cpu::features::get().i8mm {+ vec_dot_4_q8_0x4_q8_0_4x8(n, xs, ys)+ } else {+ vec_dot_4_q8_0x4_q8_0_4x4(n, xs, ys)+ }+ }+}++#[cfg(target_arch = "aarch64")]+fn matmul_q8_0_x4_gemv(+ (_m, k, n): (usize, usize, usize),+ lhs: &[f32],+ repacked: &[BlockQ8_0x4],+ dst: &mut [f32],+) -> crate::Result<()> {+ debug_assert!(k.is_multiple_of(QK8_0));+ debug_assert!(n.is_multiple_of(4));+ let k_in_blocks = k / QK8_0;+ let n_groups = n / 4;++ thread_local! {+ static LHS_Q8_0_GEMV_SCRATCH: std::cell::RefCell<Vec<u64>> =+ const { std::cell::RefCell::new(Vec::new()) };+ }++ let elem_size = std::mem::size_of::<BlockQ8_0>();+ let required_len = (k_in_blocks * elem_size).div_ceil(8);+ LHS_Q8_0_GEMV_SCRATCH.with(|cell| -> crate::Result<()> {+ let mut scratch = cell.borrow_mut();+ if scratch.len() < required_len {+ scratch.resize(required_len, 0);+ }+ let lhs_b: &mut [BlockQ8_0] = unsafe {+ std::slice::from_raw_parts_mut(scratch.as_mut_ptr() as *mut BlockQ8_0, k_in_blocks)+ };+ <BlockQ8_0 as super::GgmlType>::from_float(lhs, lhs_b);++ let pool = crate::utils::barrier_pool();+ let lhs_ptr = lhs_b.as_ptr() as usize;+ let repacked_ptr = repacked.as_ptr() as usize;+ let dst_ptr = dst.as_mut_ptr() as usize;+ let x4_block_bytes = std::mem::size_of::<BlockQ8_0x4>();++ // equal-sized gemv units: a static split skips the shared cursor, which costs+ // several us per small decode matmul+ pool.execute_static(n_groups, |range| {+ let lhs_row =+ unsafe { std::slice::from_raw_parts(lhs_ptr as *const BlockQ8_0, k_in_blocks) };+ let dst_ptr = dst_ptr as *mut f32;+ for g in range {+ let xs = unsafe {+ std::slice::from_raw_parts(+ (repacked_ptr + g * k_in_blocks * x4_block_bytes) as *const BlockQ8_0x4,+ k_in_blocks,+ )+ };+ let results = vec_dot_4_q8_0x4_q8_0(k, xs, lhs_row);+ unsafe {+ std::ptr::copy_nonoverlapping(results.as_ptr(), dst_ptr.add(g * 4), 4);+ }+ }+ });++ Ok(())+ })+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "dotprod")]+pub(crate) fn matmul_q8_0_x4(+ (m, k, n): (usize, usize, usize),+ lhs: &[f32],+ repacked: &[BlockQ8_0x4],+ dst: &mut [f32],+) -> crate::Result<()> {+ debug_assert!(k.is_multiple_of(QK8_0));+ debug_assert!(n.is_multiple_of(4));+ if m == 1 {+ return matmul_q8_0_x4_gemv((m, k, n), lhs, repacked, dst);+ }+ debug_assert!(m.is_multiple_of(4));+ let k_in_blocks = k / QK8_0;+ let n_groups = n / 4;+ let row_groups = m / 4;++ thread_local! {+ static LHS_X4_SCRATCH: std::cell::RefCell<Vec<u64>> =+ const { std::cell::RefCell::new(Vec::new()) };+ }++ let elem_size = std::mem::size_of::<BlockQ8_0x4>();+ let required_len = (row_groups * k_in_blocks * elem_size).div_ceil(8);+ LHS_X4_SCRATCH.with(|cell| -> crate::Result<()> {+ let mut scratch = cell.borrow_mut();+ if scratch.len() < required_len {+ scratch.resize(required_len, 0);+ }+ let lhs_x4: &mut [BlockQ8_0x4] = unsafe {+ std::slice::from_raw_parts_mut(+ scratch.as_mut_ptr() as *mut BlockQ8_0x4,+ row_groups * k_in_blocks,+ )+ };+ for group in 0..row_groups {+ quantize_q8_0x4_interleaved::<4>(+ lhs,+ k,+ group * 4,+ &mut lhs_x4[group * k_in_blocks..(group + 1) * k_in_blocks],+ );+ }++ let tiles_total = row_groups * n_groups;+ let pool = crate::utils::barrier_pool();+ let lhs_ptr = lhs_x4.as_ptr() as usize;+ let repacked_ptr = repacked.as_ptr() as usize;+ let dst_ptr = dst.as_mut_ptr() as usize;++ pool.execute_chunked(tiles_total, |range| {+ let lhs_ptr = lhs_ptr as *const BlockQ8_0x4;+ let repacked_ptr = repacked_ptr as *const BlockQ8_0x4;+ let dst_ptr = dst_ptr as *mut f32;+ for tile in range {+ let row_group = tile / n_groups;+ let col_group = tile - row_group * n_groups;+ let row = row_group * 4;+ let col = col_group * 4;+ let lhs_tile = unsafe {+ std::slice::from_raw_parts(lhs_ptr.add(row_group * k_in_blocks), k_in_blocks)+ };+ let rhs_tile = unsafe {+ std::slice::from_raw_parts(+ repacked_ptr.add(col_group * k_in_blocks),+ k_in_blocks,+ )+ };+ unsafe { store_q8_0x4_4x4(dst_ptr, n, row, col, lhs_tile, rhs_tile, k_in_blocks) };+ }+ });++ Ok(())+ })+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "i8mm")]+unsafe fn store_q8_0x4_4x4_i8mm(+ dst: *mut f32,+ n: usize,+ row: usize,+ col: usize,+ lhs: &[BlockQ8_0x4],+ rhs: &[BlockQ8_0x4],+ k_in_blocks: usize,+) {+ let mut sum0 = vdupq_n_f32(0.);+ let mut sum1 = vdupq_n_f32(0.);+ let mut sum2 = vdupq_n_f32(0.);+ let mut sum3 = vdupq_n_f32(0.);++ for i in 0..k_in_blocks {+ let x = &rhs[i];+ let y = &lhs[i];+ let bd = [+ x.d[0].to_f32(),+ x.d[1].to_f32(),+ x.d[2].to_f32(),+ x.d[3].to_f32(),+ ];+ let bd = vld1q_f32(bd.as_ptr());+ let ad = [+ y.d[0].to_f32(),+ y.d[1].to_f32(),+ y.d[2].to_f32(),+ y.d[3].to_f32(),+ ];++ let mut acc00 = vdupq_n_s32(0);+ let mut acc01 = vdupq_n_s32(0);+ let mut acc10 = vdupq_n_s32(0);+ let mut acc11 = vdupq_n_s32(0);++ for chunk in 0..4 {+ let offset = chunk * 32;+ let a01 = vld1q_s8(y.qs.as_ptr().add(offset));+ let a23 = vld1q_s8(y.qs.as_ptr().add(offset + 16));+ let b01 = vld1q_s8(x.qs.as_ptr().add(offset));+ let b23 = vld1q_s8(x.qs.as_ptr().add(offset + 16));++ acc00 = smmla_s32(acc00, a01, b01);+ acc01 = smmla_s32(acc01, a01, b23);+ acc10 = smmla_s32(acc10, a23, b01);+ acc11 = smmla_s32(acc11, a23, b23);+ }++ let row0 = vcombine_s32(vget_low_s32(acc00), vget_low_s32(acc01));+ let row1 = vcombine_s32(vget_high_s32(acc00), vget_high_s32(acc01));+ let row2 = vcombine_s32(vget_low_s32(acc10), vget_low_s32(acc11));+ let row3 = vcombine_s32(vget_high_s32(acc10), vget_high_s32(acc11));++ sum0 = vmlaq_f32(sum0, vcvtq_f32_s32(row0), vmulq_n_f32(bd, ad[0]));+ sum1 = vmlaq_f32(sum1, vcvtq_f32_s32(row1), vmulq_n_f32(bd, ad[1]));+ sum2 = vmlaq_f32(sum2, vcvtq_f32_s32(row2), vmulq_n_f32(bd, ad[2]));+ sum3 = vmlaq_f32(sum3, vcvtq_f32_s32(row3), vmulq_n_f32(bd, ad[3]));+ }++ vst1q_f32(dst.add(row * n + col), sum0);+ vst1q_f32(dst.add((row + 1) * n + col), sum1);+ vst1q_f32(dst.add((row + 2) * n + col), sum2);+ vst1q_f32(dst.add((row + 3) * n + col), sum3);+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "i8mm")]+pub(crate) fn matmul_q8_0_x4_i8mm(+ (m, k, n): (usize, usize, usize),+ lhs: &[f32],+ repacked: &[BlockQ8_0x4],+ dst: &mut [f32],+) -> crate::Result<()> {+ debug_assert!(k.is_multiple_of(QK8_0));+ debug_assert!(n.is_multiple_of(4));+ debug_assert!(m.is_multiple_of(4));+ let k_in_blocks = k / QK8_0;+ let n_groups = n / 4;+ let row_groups = m / 4;++ thread_local! {+ static LHS_X4_I8MM_SCRATCH: std::cell::RefCell<Vec<u64>> =+ const { std::cell::RefCell::new(Vec::new()) };+ }++ let elem_size = std::mem::size_of::<BlockQ8_0x4>();+ let required_len = (row_groups * k_in_blocks * elem_size).div_ceil(8);+ LHS_X4_I8MM_SCRATCH.with(|cell| -> crate::Result<()> {+ let mut scratch = cell.borrow_mut();+ if scratch.len() < required_len {+ scratch.resize(required_len, 0);+ }+ let lhs_x4: &mut [BlockQ8_0x4] = unsafe {+ std::slice::from_raw_parts_mut(+ scratch.as_mut_ptr() as *mut BlockQ8_0x4,+ row_groups * k_in_blocks,+ )+ };+ for group in 0..row_groups {+ quantize_q8_0x4_interleaved::<8>(+ lhs,+ k,+ group * 4,+ &mut lhs_x4[group * k_in_blocks..(group + 1) * k_in_blocks],+ );+ }++ let tiles_total = row_groups * n_groups;+ let pool = crate::utils::barrier_pool();+ let lhs_ptr = lhs_x4.as_ptr() as usize;+ let repacked_ptr = repacked.as_ptr() as usize;+ let dst_ptr = dst.as_mut_ptr() as usize;+ pool.execute_chunked(tiles_total, |range| {+ let lhs_ptr = lhs_ptr as *const BlockQ8_0x4;+ let repacked_ptr = repacked_ptr as *const BlockQ8_0x4;+ let dst_ptr = dst_ptr as *mut f32;+ for tile in range {+ let row_group = tile / n_groups;+ let col_group = tile - row_group * n_groups;+ let row = row_group * 4;+ let col = col_group * 4;+ let lhs_tile = unsafe {+ std::slice::from_raw_parts(lhs_ptr.add(row_group * k_in_blocks), k_in_blocks)+ };+ let rhs_tile = unsafe {+ std::slice::from_raw_parts(+ repacked_ptr.add(col_group * k_in_blocks),+ k_in_blocks,+ )+ };+ unsafe {+ store_q8_0x4_4x4_i8mm(dst_ptr, n, row, col, lhs_tile, rhs_tile, k_in_blocks)+ }+ }+ });+ Ok(())+ })+}++#[inline(always)]+pub(crate) fn vec_dot_q8k_q8k(n: usize, xs: &[BlockQ8K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q8k_q8k: {n} is not divisible by {QK_K}"+ );+ let mut sumf = 0f32;+ for (xs, ys) in xs.iter().zip(ys.iter()) {+ unsafe {+ let mut sum_i = vdupq_n_s32(0);+ let scale = xs.d * ys.d;+ let xs = xs.qs.as_ptr();+ let ys = ys.qs.as_ptr();+ for i in (0..QK_K).step_by(16) {+ let xs = vld1q_s8(xs.add(i));+ let ys = vld1q_s8(ys.add(i));+ let xy = vdotq_s32(xs, ys);+ sum_i = vaddq_s32(sum_i, xy)+ }+ sumf += vaddvq_s32(sum_i) as f32 * scale+ }+ }+ sumf+}++#[inline(always)]+pub(crate) fn vec_dot_q6k_q8k(n: usize, xs: &[BlockQ6K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q6k_q8k: {n} is not divisible by {QK_K}"+ );+ let mut sum = 0f32;+ unsafe {+ let m4b = vdupq_n_u8(0xF);++ let mone = vdupq_n_u8(3);++ for (x, y) in xs.iter().zip(ys.iter()) {+ let d_all = x.d.to_f32();++ let mut q6 = x.ql.as_ptr();+ let mut qh = x.qh.as_ptr();+ let mut q8 = y.qs.as_ptr();++ let mut scale = x.scales.as_ptr();++ let q8sums = vld1q_s16_x2(y.bsums.as_ptr());+ let scales = vld1q_s8(scale);+ let q6scales = int16x8x2_t(+ vmovl_s8(vget_low_s8(scales)),+ vmovl_s8(vget_high_s8(scales)),+ );++ let prod = vaddq_s32(+ vaddq_s32(+ vmull_s16(vget_low_s16(q8sums.0), vget_low_s16(q6scales.0)),+ vmull_s16(vget_high_s16(q8sums.0), vget_high_s16(q6scales.0)),+ ),+ vaddq_s32(+ vmull_s16(vget_low_s16(q8sums.1), vget_low_s16(q6scales.1)),+ vmull_s16(vget_high_s16(q8sums.1), vget_high_s16(q6scales.1)),+ ),+ );+ let isum_mins = vaddvq_s32(prod);++ let mut isum = 0i32;++ for _j in 0..QK_K / 128 {+ let qhbits = vld1q_u8_x2(qh);+ qh = qh.add(32);+ let q6bits = vld1q_u8_x4(q6);+ q6 = q6.add(64);+ let q8bytes = vld1q_s8_x4(q8);+ q8 = q8.add(64);++ let q6h_0 = vshlq_n_u8(vandq_u8(mone, qhbits.0), 4);+ let q6h_1 = vshlq_n_u8(vandq_u8(mone, qhbits.1), 4);+ let shifted = vshrq_n_u8(qhbits.0, 2);+ let q6h_2 = vshlq_n_u8(vandq_u8(mone, shifted), 4);+ let shifted = vshrq_n_u8(qhbits.1, 2);+ let q6h_3 = vshlq_n_u8(vandq_u8(mone, shifted), 4);++ let q6bytes_0 = vreinterpretq_s8_u8(vorrq_u8(vandq_u8(q6bits.0, m4b), q6h_0));+ let q6bytes_1 = vreinterpretq_s8_u8(vorrq_u8(vandq_u8(q6bits.1, m4b), q6h_1));+ let q6bytes_2 = vreinterpretq_s8_u8(vorrq_u8(vandq_u8(q6bits.2, m4b), q6h_2));+ let q6bytes_3 = vreinterpretq_s8_u8(vorrq_u8(vandq_u8(q6bits.3, m4b), q6h_3));++ let p0 = vdotq_s32(q6bytes_0, q8bytes.0);+ let p1 = vdotq_s32(q6bytes_1, q8bytes.1);+ let (scale0, scale1) = (*scale as i32, *scale.add(1) as i32);+ isum += vaddvq_s32(p0) * scale0 + vaddvq_s32(p1) * scale1;+ scale = scale.add(2);++ let p2 = vdotq_s32(q6bytes_2, q8bytes.2);+ let p3 = vdotq_s32(q6bytes_3, q8bytes.3);+ let (scale0, scale1) = (*scale as i32, *scale.add(1) as i32);+ isum += vaddvq_s32(p2) * scale0 + vaddvq_s32(p3) * scale1;+ scale = scale.add(2);++ let q8bytes = vld1q_s8_x4(q8);+ q8 = q8.add(64);++ let shifted = vshrq_n_u8(qhbits.0, 4);+ let q6h_0 = vshlq_n_u8(vandq_u8(mone, shifted), 4);+ let shifted = vshrq_n_u8(qhbits.1, 4);+ let q6h_1 = vshlq_n_u8(vandq_u8(mone, shifted), 4);+ let shifted = vshrq_n_u8(qhbits.0, 6);+ let q6h_2 = vshlq_n_u8(vandq_u8(mone, shifted), 4);+ let shifted = vshrq_n_u8(qhbits.1, 6);+ let q6h_3 = vshlq_n_u8(vandq_u8(mone, shifted), 4);++ let q6bytes_0 = vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q6bits.0, 4), q6h_0));+ let q6bytes_1 = vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q6bits.1, 4), q6h_1));+ let q6bytes_2 = vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q6bits.2, 4), q6h_2));+ let q6bytes_3 = vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q6bits.3, 4), q6h_3));++ let p0 = vdotq_s32(q6bytes_0, q8bytes.0);+ let p1 = vdotq_s32(q6bytes_1, q8bytes.1);+ let (scale0, scale1) = (*scale as i32, *scale.add(1) as i32);+ isum += vaddvq_s32(p0) * scale0 + vaddvq_s32(p1) * scale1;+ scale = scale.add(2);++ let p2 = vdotq_s32(q6bytes_2, q8bytes.2);+ let p3 = vdotq_s32(q6bytes_3, q8bytes.3);+ let (scale0, scale1) = (*scale as i32, *scale.add(1) as i32);+ isum += vaddvq_s32(p2) * scale0 + vaddvq_s32(p3) * scale1;+ scale = scale.add(2);+ }+ sum += d_all * y.d * ((isum - 32 * isum_mins) as f32);+ }+ }+ sum+}++/// Four Q6K dot products sharing one Q8K load.+#[inline(always)]+pub(crate) fn vec_dot_4_q6k_q8k(+ n: usize,+ xs0: &[BlockQ6K],+ xs1: &[BlockQ6K],+ xs2: &[BlockQ6K],+ xs3: &[BlockQ6K],+ ys: &[BlockQ8K],+) -> (f32, f32, f32, f32) {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_4_q6k_q8k: {n} is not divisible by {QK_K}"+ );++ let mut sum0 = 0f32;+ let mut sum1 = 0f32;+ let mut sum2 = 0f32;+ let mut sum3 = 0f32;++ unsafe {+ let m4b = vdupq_n_u8(0xF);+ let mone = vdupq_n_u8(3);++ for ((((x0, x1), x2), x3), y) in xs0+ .iter()+ .zip(xs1.iter())+ .zip(xs2.iter())+ .zip(xs3.iter())+ .zip(ys.iter())+ {+ let yd = y.d;++ // Q8K bsums - loaded once for all four columns.+ let q8sums = vld1q_s16_x2(y.bsums.as_ptr());++ // Compute isum_mins for each column: dot(q8sums, column_scales).+ macro_rules! col_isum_mins {+ ($x:ident) => {{+ let scales_v = vld1q_s8($x.scales.as_ptr());+ let q6sc = int16x8x2_t(+ vmovl_s8(vget_low_s8(scales_v)),+ vmovl_s8(vget_high_s8(scales_v)),+ );+ let prod = vaddq_s32(+ vaddq_s32(+ vmull_s16(vget_low_s16(q8sums.0), vget_low_s16(q6sc.0)),+ vmull_s16(vget_high_s16(q8sums.0), vget_high_s16(q6sc.0)),+ ),+ vaddq_s32(+ vmull_s16(vget_low_s16(q8sums.1), vget_low_s16(q6sc.1)),+ vmull_s16(vget_high_s16(q8sums.1), vget_high_s16(q6sc.1)),+ ),+ );+ vaddvq_s32(prod)+ }};+ }++ let isum_mins0 = col_isum_mins!(x0);+ let isum_mins1 = col_isum_mins!(x1);+ let isum_mins2 = col_isum_mins!(x2);+ let isum_mins3 = col_isum_mins!(x3);++ let mut q6_0 = x0.ql.as_ptr();+ let mut qh_0 = x0.qh.as_ptr();+ let mut sc_0 = x0.scales.as_ptr();+ let mut q6_1 = x1.ql.as_ptr();+ let mut qh_1 = x1.qh.as_ptr();+ let mut sc_1 = x1.scales.as_ptr();+ let mut q6_2 = x2.ql.as_ptr();+ let mut qh_2 = x2.qh.as_ptr();+ let mut sc_2 = x2.scales.as_ptr();+ let mut q6_3 = x3.ql.as_ptr();+ let mut qh_3 = x3.qh.as_ptr();+ let mut sc_3 = x3.scales.as_ptr();+ let mut q8 = y.qs.as_ptr();++ let mut isum0 = 0i32;+ let mut isum1 = 0i32;+ let mut isum2 = 0i32;+ let mut isum3 = 0i32;++ for _j in 0..QK_K / 128 {+ // Load Q8K bytes once - shared across all four columns.+ let q8lo = vld1q_s8_x4(q8);+ q8 = q8.add(64);+ let q8hi = vld1q_s8_x4(q8);+ q8 = q8.add(64);++ // Decode one column's Q6 bits and accumulate into its isum.+ // q8lo/q8hi are shared from the outer scope.+ macro_rules! process_col {+ ($q6:ident, $qh:ident, $sc:ident, $isum:ident) => {+ let qhb = vld1q_u8_x2($qh);+ $qh = $qh.add(32);+ let q6b = vld1q_u8_x4($q6);+ $q6 = $q6.add(64);++ // First half: low nibbles of ql + bits[1:0] and bits[3:2] of qh.+ let qh00 = vshlq_n_u8(vandq_u8(mone, qhb.0), 4);+ let qh01 = vshlq_n_u8(vandq_u8(mone, qhb.1), 4);+ let qh10 = vshlq_n_u8(vandq_u8(mone, vshrq_n_u8(qhb.0, 2)), 4);+ let qh11 = vshlq_n_u8(vandq_u8(mone, vshrq_n_u8(qhb.1, 2)), 4);++ let q6b0 = vreinterpretq_s8_u8(vorrq_u8(vandq_u8(q6b.0, m4b), qh00));+ let q6b1 = vreinterpretq_s8_u8(vorrq_u8(vandq_u8(q6b.1, m4b), qh01));+ let q6b2 = vreinterpretq_s8_u8(vorrq_u8(vandq_u8(q6b.2, m4b), qh10));+ let q6b3 = vreinterpretq_s8_u8(vorrq_u8(vandq_u8(q6b.3, m4b), qh11));++ let p0 = vdotq_s32(q6b0, q8lo.0);+ let p1 = vdotq_s32(q6b1, q8lo.1);+ $isum +=+ vaddvq_s32(p0) * (*$sc as i32) + vaddvq_s32(p1) * (*$sc.add(1) as i32);+ $sc = $sc.add(2);++ let p2 = vdotq_s32(q6b2, q8lo.2);+ let p3 = vdotq_s32(q6b3, q8lo.3);+ $isum +=+ vaddvq_s32(p2) * (*$sc as i32) + vaddvq_s32(p3) * (*$sc.add(1) as i32);+ $sc = $sc.add(2);++ // Second half: high nibbles of ql + bits[5:4] and bits[7:6] of qh.+ let qh20 = vshlq_n_u8(vandq_u8(mone, vshrq_n_u8(qhb.0, 4)), 4);+ let qh21 = vshlq_n_u8(vandq_u8(mone, vshrq_n_u8(qhb.1, 4)), 4);+ let qh30 = vshlq_n_u8(vandq_u8(mone, vshrq_n_u8(qhb.0, 6)), 4);+ let qh31 = vshlq_n_u8(vandq_u8(mone, vshrq_n_u8(qhb.1, 6)), 4);++ let q6b0 = vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q6b.0, 4), qh20));+ let q6b1 = vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q6b.1, 4), qh21));+ let q6b2 = vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q6b.2, 4), qh30));+ let q6b3 = vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q6b.3, 4), qh31));++ let p0 = vdotq_s32(q6b0, q8hi.0);+ let p1 = vdotq_s32(q6b1, q8hi.1);+ $isum +=+ vaddvq_s32(p0) * (*$sc as i32) + vaddvq_s32(p1) * (*$sc.add(1) as i32);+ $sc = $sc.add(2);++ let p2 = vdotq_s32(q6b2, q8hi.2);+ let p3 = vdotq_s32(q6b3, q8hi.3);+ $isum +=+ vaddvq_s32(p2) * (*$sc as i32) + vaddvq_s32(p3) * (*$sc.add(1) as i32);+ $sc = $sc.add(2);+ };+ }++ process_col!(q6_0, qh_0, sc_0, isum0);+ process_col!(q6_1, qh_1, sc_1, isum1);+ process_col!(q6_2, qh_2, sc_2, isum2);+ process_col!(q6_3, qh_3, sc_3, isum3);+ }++ sum0 += x0.d.to_f32() * yd * ((isum0 - 32 * isum_mins0) as f32);+ sum1 += x1.d.to_f32() * yd * ((isum1 - 32 * isum_mins1) as f32);+ sum2 += x2.d.to_f32() * yd * ((isum2 - 32 * isum_mins2) as f32);+ sum3 += x3.d.to_f32() * yd * ((isum3 - 32 * isum_mins3) as f32);+ }+ }++ (sum0, sum1, sum2, sum3)+}++#[inline(always)]+pub(crate) fn vec_dot_q5k_q8k(n: usize, xs: &[BlockQ5K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q5k_q8k: {n} is not divisible by {QK_K}"+ );+ let mut sumf = 0f32;+ let mut utmp = [0u32; 4];+ const KMASK1: u32 = 0x3f3f3f3f;+ const KMASK2: u32 = 0x0f0f0f0f;+ const KMASK3: u32 = 0x03030303;++ unsafe {+ let m4b = vdupq_n_u8(0xF);+ let mone = vdupq_n_u8(1);+ let mtwo = vdupq_n_u8(2);++ for (x, y) in xs.iter().zip(ys.iter()) {+ let d = y.d * x.d.to_f32();+ let dmin = y.d * x.dmin.to_f32();++ let q8sums = vpaddq_s16(+ vld1q_s16(y.bsums.as_ptr()),+ vld1q_s16(y.bsums.as_ptr().add(8)),+ );++ LittleEndian::read_u32_into(&x.scales, &mut utmp[0..3]);++ utmp[3] = ((utmp[2] >> 4) & KMASK2) | (((utmp[1] >> 6) & KMASK3) << 4);+ let uaux = utmp[1] & KMASK1;+ utmp[1] = (utmp[2] & KMASK2) | (((utmp[0] >> 6) & KMASK3) << 4);+ utmp[2] = uaux;+ utmp[0] &= KMASK1;++ let mins8 = vld1_u8((utmp.as_ptr() as *const u8).add(8));+ let mins = vreinterpretq_s16_u16(vmovl_u8(mins8));+ let prod = vaddq_s32(+ vmull_s16(vget_low_s16(q8sums), vget_low_s16(mins)),+ vmull_s16(vget_high_s16(q8sums), vget_high_s16(mins)),+ );+ let sumi_mins = vaddvq_s32(prod);++ let mut scales = utmp.as_ptr() as *const u8;++ let mut q5 = x.qs.as_ptr();+ let mut q8 = y.qs.as_ptr();++ let mut qhbits = vld1q_u8_x2(x.qh.as_ptr());++ let mut sumi = 0i32;++ for _j in 0..QK_K / 64 {+ let q5bits = vld1q_u8_x2(q5);+ q5 = q5.add(32);+ let q8bytes = vld1q_s8_x4(q8);+ q8 = q8.add(64);++ let q5h_0 = vshlq_n_u8(vandq_u8(mone, qhbits.0), 4);+ let q5h_1 = vshlq_n_u8(vandq_u8(mone, qhbits.1), 4);+ let q5h_2 = vshlq_n_u8(vandq_u8(mtwo, qhbits.0), 3);+ let q5h_3 = vshlq_n_u8(vandq_u8(mtwo, qhbits.1), 3);+ qhbits.0 = vshrq_n_u8(qhbits.0, 2);+ qhbits.1 = vshrq_n_u8(qhbits.1, 2);++ let q5bytes_0 = vreinterpretq_s8_u8(vorrq_u8(vandq_u8(q5bits.0, m4b), q5h_0));+ let q5bytes_1 = vreinterpretq_s8_u8(vorrq_u8(vandq_u8(q5bits.1, m4b), q5h_1));+ let q5bytes_2 = vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q5bits.0, 4), q5h_2));+ let q5bytes_3 = vreinterpretq_s8_u8(vorrq_u8(vshrq_n_u8(q5bits.1, 4), q5h_3));++ let p0 = vdotq_s32(q5bytes_0, q8bytes.0);+ let p1 = vdotq_s32(q5bytes_1, q8bytes.1);+ sumi += vaddvq_s32(vaddq_s32(p0, p1)) * *scales as i32;+ scales = scales.add(1);++ let p2 = vdotq_s32(q5bytes_2, q8bytes.2);+ let p3 = vdotq_s32(q5bytes_3, q8bytes.3);+ sumi += vaddvq_s32(vaddq_s32(p2, p3)) * *scales as i32;+ scales = scales.add(1);+ }+ sumf += d * sumi as f32 - dmin * sumi_mins as f32;+ }+ }+ sumf+}++#[inline(always)]+pub(crate) fn vec_dot_q4k_q8k(n: usize, xs: &[BlockQ4K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q4k_q8k: {n} is not divisible by {QK_K}"+ );+ let mut sumf = 0f32;+ let mut utmp = [0u32; 4];+ let mut scales = [0u8; 16];+ const KMASK1: u32 = 0x3f3f3f3f;+ const KMASK2: u32 = 0x0f0f0f0f;+ const KMASK3: u32 = 0x03030303;++ unsafe {+ let m4b = vdupq_n_u8(0xF);++ for (x, y) in xs.iter().zip(ys.iter()) {+ let d = y.d * x.d.to_f32();+ let dmin = y.d * x.dmin.to_f32();++ let q8sums = vpaddq_s16(+ vld1q_s16(y.bsums.as_ptr()),+ vld1q_s16(y.bsums.as_ptr().add(8)),+ );++ LittleEndian::read_u32_into(&x.scales, &mut utmp[0..3]);++ let mins8 = vld1_u32(+ [+ utmp[1] & KMASK1,+ ((utmp[2] >> 4) & KMASK2) | (((utmp[1] >> 6) & KMASK3) << 4),+ ]+ .as_ptr(),+ );+ utmp[1] = (utmp[2] & KMASK2) | (((utmp[0] >> 6) & KMASK3) << 4);+ utmp[0] &= KMASK1;++ let mins = vreinterpretq_s16_u16(vmovl_u8(vreinterpret_u8_u32(mins8)));+ let prod = vaddq_s32(+ vmull_s16(vget_low_s16(q8sums), vget_low_s16(mins)),+ vmull_s16(vget_high_s16(q8sums), vget_high_s16(mins)),+ );+ sumf -= dmin * vaddvq_s32(prod) as f32;++ LittleEndian::write_u32_into(&utmp, &mut scales);++ let mut q4 = x.qs.as_ptr();+ let mut q8 = y.qs.as_ptr();++ let mut sumi1 = 0i32;+ let mut sumi2 = 0i32;++ for j in 0..QK_K / 64 {+ let q4bits = vld1q_u8_x2(q4);+ q4 = q4.add(32);+ let q8bytes = vld1q_s8_x2(q8);+ q8 = q8.add(32);+ let q4bytes = int8x16x2_t(+ vreinterpretq_s8_u8(vandq_u8(q4bits.0, m4b)),+ vreinterpretq_s8_u8(vandq_u8(q4bits.1, m4b)),+ );+ let p0 = vdotq_s32(q4bytes.0, q8bytes.0);+ let p1 = vdotq_s32(q4bytes.1, q8bytes.1);+ sumi1 += vaddvq_s32(vaddq_s32(p0, p1)) * scales[2 * j] as i32;++ let q8bytes = vld1q_s8_x2(q8);+ q8 = q8.add(32);+ let q4bytes = int8x16x2_t(+ vreinterpretq_s8_u8(vshrq_n_u8(q4bits.0, 4)),+ vreinterpretq_s8_u8(vshrq_n_u8(q4bits.1, 4)),+ );+ let p2 = vdotq_s32(q4bytes.0, q8bytes.0);+ let p3 = vdotq_s32(q4bytes.1, q8bytes.1);+ sumi2 += vaddvq_s32(vaddq_s32(p2, p3)) * scales[2 * j + 1] as i32;+ }+ sumf += d * (sumi1 + sumi2) as f32;+ }+ }+ sumf+}++/// Four Q4K dot products sharing one Q8K load.+#[inline(always)]+pub(crate) fn vec_dot_4_q4k_q8k(+ n: usize,+ xs0: &[BlockQ4K],+ xs1: &[BlockQ4K],+ xs2: &[BlockQ4K],+ xs3: &[BlockQ4K],+ ys: &[BlockQ8K],+) -> (f32, f32, f32, f32) {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_4_q4k_q8k: {n} is not divisible by {QK_K}"+ );++ let mut sum0 = 0f32;+ let mut sum1 = 0f32;+ let mut sum2 = 0f32;+ let mut sum3 = 0f32;++ let mut utmp = [0u32; 4];+ let mut sc0 = [0u8; 16];+ let mut sc1 = [0u8; 16];+ let mut sc2 = [0u8; 16];+ let mut sc3 = [0u8; 16];++ const KMASK1: u32 = 0x3f3f3f3f;+ const KMASK2: u32 = 0x0f0f0f0f;+ const KMASK3: u32 = 0x03030303;++ // Decode Q4K scale+min bytes into an 8-entry scales array and a float32x4x2+ // mins vector. Returns the mins vector so the caller can compute min correction.+ macro_rules! decode_q4k_scales {+ ($x:ident, $sc:ident) => {{+ LittleEndian::read_u32_into(&$x.scales, &mut utmp[0..3]);+ let mins8 = vld1_u32(+ [+ utmp[1] & KMASK1,+ ((utmp[2] >> 4) & KMASK2) | (((utmp[1] >> 6) & KMASK3) << 4),+ ]+ .as_ptr(),+ );+ utmp[1] = (utmp[2] & KMASK2) | (((utmp[0] >> 6) & KMASK3) << 4);+ utmp[0] &= KMASK1;+ LittleEndian::write_u32_into(&utmp, &mut $sc);+ vreinterpretq_s16_u16(vmovl_u8(vreinterpret_u8_u32(mins8)))+ }};+ }++ // Accumulate one column's nibble-group contribution into a scalar sumi.+ macro_rules! dot_col {+ ($q4:ident, $sc:ident, $vsum1:ident, $vsum2:ident, $q8lo:ident, $q8hi:ident, $j:ident, $m4b:ident) => {+ let bits = vld1q_u8_x2($q4);+ $q4 = $q4.add(32);+ let q4lo = int8x16x2_t(+ vreinterpretq_s8_u8(vandq_u8(bits.0, $m4b)),+ vreinterpretq_s8_u8(vandq_u8(bits.1, $m4b)),+ );+ $vsum1 = vmlaq_n_s32(+ $vsum1,+ vdotq_s32_pair(q4lo.0, $q8lo.0, q4lo.1, $q8lo.1),+ $sc[2 * $j] as i32,+ );+ let q4hi = int8x16x2_t(+ vreinterpretq_s8_u8(vshrq_n_u8(bits.0, 4)),+ vreinterpretq_s8_u8(vshrq_n_u8(bits.1, 4)),+ );+ $vsum2 = vmlaq_n_s32(+ $vsum2,+ vdotq_s32_pair(q4hi.0, $q8hi.0, q4hi.1, $q8hi.1),+ $sc[2 * $j + 1] as i32,+ );+ };+ }++ unsafe {+ let m4b = vdupq_n_u8(0xF);++ for ((((x0, x1), x2), x3), y) in xs0+ .iter()+ .zip(xs1.iter())+ .zip(xs2.iter())+ .zip(xs3.iter())+ .zip(ys.iter())+ {+ let yd = y.d;++ // Q8K bsums - loaded once for all four columns.+ let q8sums = vpaddq_s16(+ vld1q_s16(y.bsums.as_ptr()),+ vld1q_s16(y.bsums.as_ptr().add(8)),+ );++ // Decode scales and apply min correction for each column.+ let mins0 = decode_q4k_scales!(x0, sc0);+ let mins1 = decode_q4k_scales!(x1, sc1);+ let mins2 = decode_q4k_scales!(x2, sc2);+ let mins3 = decode_q4k_scales!(x3, sc3);++ let d0 = yd * x0.d.to_f32();+ let d1 = yd * x1.d.to_f32();+ let d2 = yd * x2.d.to_f32();+ let d3 = yd * x3.d.to_f32();++ // min correction: sum -= dmin * dot(q8sums, mins)+ macro_rules! min_correct {+ ($mins:ident, $dmin:expr, $sum:ident) => {+ let prod = vaddq_s32(+ vmull_s16(vget_low_s16(q8sums), vget_low_s16($mins)),+ vmull_s16(vget_high_s16(q8sums), vget_high_s16($mins)),+ );+ $sum -= $dmin * vaddvq_s32(prod) as f32;+ };+ }+ min_correct!(mins0, yd * x0.dmin.to_f32(), sum0);+ min_correct!(mins1, yd * x1.dmin.to_f32(), sum1);+ min_correct!(mins2, yd * x2.dmin.to_f32(), sum2);+ min_correct!(mins3, yd * x3.dmin.to_f32(), sum3);++ let mut q4_0 = x0.qs.as_ptr();+ let mut q4_1 = x1.qs.as_ptr();+ let mut q4_2 = x2.qs.as_ptr();+ let mut q4_3 = x3.qs.as_ptr();+ let mut q8 = y.qs.as_ptr();++ let mut s0a = vdupq_n_s32(0);+ let mut s0b = vdupq_n_s32(0);+ let mut s1a = vdupq_n_s32(0);+ let mut s1b = vdupq_n_s32(0);+ let mut s2a = vdupq_n_s32(0);+ let mut s2b = vdupq_n_s32(0);+ let mut s3a = vdupq_n_s32(0);+ let mut s3b = vdupq_n_s32(0);++ for j in 0..QK_K / 64 {+ // Load Q8K once and reuse across all four columns.+ let q8lo = vld1q_s8_x2(q8);+ q8 = q8.add(32);+ let q8hi = vld1q_s8_x2(q8);+ q8 = q8.add(32);++ dot_col!(q4_0, sc0, s0a, s0b, q8lo, q8hi, j, m4b);+ dot_col!(q4_1, sc1, s1a, s1b, q8lo, q8hi, j, m4b);+ dot_col!(q4_2, sc2, s2a, s2b, q8lo, q8hi, j, m4b);+ dot_col!(q4_3, sc3, s3a, s3b, q8lo, q8hi, j, m4b);+ }++ sum0 += d0 * vaddvq_s32(vaddq_s32(s0a, s0b)) as f32;+ sum1 += d1 * vaddvq_s32(vaddq_s32(s1a, s1b)) as f32;+ sum2 += d2 * vaddvq_s32(vaddq_s32(s2a, s2b)) as f32;+ sum3 += d3 * vaddvq_s32(vaddq_s32(s3a, s3b)) as f32;+ }+ }++ (sum0, sum1, sum2, sum3)+}++#[inline(always)]+pub(crate) fn vec_dot_q3k_q8k(n: usize, xs: &[BlockQ3K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q3k_q8k: {n} is not divisible by {QK_K}"+ );+ let mut sumf = 0f32;+ let mut utmp = [0u32; 4];+ let mut aux = [0u32; 3];+ const KMASK1: u32 = 0x03030303;+ const KMASK2: u32 = 0x0f0f0f0f;++ unsafe {+ let m3b = vdupq_n_u8(0x3);+ let m0 = vdupq_n_u8(1);+ let m1 = vshlq_n_u8(m0, 1);+ let m2 = vshlq_n_u8(m0, 2);+ let m3 = vshlq_n_u8(m0, 3);+ for (x, y) in xs.iter().zip(ys.iter()) {+ let d = y.d * x.d.to_f32();+ let mut q3 = x.qs.as_ptr();+ let qh = x.hmask.as_ptr();+ let mut q8 = y.qs.as_ptr();++ let mut qhbits = vld1q_u8_x2(qh);++ let mut isum = 0i32;++ // Set up scales+ LittleEndian::read_u32_into(&x.scales, &mut aux);++ utmp[3] = ((aux[1] >> 4) & KMASK2) | (((aux[2] >> 6) & KMASK1) << 4);+ utmp[2] = ((aux[0] >> 4) & KMASK2) | (((aux[2] >> 4) & KMASK1) << 4);+ utmp[1] = (aux[1] & KMASK2) | (((aux[2] >> 2) & KMASK1) << 4);+ utmp[0] = (aux[0] & KMASK2) | ((aux[2] & KMASK1) << 4);++ let mut scale = utmp.as_mut_ptr() as *mut i8;+ for j in 0..16 {+ *scale.add(j) -= 32i8+ }++ for j in 0..QK_K / 128 {+ let q3bits = vld1q_u8_x2(q3);+ q3 = q3.add(32);+ let q8bytes_1 = vld1q_s8_x4(q8);+ q8 = q8.add(64);+ let q8bytes_2 = vld1q_s8_x4(q8);+ q8 = q8.add(64);++ let q3h_0 = vshlq_n_u8(vbicq_u8(m0, qhbits.0), 2);+ let q3h_1 = vshlq_n_u8(vbicq_u8(m0, qhbits.1), 2);+ let q3h_2 = vshlq_n_u8(vbicq_u8(m1, qhbits.0), 1);+ let q3h_3 = vshlq_n_u8(vbicq_u8(m1, qhbits.1), 1);++ let q3bytes_0 = vsubq_s8(+ vreinterpretq_s8_u8(vandq_u8(q3bits.0, m3b)),+ vreinterpretq_s8_u8(q3h_0),+ );+ let q3bytes_1 = vsubq_s8(+ vreinterpretq_s8_u8(vandq_u8(q3bits.1, m3b)),+ vreinterpretq_s8_u8(q3h_1),+ );+ let q3bytes_2 = vsubq_s8(+ vreinterpretq_s8_u8(vandq_u8(vshrq_n_u8(q3bits.0, 2), m3b)),+ vreinterpretq_s8_u8(q3h_2),+ );+ let q3bytes_3 = vsubq_s8(+ vreinterpretq_s8_u8(vandq_u8(vshrq_n_u8(q3bits.1, 2), m3b)),+ vreinterpretq_s8_u8(q3h_3),+ );++ let p0 = vdotq_s32(q3bytes_0, q8bytes_1.0);+ let p1 = vdotq_s32(q3bytes_1, q8bytes_1.1);+ let p2 = vdotq_s32(q3bytes_2, q8bytes_1.2);+ let p3 = vdotq_s32(q3bytes_3, q8bytes_1.3);+ isum += vaddvq_s32(p0) * *scale as i32+ + vaddvq_s32(p1) * *scale.add(1) as i32+ + vaddvq_s32(p2) * *scale.add(2) as i32+ + vaddvq_s32(p3) * *scale.add(3) as i32;+ scale = scale.add(4);++ let q3h_0 = vbicq_u8(m2, qhbits.0);+ let q3h_1 = vbicq_u8(m2, qhbits.1);+ let q3h_2 = vshrq_n_u8(vbicq_u8(m3, qhbits.0), 1);+ let q3h_3 = vshrq_n_u8(vbicq_u8(m3, qhbits.1), 1);++ let q3bytes_0 = vsubq_s8(+ vreinterpretq_s8_u8(vandq_u8(vshrq_n_u8(q3bits.0, 4), m3b)),+ vreinterpretq_s8_u8(q3h_0),+ );+ let q3bytes_1 = vsubq_s8(+ vreinterpretq_s8_u8(vandq_u8(vshrq_n_u8(q3bits.1, 4), m3b)),+ vreinterpretq_s8_u8(q3h_1),+ );+ let q3bytes_2 = vsubq_s8(+ vreinterpretq_s8_u8(vandq_u8(vshrq_n_u8(q3bits.0, 6), m3b)),+ vreinterpretq_s8_u8(q3h_2),+ );+ let q3bytes_3 = vsubq_s8(+ vreinterpretq_s8_u8(vandq_u8(vshrq_n_u8(q3bits.1, 6), m3b)),+ vreinterpretq_s8_u8(q3h_3),+ );++ let p0 = vdotq_s32(q3bytes_0, q8bytes_2.0);+ let p1 = vdotq_s32(q3bytes_1, q8bytes_2.1);+ let p2 = vdotq_s32(q3bytes_2, q8bytes_2.2);+ let p3 = vdotq_s32(q3bytes_3, q8bytes_2.3);+ isum += vaddvq_s32(p0) * *scale as i32+ + vaddvq_s32(p1) * *scale.add(1) as i32+ + vaddvq_s32(p2) * *scale.add(2) as i32+ + vaddvq_s32(p3) * *scale.add(3) as i32;+ scale = scale.add(4);++ if j == 0 {+ qhbits.0 = vshrq_n_u8(qhbits.0, 4);+ qhbits.1 = vshrq_n_u8(qhbits.1, 4);+ }+ }+ sumf += d * isum as f32;+ }+ }+ sumf+}++#[inline(always)]+pub(crate) fn vec_dot_q2k_q8k(n: usize, xs: &[BlockQ2K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q2k_q8k: {n} is not divisible by {QK_K}"+ );+ let mut sumf = 0f32;+ let mut aux = [0u8; 16];++ unsafe {+ let m3 = vdupq_n_u8(0x3);+ let m4 = vdupq_n_u8(0xF);++ for (x, y) in xs.iter().zip(ys.iter()) {+ let d = y.d * x.d.to_f32();+ let dmin = -y.d * x.dmin.to_f32();++ let mut q2 = x.qs.as_ptr();+ let mut q8 = y.qs.as_ptr();+ let sc = x.scales.as_ptr();++ let mins_and_scales = vld1q_u8(sc);+ let scales = vandq_u8(mins_and_scales, m4);+ vst1q_u8(aux.as_mut_ptr(), scales);++ let mins = vshrq_n_u8(mins_and_scales, 4);+ let q8sums = vld1q_s16_x2(y.bsums.as_ptr());+ let mins16 = int16x8x2_t(+ vreinterpretq_s16_u16(vmovl_u8(vget_low_u8(mins))),+ vreinterpretq_s16_u16(vmovl_u8(vget_high_u8(mins))),+ );+ let s0 = vaddq_s32(+ vmull_s16(vget_low_s16(mins16.0), vget_low_s16(q8sums.0)),+ vmull_s16(vget_high_s16(mins16.0), vget_high_s16(q8sums.0)),+ );+ let s1 = vaddq_s32(+ vmull_s16(vget_low_s16(mins16.1), vget_low_s16(q8sums.1)),+ vmull_s16(vget_high_s16(mins16.1), vget_high_s16(q8sums.1)),+ );+ sumf += dmin * vaddvq_s32(vaddq_s32(s0, s1)) as f32;++ let mut isum = 0i32;+ let mut is = 0usize;++ // TODO: dotprod+ for _j in 0..QK_K / 128 {+ let q2bits = vld1q_u8_x2(q2);+ q2 = q2.add(32);++ let q8bytes = vld1q_s8_x2(q8);+ q8 = q8.add(32);+ let mut q2bytes = int8x16x2_t(+ vreinterpretq_s8_u8(vandq_u8(q2bits.0, m3)),+ vreinterpretq_s8_u8(vandq_u8(q2bits.1, m3)),+ );+ isum += multiply_accum_with_scale(&aux, is, 0, q2bytes, q8bytes);++ let q8bytes = vld1q_s8_x2(q8);+ q8 = q8.add(32);+ q2bytes.0 = vreinterpretq_s8_u8(vandq_u8(vshrq_n_u8(q2bits.0, 2), m3));+ q2bytes.1 = vreinterpretq_s8_u8(vandq_u8(vshrq_n_u8(q2bits.1, 2), m3));+ isum += multiply_accum_with_scale(&aux, is, 2, q2bytes, q8bytes);++ let q8bytes = vld1q_s8_x2(q8);+ q8 = q8.add(32);+ q2bytes.0 = vreinterpretq_s8_u8(vandq_u8(vshrq_n_u8(q2bits.0, 4), m3));+ q2bytes.1 = vreinterpretq_s8_u8(vandq_u8(vshrq_n_u8(q2bits.1, 4), m3));+ isum += multiply_accum_with_scale(&aux, is, 4, q2bytes, q8bytes);++ let q8bytes = vld1q_s8_x2(q8);+ q8 = q8.add(32);+ q2bytes.0 = vreinterpretq_s8_u8(vandq_u8(vshrq_n_u8(q2bits.0, 6), m3));+ q2bytes.1 = vreinterpretq_s8_u8(vandq_u8(vshrq_n_u8(q2bits.1, 6), m3));+ isum += multiply_accum_with_scale(&aux, is, 6, q2bytes, q8bytes);++ is += 8;+ }+ sumf += d * isum as f32;+ }+ }+ sumf+}++/// Quantize a row of f32 activations into Q8K format+#[inline(always)]+pub(crate) fn quantize_row_q8k(xs: &[f32], ys: &mut [BlockQ8K]) {+ debug_assert!(+ xs.len().is_multiple_of(QK_K),+ "quantize_row_q8k: {} is not a multiple of {QK_K}",+ xs.len()+ );+ unsafe {+ let (x_chunks, _) = xs.as_chunks::<QK_K>();+ for (chunk, y) in x_chunks.iter().zip(ys.iter_mut()) {+ // Find the element with the maximum absolute value, preserving its sign.+ let (mut vabs_max0, mut vsmax0) = (vdupq_n_f32(0.0), vdupq_n_f32(0.0));+ let (mut vabs_max1, mut vsmax1) = (vdupq_n_f32(0.0), vdupq_n_f32(0.0));+ let (mut vabs_max2, mut vsmax2) = (vdupq_n_f32(0.0), vdupq_n_f32(0.0));+ let (mut vabs_max3, mut vsmax3) = (vdupq_n_f32(0.0), vdupq_n_f32(0.0));+ let mut p = chunk.as_ptr();+ for _ in 0..QK_K / 16 {+ let (v0, v1) = (vld1q_f32(p), vld1q_f32(p.add(4)));+ let (v2, v3) = (vld1q_f32(p.add(8)), vld1q_f32(p.add(12)));+ p = p.add(16);+ (vabs_max0, vsmax0) = merge_signed_max(vabs_max0, vsmax0, vabsq_f32(v0), v0);+ (vabs_max1, vsmax1) = merge_signed_max(vabs_max1, vsmax1, vabsq_f32(v1), v1);+ (vabs_max2, vsmax2) = merge_signed_max(vabs_max2, vsmax2, vabsq_f32(v2), v2);+ (vabs_max3, vsmax3) = merge_signed_max(vabs_max3, vsmax3, vabsq_f32(v3), v3);+ }+ // Tree-reduce 4 accumulators to 1.+ let (abs01, smax01) = merge_signed_max(vabs_max0, vsmax0, vabs_max1, vsmax1);+ let (abs23, smax23) = merge_signed_max(vabs_max2, vsmax2, vabs_max3, vsmax3);+ let (abs_v, smax_v) = merge_signed_max(abs01, smax01, abs23, smax23);+ // Cross lane reduce to scalar+ let mask_lohi = vcgt_f32(vget_high_f32(abs_v), vget_low_f32(abs_v));+ let abs_pair = vmax_f32(vget_low_f32(abs_v), vget_high_f32(abs_v));+ let smax_pair = vbsl_f32(mask_lohi, vget_high_f32(smax_v), vget_low_f32(smax_v));+ let max_signed = if vget_lane_f32(abs_pair, 1) > vget_lane_f32(abs_pair, 0) {+ vget_lane_f32(smax_pair, 1)+ } else {+ vget_lane_f32(smax_pair, 0)+ };++ if max_signed == 0.0f32 {+ y.d = 0.0f32;+ y.qs.fill(0);+ y.bsums.fill(0);+ continue;+ }++ let iscale = -127.0f32 / max_signed;+ let vscale = vdupq_n_f32(iscale);++ // Quantize f32 -> i8. Multiply, round-to-nearest, saturating narrow.+ let mut out = y.qs.as_mut_ptr();+ let mut p = chunk.as_ptr();+ for j in 0..QK_K / 16 {+ let f0 = vmulq_f32(vld1q_f32(p), vscale);+ let f1 = vmulq_f32(vld1q_f32(p.add(4)), vscale);+ let f2 = vmulq_f32(vld1q_f32(p.add(8)), vscale);+ let f3 = vmulq_f32(vld1q_f32(p.add(12)), vscale);+ p = p.add(16);+ let s01 = vcombine_s16(+ vqmovn_s32(vcvtaq_s32_f32(f0)),+ vqmovn_s32(vcvtaq_s32_f32(f1)),+ );+ let s23 = vcombine_s16(+ vqmovn_s32(vcvtaq_s32_f32(f2)),+ vqmovn_s32(vcvtaq_s32_f32(f3)),+ );+ let q = vcombine_s8(vqmovn_s16(s01), vqmovn_s16(s23));+ vst1q_s8(out, q);+ out = out.add(16);+ y.bsums[j] = vaddvq_s32(vpaddlq_s16(vpaddlq_s8(q))) as i16;+ }++ y.d = 1.0f32 / iscale;+ }+ }+}++#[inline(always)]+unsafe fn multiply_accum_with_scale(+ aux: &[u8; 16],+ is: usize,+ index: usize,+ q2bytes: int8x16x2_t,+ q8bytes: int8x16x2_t,+) -> i32 {+ let p1 = vdotq_s32(q2bytes.0, q8bytes.0);+ let p2 = vdotq_s32(q2bytes.1, q8bytes.1);+ vaddvq_s32(p1) * aux[is + index] as i32 + vaddvq_s32(p2) * aux[is + 1 + index] as i32+}++#[cfg(target_arch = "aarch64")]+#[target_feature(enable = "dotprod")]+pub(crate) fn vec_dot_8_q4k_q8k(n: usize, xs: &[BlockQ4Kx8], ys: &[BlockQ8K]) -> [f32; 8] {+ debug_assert!(n.is_multiple_of(QK_K));+ let mut out = [0f32; 8];++ #[inline(always)]+ unsafe fn load_f16x4(ptr: *const half::f16) -> float32x4_t {+ let raw = vld1_u64(ptr as *const u64);+ let mut result: float32x4_t;+ core::arch::asm!(+ "fcvtl {out:v}.4s, {inp:v}.4h",+ inp = in(vreg) raw,+ out = out(vreg) result,+ options(nostack, nomem),+ );+ result+ }++ unsafe {+ let mut vacc_0 = vdupq_n_f32(0.0f32);+ let mut vacc_1 = vdupq_n_f32(0.0f32);+ let m4b = vdupq_n_u8(0x0f);+ for (q4, q8) in xs.iter().zip(ys.iter()) {+ let q8d = q8.d;+ let q8d_v = vdupq_n_f32(q8d);+ let sb_scale_0 = vmulq_f32(load_f16x4(q4.d.as_ptr()), q8d_v);+ let sb_scale_1 = vmulq_f32(load_f16x4(q4.d.as_ptr().add(4)), q8d_v);+ let sb_min_0 = vmulq_f32(load_f16x4(q4.dmin.as_ptr()), q8d_v);+ let sb_min_1 = vmulq_f32(load_f16x4(q4.dmin.as_ptr().add(4)), q8d_v);+ let bsums = vpaddq_s16(+ vld1q_s16(q8.bsums.as_ptr()),+ vld1q_s16(q8.bsums.as_ptr().add(8)),+ );+ let mut bias_0 = vdupq_n_s32(0);+ let mut bias_1 = vdupq_n_s32(0);+ macro_rules! process_sb {+ ($sb:literal) => {{+ let (mins0, sc0) = decode_q4kx8_scales(q4.scales.as_ptr().add($sb * 24));+ let (mins1, sc1) = decode_q4kx8_scales(q4.scales.as_ptr().add($sb * 24 + 12));+ let q8p = q8.qs.as_ptr().add($sb * 64);+ let q8_0 = vreinterpretq_s8_s64(vld1q_dup_s64(q8p as *const i64));+ let q8_1 = vreinterpretq_s8_s64(vld1q_dup_s64(q8p.add(8) as *const i64));+ let q8_2 = vreinterpretq_s8_s64(vld1q_dup_s64(q8p.add(16) as *const i64));+ let q8_3 = vreinterpretq_s8_s64(vld1q_dup_s64(q8p.add(24) as *const i64));+ let q8_4 = vreinterpretq_s8_s64(vld1q_dup_s64(q8p.add(32) as *const i64));+ let q8_5 = vreinterpretq_s8_s64(vld1q_dup_s64(q8p.add(40) as *const i64));+ let q8_6 = vreinterpretq_s8_s64(vld1q_dup_s64(q8p.add(48) as *const i64));+ let q8_7 = vreinterpretq_s8_s64(vld1q_dup_s64(q8p.add(56) as *const i64));+ let q4p = q4.qs.as_ptr().add($sb * QK_K);+ let s0 = vld1q_u8_x2(q4p);+ let s0h = vld1q_u8_x2(q4p.add(32));+ let s1 = vld1q_u8_x2(q4p.add(64));+ let s1h = vld1q_u8_x2(q4p.add(96));+ let s2 = vld1q_u8_x2(q4p.add(128));+ let s2h = vld1q_u8_x2(q4p.add(160));+ let s3 = vld1q_u8_x2(q4p.add(192));+ let s3h = vld1q_u8_x2(q4p.add(224));+ let (b00, b10) = (s0.0, s0.1);+ let (b20, b30) = (s0h.0, s0h.1);+ let (b01, b11) = (s1.0, s1.1);+ let (b21, b31) = (s1h.0, s1h.1);+ let (b02, b12) = (s2.0, s2.1);+ let (b22, b32) = (s2h.0, s2h.1);+ let (b03, b13) = (s3.0, s3.1);+ let (b23, b33) = (s3h.0, s3h.1);+ let mut a0 = vdupq_n_s32(0);+ a0 = sdot_acc(a0, vreinterpretq_s8_u8(vandq_u8(b00, m4b)), q8_0);+ a0 = sdot_acc(a0, vreinterpretq_s8_u8(vandq_u8(b01, m4b)), q8_1);+ a0 = sdot_acc(a0, vreinterpretq_s8_u8(vandq_u8(b02, m4b)), q8_2);+ a0 = sdot_acc(a0, vreinterpretq_s8_u8(vandq_u8(b03, m4b)), q8_3);+ let mut h0 = vdupq_n_s32(0);+ h0 = sdot_acc(h0, vreinterpretq_s8_u8(vshrq_n_u8(b00, 4)), q8_4);+ h0 = sdot_acc(h0, vreinterpretq_s8_u8(vshrq_n_u8(b01, 4)), q8_5);+ h0 = sdot_acc(h0, vreinterpretq_s8_u8(vshrq_n_u8(b02, 4)), q8_6);+ h0 = sdot_acc(h0, vreinterpretq_s8_u8(vshrq_n_u8(b03, 4)), q8_7);+ let mut a1 = vdupq_n_s32(0);+ a1 = sdot_acc(a1, vreinterpretq_s8_u8(vandq_u8(b10, m4b)), q8_0);+ a1 = sdot_acc(a1, vreinterpretq_s8_u8(vandq_u8(b11, m4b)), q8_1);+ a1 = sdot_acc(a1, vreinterpretq_s8_u8(vandq_u8(b12, m4b)), q8_2);+ a1 = sdot_acc(a1, vreinterpretq_s8_u8(vandq_u8(b13, m4b)), q8_3);+ let mut h1 = vdupq_n_s32(0);+ h1 = sdot_acc(h1, vreinterpretq_s8_u8(vshrq_n_u8(b10, 4)), q8_4);+ h1 = sdot_acc(h1, vreinterpretq_s8_u8(vshrq_n_u8(b11, 4)), q8_5);+ h1 = sdot_acc(h1, vreinterpretq_s8_u8(vshrq_n_u8(b12, 4)), q8_6);+ h1 = sdot_acc(h1, vreinterpretq_s8_u8(vshrq_n_u8(b13, 4)), q8_7);+ let sumf_lo_03 =+ vcvtq_f32_s32(vmulq_s32(vmovl_s16(vget_low_s16(sc0)), vpaddq_s32(a0, a1)));+ vacc_0 = vfmaq_f32(vacc_0, sb_scale_0, sumf_lo_03);+ let sumf_hi_03 =+ vcvtq_f32_s32(vmulq_s32(vmovl_s16(vget_low_s16(sc1)), vpaddq_s32(h0, h1)));+ vacc_0 = vfmaq_f32(vacc_0, sb_scale_0, sumf_hi_03);+ let mut a2 = vdupq_n_s32(0);+ a2 = sdot_acc(a2, vreinterpretq_s8_u8(vandq_u8(b20, m4b)), q8_0);+ a2 = sdot_acc(a2, vreinterpretq_s8_u8(vandq_u8(b21, m4b)), q8_1);+ a2 = sdot_acc(a2, vreinterpretq_s8_u8(vandq_u8(b22, m4b)), q8_2);+ a2 = sdot_acc(a2, vreinterpretq_s8_u8(vandq_u8(b23, m4b)), q8_3);+ let mut h2 = vdupq_n_s32(0);+ h2 = sdot_acc(h2, vreinterpretq_s8_u8(vshrq_n_u8(b20, 4)), q8_4);+ h2 = sdot_acc(h2, vreinterpretq_s8_u8(vshrq_n_u8(b21, 4)), q8_5);+ h2 = sdot_acc(h2, vreinterpretq_s8_u8(vshrq_n_u8(b22, 4)), q8_6);+ h2 = sdot_acc(h2, vreinterpretq_s8_u8(vshrq_n_u8(b23, 4)), q8_7);+ let mut a3 = vdupq_n_s32(0);+ a3 = sdot_acc(a3, vreinterpretq_s8_u8(vandq_u8(b30, m4b)), q8_0);+ a3 = sdot_acc(a3, vreinterpretq_s8_u8(vandq_u8(b31, m4b)), q8_1);+ a3 = sdot_acc(a3, vreinterpretq_s8_u8(vandq_u8(b32, m4b)), q8_2);+ a3 = sdot_acc(a3, vreinterpretq_s8_u8(vandq_u8(b33, m4b)), q8_3);+ let mut h3 = vdupq_n_s32(0);+ h3 = sdot_acc(h3, vreinterpretq_s8_u8(vshrq_n_u8(b30, 4)), q8_4);+ h3 = sdot_acc(h3, vreinterpretq_s8_u8(vshrq_n_u8(b31, 4)), q8_5);+ h3 = sdot_acc(h3, vreinterpretq_s8_u8(vshrq_n_u8(b32, 4)), q8_6);+ h3 = sdot_acc(h3, vreinterpretq_s8_u8(vshrq_n_u8(b33, 4)), q8_7);+ let sumf_lo_47 =+ vcvtq_f32_s32(vmulq_s32(vmovl_s16(vget_high_s16(sc0)), vpaddq_s32(a2, a3)));+ vacc_1 = vfmaq_f32(vacc_1, sb_scale_1, sumf_lo_47);+ let sumf_hi_47 =+ vcvtq_f32_s32(vmulq_s32(vmovl_s16(vget_high_s16(sc1)), vpaddq_s32(h2, h3)));+ vacc_1 = vfmaq_f32(vacc_1, sb_scale_1, sumf_hi_47);+ let bl = vdup_n_s16(vgetq_lane_s16::<{ $sb * 2 }>(bsums));+ let bh = vdup_n_s16(vgetq_lane_s16::<{ $sb * 2 + 1 }>(bsums));+ bias_0 = vmlal_s16(bias_0, bl, vget_low_s16(mins0));+ bias_0 = vmlal_s16(bias_0, bh, vget_low_s16(mins1));+ bias_1 = vmlal_s16(bias_1, bl, vget_high_s16(mins0));+ bias_1 = vmlal_s16(bias_1, bh, vget_high_s16(mins1));+ }};+ }+ process_sb!(0);+ process_sb!(1);+ process_sb!(2);+ process_sb!(3);+ vacc_0 = vmlsq_f32(vacc_0, vcvtq_f32_s32(bias_0), sb_min_0);+ vacc_1 = vmlsq_f32(vacc_1, vcvtq_f32_s32(bias_1), sb_min_1);+ }+ vst1q_f32(out.as_mut_ptr(), vacc_0);+ vst1q_f32(out.as_mut_ptr().add(4), vacc_1);+ }+ out+}diff --git a/vendor/candle-core/src/quantized/repack.rs b/vendor/candle-core/src/quantized/repack.rsnew file mode 100644index 0000000..0ea3c49--- /dev/null+++ b/vendor/candle-core/src/quantized/repack.rs@@ -0,0 +1,1083 @@+#[cfg(target_arch = "aarch64")]+use super::GgmlDType;+use super::QuantizedType;+use crate::Result;+use std::sync::OnceLock;++#[cfg(target_arch = "aarch64")]+use super::k_quants::{BlockQ4K, BlockQ4_0, QK4_0};+#[cfg(target_arch = "aarch64")]+use super::k_quants::{BlockQ5K, BlockQ6K, QK_K};+#[cfg(target_arch = "aarch64")]+use super::k_quants::{BlockQ8_0, QK8_0};++#[cfg(target_arch = "aarch64")]+use half::f16;++#[derive(Clone, Copy, Debug, PartialEq, Eq)]+#[cfg(target_arch = "aarch64")]+enum PackedKind {+ Q4_0x4,+ Q4Kx8,+ Q5Kx8,+ Q6Kx8,+ Q8_0x4,+}++#[cfg(target_arch = "aarch64")]+pub(crate) enum PackedStorage {+ Q4_0x4(Vec<BlockQ4_0x4>),+ Q4Kx8(Vec<BlockQ4Kx8>),+ Q5Kx8(Vec<BlockQ5Kx8>),+ Q6Kx8(Vec<BlockQ6Kx8>),+ Q8_0x4(Vec<BlockQ8_0x4>),+}++#[allow(dead_code)]+pub(crate) struct PackedCache {+ #[cfg(target_arch = "x86_64")]+ x86: OnceLock<super::repack_x86::PackedX86>,+ #[cfg(target_arch = "x86_64")]+ x86_amx: OnceLock<Vec<super::repack_x86::TileQ4KAmx>>,+ #[cfg(target_arch = "aarch64")]+ q4_0x4: OnceLock<PackedStorage>,+ #[cfg(target_arch = "aarch64")]+ q4kx8: OnceLock<PackedStorage>,+ #[cfg(target_arch = "aarch64")]+ q5kx8: OnceLock<PackedStorage>,+ #[cfg(target_arch = "aarch64")]+ q6kx8: OnceLock<PackedStorage>,+ #[cfg(target_arch = "aarch64")]+ q8_0x4: OnceLock<PackedStorage>,+}++impl PackedCache {+ #[cfg(target_arch = "x86_64")]+ pub(crate) fn x86_get_or_init(+ &self,+ init: impl FnOnce() -> super::repack_x86::PackedX86,+ ) -> &super::repack_x86::PackedX86 {+ self.x86.get_or_init(init)+ }++ pub(crate) fn new() -> Self {+ Self {+ #[cfg(target_arch = "x86_64")]+ x86: OnceLock::new(),+ #[cfg(target_arch = "x86_64")]+ x86_amx: OnceLock::new(),+ #[cfg(target_arch = "aarch64")]+ q4_0x4: OnceLock::new(),+ #[cfg(target_arch = "aarch64")]+ q4kx8: OnceLock::new(),+ #[cfg(target_arch = "aarch64")]+ q5kx8: OnceLock::new(),+ #[cfg(target_arch = "aarch64")]+ q6kx8: OnceLock::new(),+ #[cfg(target_arch = "aarch64")]+ q8_0x4: OnceLock::new(),+ }+ }++ #[cfg(target_arch = "aarch64")]+ fn get_or_init(+ &self,+ kind: PackedKind,+ init: impl FnOnce() -> PackedStorage,+ ) -> &PackedStorage {+ match kind {+ #[cfg(target_arch = "aarch64")]+ PackedKind::Q4_0x4 => self.q4_0x4.get_or_init(init),+ #[cfg(target_arch = "aarch64")]+ PackedKind::Q4Kx8 => self.q4kx8.get_or_init(init),+ #[cfg(target_arch = "aarch64")]+ PackedKind::Q5Kx8 => self.q5kx8.get_or_init(init),+ #[cfg(target_arch = "aarch64")]+ PackedKind::Q6Kx8 => self.q6kx8.get_or_init(init),+ #[cfg(target_arch = "aarch64")]+ PackedKind::Q8_0x4 => self.q8_0x4.get_or_init(init),+ }+ }+}++#[cfg(target_arch = "aarch64")]+#[derive(+ Clone,+ Copy,+ zerocopy::FromBytes,+ zerocopy::IntoBytes,+ zerocopy::KnownLayout,+ zerocopy::Immutable,+)]+#[repr(C)]+pub(crate) struct BlockQ8_0x4 {+ pub(crate) d: [f16; 4],+ pub(crate) qs: [i8; QK8_0 * 4],+}++#[cfg(target_arch = "aarch64")]+const _: () = assert!(std::mem::size_of::<BlockQ8_0x4>() == 136);++#[cfg(target_arch = "aarch64")]+#[derive(+ Clone,+ Copy,+ zerocopy::FromBytes,+ zerocopy::IntoBytes,+ zerocopy::KnownLayout,+ zerocopy::Immutable,+)]+#[repr(C)]+pub(crate) struct BlockQ4_0x4 {+ pub(crate) d: [f16; 4],+ pub(crate) qs: [i8; QK4_0 * 2],+}++#[cfg(target_arch = "aarch64")]+const _: () = assert!(std::mem::size_of::<BlockQ4_0x4>() == 72);++#[cfg(target_arch = "aarch64")]+#[derive(+ Clone,+ Copy,+ zerocopy::FromBytes,+ zerocopy::IntoBytes,+ zerocopy::KnownLayout,+ zerocopy::Immutable,+)]+#[repr(C)]+pub(crate) struct BlockQ4Kx8 {+ pub(crate) d: [f16; 8],+ pub(crate) dmin: [f16; 8],+ pub(crate) scales: [u8; 96],+ pub(crate) qs: [u8; 1024],+}++#[cfg(target_arch = "aarch64")]+const _: () = assert!(std::mem::size_of::<BlockQ4Kx8>() == 1152);++#[cfg(target_arch = "aarch64")]+#[derive(+ Clone,+ Copy,+ zerocopy::FromBytes,+ zerocopy::IntoBytes,+ zerocopy::KnownLayout,+ zerocopy::Immutable,+)]+#[repr(C)]+pub(crate) struct BlockQ5Kx8 {+ pub(crate) d: [f16; 8],+ pub(crate) dmin: [f16; 8],+ pub(crate) scales: [u8; 96],+ pub(crate) qh: [u8; QK_K],+ pub(crate) qs: [u8; QK_K / 2 * 8],+}++#[cfg(target_arch = "aarch64")]+const _: () = assert!(std::mem::size_of::<BlockQ5Kx8>() == 1408);++#[cfg(target_arch = "aarch64")]+#[derive(+ Clone,+ Copy,+ zerocopy::FromBytes,+ zerocopy::IntoBytes,+ zerocopy::KnownLayout,+ zerocopy::Immutable,+)]+#[repr(C)]+pub(crate) struct BlockQ6Kx8 {+ pub(crate) d: [f16; 8],+ pub(crate) scales: [i8; QK_K / 16 * 8],+ pub(crate) ql: [u8; QK_K / 2 * 8],+ pub(crate) qh: [u8; QK_K / 4 * 8],+}++#[cfg(target_arch = "aarch64")]+const _: () = assert!(std::mem::size_of::<BlockQ6Kx8>() == 1680);++#[cfg(target_arch = "aarch64")]+#[derive(+ Clone,+ Copy,+ zerocopy::FromBytes,+ zerocopy::IntoBytes,+ zerocopy::KnownLayout,+ zerocopy::Immutable,+)]+#[repr(C)]+pub(crate) struct BlockQ8Kx4 {+ pub(crate) d: [f32; 4],+ pub(crate) qs: [i8; QK_K * 4],+ pub(crate) bsums: [i16; QK_K / 4],+}++#[cfg(target_arch = "aarch64")]+const _: () = assert!(std::mem::size_of::<BlockQ8Kx4>() == 1168);++pub(crate) fn try_matmul_f32(+ storage: &dyn QuantizedType,+ packed: &PackedCache,+ mkn: (usize, usize, usize),+ lhs: &[f32],+ dst: &mut [f32],+) -> Result<bool> {+ #[cfg(target_arch = "aarch64")]+ {+ let (m, k, n) = mkn;+ let dtype = storage.dtype();+ if let Some(kind) = PackedKind::select(dtype, mkn) {+ let packed = packed.get_or_init(kind, || kind.pack(storage, n));+ kind.matmul(mkn, lhs, packed, dst)?;+ return Ok(true);+ }+ // tiled kernels need m % 4 == 0: run the aligned rows tiled and the tail rows as+ // gemvs on the same packing instead of dropping the whole matmul to the unpacked path+ let Some(kind) = PackedKind::select(dtype, (1, k, n)) else {+ return Ok(false);+ };+ let m_tiled = m & !3;+ if m_tiled > 0 && PackedKind::select(dtype, (m_tiled, k, n)) != Some(kind) {+ return Ok(false);+ }+ let packed = packed.get_or_init(kind, || kind.pack(storage, n));+ let (lhs_tiled, lhs_tail) = lhs.split_at(m_tiled * k);+ let (dst_tiled, dst_tail) = dst.split_at_mut(m_tiled * n);+ if m_tiled > 0 {+ kind.matmul((m_tiled, k, n), lhs_tiled, packed, dst_tiled)?;+ }+ for (lhs_row, dst_row) in lhs_tail.chunks_exact(k).zip(dst_tail.chunks_exact_mut(n)) {+ kind.matmul((1, k, n), lhs_row, packed, dst_row)?;+ }+ Ok(true)+ }++ #[cfg(target_arch = "x86_64")]+ {+ let (m, k, n) = mkn;+ if !super::repack_x86::select(storage.dtype(), m, n, k) {+ return Ok(false);+ }+ let dtype = storage.dtype();+ let lhs_q = super::repack_x86::quantize_lhs(lhs, m, k);+ if m >= 32 && dtype == super::GgmlDType::Q4K && super::repack_x86::amx_available() {+ let tiles = packed+ .x86_amx+ .get_or_init(|| super::repack_x86::pack_q4k_amx(storage, n, k));+ super::repack_x86::matmul_amx_q4k(tiles, &lhs_q, m, k, n, dst);+ return Ok(true);+ }+ let tiles = packed+ .x86+ .get_or_init(|| super::repack_x86::pack(dtype, storage, n, k));+ super::repack_x86::matmul_tiles(tiles, &lhs_q, m, k, n, dst);+ Ok(true)+ }++ #[cfg(not(any(target_arch = "aarch64", target_arch = "x86_64")))]+ {+ let _ = (storage, packed, mkn, lhs, dst);+ Ok(false)+ }+}++#[cfg(target_arch = "aarch64")]+impl PackedKind {+ fn select(dtype: GgmlDType, (m, _k, n): (usize, usize, usize)) -> Option<Self> {+ let features = crate::cpu::features::get();+ let has_dotprod_gemv = features.dotprod && m == 1;+ let has_tiled_matmul = (features.dotprod || features.i8mm) && m >= 4 && m.is_multiple_of(4);+ match dtype {+ GgmlDType::Q4_0 if n.is_multiple_of(4) && (has_dotprod_gemv || has_tiled_matmul) => {+ Some(Self::Q4_0x4)+ }+ GgmlDType::Q4K if n.is_multiple_of(8) && (has_dotprod_gemv || has_tiled_matmul) => {+ Some(Self::Q4Kx8)+ }+ GgmlDType::Q5K if n.is_multiple_of(8) && (has_dotprod_gemv || has_tiled_matmul) => {+ Some(Self::Q5Kx8)+ }+ GgmlDType::Q6K if n.is_multiple_of(8) && (has_dotprod_gemv || has_tiled_matmul) => {+ Some(Self::Q6Kx8)+ }+ GgmlDType::Q8_0 if n.is_multiple_of(4) && (has_dotprod_gemv || has_tiled_matmul) => {+ Some(Self::Q8_0x4)+ }+ _ => None,+ }+ }++ fn pack(self, storage: &dyn QuantizedType, n: usize) -> PackedStorage {+ let interleave = if crate::cpu::features::get().i8mm {+ 8+ } else {+ 4+ };+ match self {+ #[cfg(target_arch = "aarch64")]+ Self::Q4_0x4 => {+ let total_blocks =+ storage.storage_size_in_bytes() / std::mem::size_of::<BlockQ4_0>();+ let blocks = unsafe {+ std::slice::from_raw_parts(storage.as_ptr() as *const BlockQ4_0, total_blocks)+ };+ PackedStorage::Q4_0x4(pack_to_q4_0x4(blocks, n, interleave))+ }+ #[cfg(target_arch = "aarch64")]+ Self::Q4Kx8 => {+ let total_blocks =+ storage.storage_size_in_bytes() / std::mem::size_of::<BlockQ4K>();+ let blocks = unsafe {+ std::slice::from_raw_parts(storage.as_ptr() as *const BlockQ4K, total_blocks)+ };+ PackedStorage::Q4Kx8(pack_to_q4kx8(blocks, n))+ }+ #[cfg(target_arch = "aarch64")]+ Self::Q5Kx8 => {+ let total_blocks =+ storage.storage_size_in_bytes() / std::mem::size_of::<BlockQ5K>();+ let blocks = unsafe {+ std::slice::from_raw_parts(storage.as_ptr() as *const BlockQ5K, total_blocks)+ };+ PackedStorage::Q5Kx8(pack_to_q5kx8(blocks, n, interleave))+ }+ #[cfg(target_arch = "aarch64")]+ Self::Q6Kx8 => {+ let total_blocks =+ storage.storage_size_in_bytes() / std::mem::size_of::<BlockQ6K>();+ let blocks = unsafe {+ std::slice::from_raw_parts(storage.as_ptr() as *const BlockQ6K, total_blocks)+ };+ PackedStorage::Q6Kx8(pack_to_q6kx8(blocks, n, interleave))+ }+ #[cfg(target_arch = "aarch64")]+ Self::Q8_0x4 => {+ let total_blocks =+ storage.storage_size_in_bytes() / std::mem::size_of::<BlockQ8_0>();+ let blocks = unsafe {+ std::slice::from_raw_parts(storage.as_ptr() as *const BlockQ8_0, total_blocks)+ };+ PackedStorage::Q8_0x4(pack_to_q8_0x4(blocks, n, interleave))+ }+ }+ }++ // matmul against one expert's row block inside stacked packed storage; subslices+ // per arm so nothing is copied+ #[cfg(target_arch = "aarch64")]+ fn matmul_expert_range(+ self,+ mkn: (usize, usize, usize),+ lhs: &[f32],+ packed: &PackedStorage,+ expert: usize,+ dst: &mut [f32],+ ) -> Result<()> {+ let (_m, k, n_out) = mkn;+ let features = crate::cpu::features::get();+ macro_rules! sub {+ ($v:expr, $width:expr, $qk:expr) => {{+ let kb = k / $qk;+ let groups = n_out / $width;+ &$v[expert * groups * kb..(expert + 1) * groups * kb]+ }};+ }+ match (self, packed) {+ (Self::Q4Kx8, PackedStorage::Q4Kx8(v)) => {+ let p = sub!(v, 8, QK_K);+ if features.i8mm && mkn.0 >= 4 && mkn.0.is_multiple_of(4) {+ unsafe { super::neon::matmul_q4k_x8_i8mm(mkn, lhs, p, dst) }+ } else {+ unsafe { super::k_quants::matmul_q4k_x8(mkn, lhs, p, dst) }+ }+ }+ (Self::Q6Kx8, PackedStorage::Q6Kx8(v)) => {+ super::neon::matmul_q6k_x8(mkn, lhs, sub!(v, 8, QK_K), dst)+ }+ (Self::Q8_0x4, PackedStorage::Q8_0x4(v)) => {+ let p = sub!(v, 4, QK8_0);+ if features.i8mm && mkn.0 >= 4 && mkn.0.is_multiple_of(4) {+ unsafe { super::neon::matmul_q8_0_x4_i8mm(mkn, lhs, p, dst) }+ } else {+ unsafe { super::neon::matmul_q8_0_x4(mkn, lhs, p, dst) }+ }+ }+ _ => crate::bail!("unsupported expert-range kind"),+ }+ }++ fn matmul(+ self,+ mkn: (usize, usize, usize),+ lhs: &[f32],+ packed: &PackedStorage,+ dst: &mut [f32],+ ) -> Result<()> {+ let features = crate::cpu::features::get();+ match (self, packed) {+ #[cfg(target_arch = "aarch64")]+ (Self::Q4_0x4, PackedStorage::Q4_0x4(packed)) => {+ if features.i8mm && mkn.0 >= 4 {+ unsafe { super::neon::matmul_q4_0_x4_i8mm(mkn, lhs, packed, dst) }+ } else {+ unsafe { super::neon::matmul_q4_0_x4(mkn, lhs, packed, dst) }+ }+ }+ #[cfg(target_arch = "aarch64")]+ (Self::Q4Kx8, PackedStorage::Q4Kx8(packed)) => {+ if features.i8mm && mkn.0 >= 4 {+ unsafe { super::neon::matmul_q4k_x8_i8mm(mkn, lhs, packed, dst) }+ } else {+ unsafe { super::k_quants::matmul_q4k_x8(mkn, lhs, packed, dst) }+ }+ }+ #[cfg(target_arch = "aarch64")]+ (Self::Q5Kx8, PackedStorage::Q5Kx8(packed)) => {+ super::neon::matmul_q5k_x8(mkn, lhs, packed, dst)+ }+ #[cfg(target_arch = "aarch64")]+ (Self::Q6Kx8, PackedStorage::Q6Kx8(packed)) => {+ super::neon::matmul_q6k_x8(mkn, lhs, packed, dst)+ }+ #[cfg(target_arch = "aarch64")]+ (Self::Q8_0x4, PackedStorage::Q8_0x4(packed)) => {+ if features.i8mm && mkn.0 >= 4 {+ unsafe { super::neon::matmul_q8_0_x4_i8mm(mkn, lhs, packed, dst) }+ } else {+ unsafe { super::neon::matmul_q8_0_x4(mkn, lhs, packed, dst) }+ }+ }+ #[allow(unreachable_patterns)]+ _ => crate::bail!("packed storage kind mismatch"),+ }+ }+}++#[cfg(target_arch = "aarch64")]+fn pack_to_q4kx8(blocks: &[BlockQ4K], n: usize) -> Vec<BlockQ4Kx8> {+ debug_assert!(n.is_multiple_of(8));+ debug_assert_eq!(blocks.len() % n, 0);+ let k_blocks = blocks.len() / n;+ let n_groups = n / 8;+ let mut packed = Vec::with_capacity(n_groups * k_blocks);+ for g in 0..n_groups {+ for b in 0..k_blocks {+ let src: [&BlockQ4K; 8] = std::array::from_fn(|i| &blocks[(g * 8 + i) * k_blocks + b]);+ let mut p = BlockQ4Kx8 {+ d: [f16::ZERO; 8],+ dmin: [f16::ZERO; 8],+ scales: [0; 96],+ qs: [0; 1024],+ };+ for (i, s) in src.iter().enumerate() {+ p.d[i] = s.d;+ p.dmin[i] = s.dmin;+ }+ for i in 0..128usize {+ let src_id = i % 8;+ let src_offset = (i / 8) * 8;+ let dst_offset = i * 8;+ p.qs[dst_offset..dst_offset + 8]+ .copy_from_slice(&src[src_id].qs[src_offset..src_offset + 8]);+ }+ pack_k_scale_min_x8(&src.map(|s| &s.scales), &mut p.scales);+ packed.push(p);+ }+ }+ packed+}++#[cfg(target_arch = "aarch64")]+fn pack_to_q5kx8(blocks: &[BlockQ5K], n: usize, block_len: usize) -> Vec<BlockQ5Kx8> {+ debug_assert!(n.is_multiple_of(8));+ debug_assert!((QK_K / 2).is_multiple_of(block_len));+ debug_assert!((QK_K / 8).is_multiple_of(block_len));+ debug_assert_eq!(blocks.len() % n, 0);+ let k_blocks = blocks.len() / n;+ let n_groups = n / 8;+ let mut packed = Vec::with_capacity(n_groups * k_blocks);+ for g in 0..n_groups {+ for b in 0..k_blocks {+ let src: [&BlockQ5K; 8] = std::array::from_fn(|i| &blocks[(g * 8 + i) * k_blocks + b]);+ let mut p = BlockQ5Kx8 {+ d: [f16::ZERO; 8],+ dmin: [f16::ZERO; 8],+ scales: [0; 96],+ qh: [0; QK_K],+ qs: [0; QK_K / 2 * 8],+ };+ for (i, s) in src.iter().enumerate() {+ p.d[i] = s.d;+ p.dmin[i] = s.dmin;+ }+ for i in 0..(QK_K / 2 * 8) / block_len {+ let src_id = i % 8;+ let src_offset = (i / 8) * block_len;+ let dst_offset = i * block_len;+ p.qs[dst_offset..dst_offset + block_len]+ .copy_from_slice(&src[src_id].qs[src_offset..src_offset + block_len]);+ }+ for i in 0..(QK_K / 8 * 8) / block_len {+ let src_id = i % 8;+ let src_offset = (i / 8) * block_len;+ let dst_offset = i * block_len;+ p.qh[dst_offset..dst_offset + block_len]+ .copy_from_slice(&src[src_id].qh[src_offset..src_offset + block_len]);+ }+ pack_k_scale_min_x8(&src.map(|s| &s.scales), &mut p.scales);+ packed.push(p);+ }+ }+ packed+}++#[cfg(target_arch = "aarch64")]+fn pack_to_q6kx8(blocks: &[BlockQ6K], n: usize, block_len: usize) -> Vec<BlockQ6Kx8> {+ debug_assert!(n.is_multiple_of(8));+ debug_assert!((QK_K / 2).is_multiple_of(block_len));+ debug_assert!((QK_K / 4).is_multiple_of(block_len));+ debug_assert_eq!(blocks.len() % n, 0);+ let k_blocks = blocks.len() / n;+ let n_groups = n / 8;+ let mut packed = Vec::with_capacity(n_groups * k_blocks);+ for g in 0..n_groups {+ for b in 0..k_blocks {+ let src: [&BlockQ6K; 8] = std::array::from_fn(|i| &blocks[(g * 8 + i) * k_blocks + b]);+ let mut p = BlockQ6Kx8 {+ d: [f16::ZERO; 8],+ scales: [0; QK_K / 16 * 8],+ ql: [0; QK_K / 2 * 8],+ qh: [0; QK_K / 4 * 8],+ };+ for (i, s) in src.iter().enumerate() {+ p.d[i] = s.d;+ }+ for i in 0..(QK_K / 2 * 8) / block_len {+ let src_id = i % 8;+ let src_offset = (i / 8) * block_len;+ let dst_offset = i * block_len;+ p.ql[dst_offset..dst_offset + block_len]+ .copy_from_slice(&src[src_id].ql[src_offset..src_offset + block_len]);+ }+ for i in 0..(QK_K / 4 * 8) / block_len {+ let src_id = i % 8;+ let src_offset = (i / 8) * block_len;+ let dst_offset = i * block_len;+ p.qh[dst_offset..dst_offset + block_len]+ .copy_from_slice(&src[src_id].qh[src_offset..src_offset + block_len]);+ }+ for (col, s) in src.iter().enumerate() {+ for j in 0..QK_K / 16 {+ p.scales[j * 8 + col] = s.scales[j];+ }+ }+ packed.push(p);+ }+ }+ packed+}++#[cfg(target_arch = "aarch64")]+fn pack_k_scale_min_x8(src: &[&[u8; 12]; 8], dst: &mut [u8; 96]) {+ for i in 0..4usize {+ let mut s = [0u8; 8];+ let mut m = [0u8; 8];+ for j in 0..8 {+ s[j] = src[j][i] & 63;+ m[j] = src[j][i + 4] & 63;+ }+ let b12 = i * 12;+ dst[b12] = (s[0] & 63) + ((s[4] & 48) << 2);+ dst[b12 + 1] = (s[1] & 63) + ((s[5] & 48) << 2);+ dst[b12 + 2] = (s[2] & 63) + ((s[6] & 48) << 2);+ dst[b12 + 3] = (s[3] & 63) + ((s[7] & 48) << 2);+ dst[b12 + 4] = (m[0] & 63) + ((m[4] & 48) << 2);+ dst[b12 + 5] = (m[1] & 63) + ((m[5] & 48) << 2);+ dst[b12 + 6] = (m[2] & 63) + ((m[6] & 48) << 2);+ dst[b12 + 7] = (m[3] & 63) + ((m[7] & 48) << 2);+ dst[b12 + 8] = (s[4] & 15) + ((m[4] & 15) << 4);+ dst[b12 + 9] = (s[5] & 15) + ((m[5] & 15) << 4);+ dst[b12 + 10] = (s[6] & 15) + ((m[6] & 15) << 4);+ dst[b12 + 11] = (s[7] & 15) + ((m[7] & 15) << 4);+ }+ for i in 0..4usize {+ let mut s = [0u8; 8];+ let mut m = [0u8; 8];+ for j in 0..8 {+ s[j] = ((src[j][i] & 192) >> 2) | (src[j][i + 8] & 15);+ m[j] = ((src[j][i + 4] & 192) >> 2) | ((src[j][i + 8] & 240) >> 4);+ }+ let b12 = i * 12 + 48;+ dst[b12] = (s[0] & 63) + ((s[4] & 48) << 2);+ dst[b12 + 1] = (s[1] & 63) + ((s[5] & 48) << 2);+ dst[b12 + 2] = (s[2] & 63) + ((s[6] & 48) << 2);+ dst[b12 + 3] = (s[3] & 63) + ((s[7] & 48) << 2);+ dst[b12 + 4] = (m[0] & 63) + ((m[4] & 48) << 2);+ dst[b12 + 5] = (m[1] & 63) + ((m[5] & 48) << 2);+ dst[b12 + 6] = (m[2] & 63) + ((m[6] & 48) << 2);+ dst[b12 + 7] = (m[3] & 63) + ((m[7] & 48) << 2);+ dst[b12 + 8] = (s[4] & 15) + ((m[4] & 15) << 4);+ dst[b12 + 9] = (s[5] & 15) + ((m[5] & 15) << 4);+ dst[b12 + 10] = (s[6] & 15) + ((m[6] & 15) << 4);+ dst[b12 + 11] = (s[7] & 15) + ((m[7] & 15) << 4);+ }+}++#[cfg(target_arch = "aarch64")]+fn pack_to_q4_0x4(blocks: &[BlockQ4_0], n: usize, block_len: usize) -> Vec<BlockQ4_0x4> {+ debug_assert!(n.is_multiple_of(4));+ debug_assert!((QK4_0 / 2).is_multiple_of(block_len));+ debug_assert_eq!(blocks.len() % n, 0);+ let k_blocks = blocks.len() / n;+ let n_groups = n / 4;+ let mut packed = Vec::with_capacity(n_groups * k_blocks);+ for g in 0..n_groups {+ for b in 0..k_blocks {+ let src: [&BlockQ4_0; 4] = std::array::from_fn(|i| &blocks[(g * 4 + i) * k_blocks + b]);+ let mut p = BlockQ4_0x4 {+ d: [f16::ZERO; 4],+ qs: [0; QK4_0 * 2],+ };+ for (i, s) in src.iter().enumerate() {+ p.d[i] = s.d;+ }+ for i in 0..(QK4_0 / 2) / block_len {+ let offset = i * block_len;+ let dst_offset = i * block_len * 4;+ for (j, s) in src.iter().enumerate() {+ let dst_offset = dst_offset + j * block_len;+ for k in 0..block_len {+ p.qs[dst_offset + k] = (s.qs[offset + k] ^ 0x88) as i8;+ }+ }+ }+ packed.push(p);+ }+ }+ packed+}++#[cfg(target_arch = "aarch64")]+fn pack_to_q8_0x4(blocks: &[BlockQ8_0], n: usize, block_len: usize) -> Vec<BlockQ8_0x4> {+ debug_assert!(n.is_multiple_of(4));+ debug_assert!(QK8_0.is_multiple_of(block_len));+ debug_assert_eq!(blocks.len() % n, 0);+ let k_blocks = blocks.len() / n;+ let n_groups = n / 4;+ let mut packed = Vec::with_capacity(n_groups * k_blocks);+ for g in 0..n_groups {+ for b in 0..k_blocks {+ let src: [&BlockQ8_0; 4] = std::array::from_fn(|i| &blocks[(g * 4 + i) * k_blocks + b]);+ let mut p = BlockQ8_0x4 {+ d: [f16::ZERO; 4],+ qs: [0; QK8_0 * 4],+ };+ for (i, s) in src.iter().enumerate() {+ p.d[i] = s.d;+ }+ for i in 0..QK8_0 / block_len {+ let offset = i * block_len;+ let dst_offset = i * block_len * 4;+ for (j, s) in src.iter().enumerate() {+ let dst_offset = dst_offset + j * block_len;+ p.qs[dst_offset..dst_offset + block_len]+ .copy_from_slice(&s.qs[offset..offset + block_len]);+ }+ }+ packed.push(p);+ }+ }+ packed+}++// One lhs quantization and one barrier region for several same-dtype m==1 matmuls+// sharing the lhs (qkv / gate+up in decode), instead of one of each per projection.+#[cfg(target_arch = "aarch64")]+macro_rules! define_gemv_fused {+ ($fname:ident, $blk:ty, $variant:ident, $kernel:path, $w:expr, $lhs_blk:ty, $qk:expr) => {+ #[target_feature(enable = "dotprod")]+ unsafe fn $fname(+ kind: PackedKind,+ k: usize,+ lhs: &[f32],+ parts: &[(&dyn QuantizedType, &PackedCache)],+ dsts: &mut [Vec<f32>],+ ) {+ use super::GgmlType as _;+ let k_in_blocks = k / $qk;++ thread_local! {+ static FUSED_LHS_SCRATCH: std::cell::RefCell<Vec<u64>> =+ const { std::cell::RefCell::new(Vec::new()) };+ }+ FUSED_LHS_SCRATCH.with(|cell| {+ let mut scratch = cell.borrow_mut();+ let elem_size = std::mem::size_of::<$lhs_blk>();+ let required_len = (k_in_blocks * elem_size).div_ceil(8);+ if scratch.len() < required_len {+ scratch.resize(required_len, 0);+ }+ let lhs_b: &mut [$lhs_blk] = unsafe {+ std::slice::from_raw_parts_mut(+ scratch.as_mut_ptr() as *mut $lhs_blk,+ k_in_blocks,+ )+ };+ <$lhs_blk>::from_float(lhs, lhs_b);++ // (packed ptr, n_groups, dst ptr, running offset) per projection+ let mut metas: Vec<(usize, usize, usize, usize)> = Vec::with_capacity(parts.len());+ let mut total = 0usize;+ for (i, (storage, cache)) in parts.iter().enumerate() {+ let n = dsts[i].len();+ let ps = cache.get_or_init(kind, || kind.pack(*storage, n));+ let PackedStorage::$variant(v) = ps else {+ return;+ };+ metas.push((+ v.as_ptr() as usize,+ n / $w,+ dsts[i].as_mut_ptr() as usize,+ total,+ ));+ total += n / $w;+ }++ let blk_bytes = std::mem::size_of::<$blk>();+ let lhs_ptr = lhs_b.as_ptr() as usize;+ let pool = crate::utils::barrier_pool();+ // equal-sized gemv units: static split, as in the single-projection gemvs+ pool.execute_static(total, |range| {+ let lhs_row: &[$lhs_blk] = unsafe {+ std::slice::from_raw_parts(lhs_ptr as *const $lhs_blk, k_in_blocks)+ };+ for gg in range {+ let &(pptr, _ng, dptr, off) = metas+ .iter()+ .find(|&&(_, ng, _, off)| gg < off + ng)+ .unwrap();+ let g = gg - off;+ let xs = unsafe {+ std::slice::from_raw_parts(+ (pptr + g * k_in_blocks * blk_bytes) as *const $blk,+ k_in_blocks,+ )+ };+ let r = $kernel(k, xs, lhs_row);+ unsafe {+ std::ptr::copy_nonoverlapping(+ r.as_ptr(),+ (dptr as *mut f32).add(g * $w),+ $w,+ );+ }+ }+ });+ });+ }+ };+}++#[cfg(target_arch = "aarch64")]+define_gemv_fused!(+ gemv_fused_q4k,+ BlockQ4Kx8,+ Q4Kx8,+ super::neon::vec_dot_8_q4k_q8k,+ 8,+ super::k_quants::BlockQ8K,+ QK_K+);+#[cfg(target_arch = "aarch64")]+define_gemv_fused!(+ gemv_fused_q6k,+ BlockQ6Kx8,+ Q6Kx8,+ super::neon::vec_dot_8_q6kx8_q8k,+ 8,+ super::k_quants::BlockQ8K,+ QK_K+);+#[cfg(target_arch = "aarch64")]+define_gemv_fused!(+ gemv_fused_q8_0,+ BlockQ8_0x4,+ Q8_0x4,+ super::neon::vec_dot_4_q8_0x4_q8_0,+ 4,+ BlockQ8_0,+ QK8_0+);++#[cfg(target_arch = "aarch64")]+pub(crate) fn try_gemv_fused(+ k: usize,+ lhs: &[f32],+ parts: &[(&dyn QuantizedType, &PackedCache)],+ dsts: &mut [Vec<f32>],+) -> Result<bool> {+ #[cfg(target_arch = "aarch64")]+ {+ if parts.is_empty() || parts.len() != dsts.len() {+ return Ok(false);+ }+ let features = crate::cpu::features::get();+ if !features.dotprod {+ return Ok(false);+ }+ let dtype = parts[0].0.dtype();+ if parts.iter().any(|(s, _)| s.dtype() != dtype) {+ return Ok(false);+ }+ let Some(kind) = PackedKind::select(dtype, (1, k, dsts[0].len())) else {+ return Ok(false);+ };+ for d in dsts.iter() {+ if PackedKind::select(dtype, (1, k, d.len())) != Some(kind) {+ return Ok(false);+ }+ }+ match kind {+ PackedKind::Q4Kx8 if k.is_multiple_of(QK_K) => {+ unsafe { gemv_fused_q4k(kind, k, lhs, parts, dsts) };+ Ok(true)+ }+ PackedKind::Q6Kx8 if k.is_multiple_of(QK_K) => {+ unsafe { gemv_fused_q6k(kind, k, lhs, parts, dsts) };+ Ok(true)+ }+ PackedKind::Q8_0x4 if k.is_multiple_of(QK8_0) => {+ unsafe { gemv_fused_q8_0(kind, k, lhs, parts, dsts) };+ Ok(true)+ }+ _ => Ok(false),+ }+ }+ #[cfg(not(target_arch = "aarch64"))]+ {+ let _ = (k, lhs, parts, dsts);+ Ok(false)+ }+}++// Indexed (MoE) gemv: each (token, expert) pair runs a gemv against one expert's row+// block inside the packed [n_experts * n_out, k] weights, all in one barrier region.+#[cfg(target_arch = "aarch64")]+macro_rules! define_indexed_gemv {+ ($fname:ident, $blk:ty, $variant:ident, $kernel:path, $w:expr, $lhs_blk:ty, $qk:expr) => {+ #[target_feature(enable = "dotprod")]+ #[allow(clippy::too_many_arguments)]+ unsafe fn $fname(+ kind: PackedKind,+ storage: &dyn QuantizedType,+ cache: &PackedCache,+ n_experts: usize,+ n_out: usize,+ k: usize,+ lhs: &[f32],+ n_rows: usize,+ ids: &[u32],+ topk: usize,+ dst: &mut [f32],+ ) {+ use super::GgmlType as _;+ let k_in_blocks = k / $qk;+ let total_rows = n_experts * n_out;+ let ps = cache.get_or_init(kind, || kind.pack(storage, total_rows));+ let PackedStorage::$variant(packed) = ps else {+ return;+ };++ // quantize each distinct lhs row once+ let mut lhs_q: Vec<$lhs_blk> = Vec::with_capacity(n_rows * k_in_blocks);+ #[allow(clippy::uninit_vec)]+ unsafe {+ lhs_q.set_len(n_rows * k_in_blocks)+ };+ let lhs_q_ptr = lhs_q.as_mut_ptr() as usize;+ let pool = crate::utils::barrier_pool();+ pool.execute_chunked(n_rows, |range| {+ let lhs_q_ptr = lhs_q_ptr as *mut $lhs_blk;+ for r in range {+ let row = &lhs[r * k..(r + 1) * k];+ let out = unsafe {+ std::slice::from_raw_parts_mut(lhs_q_ptr.add(r * k_in_blocks), k_in_blocks)+ };+ <$lhs_blk>::from_float(row, out);+ }+ });++ let groups_per_expert = n_out / $w;+ let n_pairs = ids.len();+ let total_units = n_pairs * groups_per_expert;+ let packed_ptr = packed.as_ptr() as usize;+ let lhs_q_ptr = lhs_q.as_ptr() as usize;+ let dst_ptr = dst.as_mut_ptr() as usize;+ let blk_bytes = std::mem::size_of::<$blk>();++ pool.execute_chunked(total_units, |range| {+ let packed_ptr = packed_ptr as *const u8;+ let lhs_q_ptr = lhs_q_ptr as *const $lhs_blk;+ let dst_ptr = dst_ptr as *mut f32;+ for unit in range {+ let pair = unit / groups_per_expert;+ let g = unit % groups_per_expert;+ let expert = ids[pair] as usize;+ // x may carry one row per token (shared across that token's experts)+ let row = if n_rows == n_pairs { pair } else { pair / topk };+ let lhs_row = unsafe {+ std::slice::from_raw_parts(lhs_q_ptr.add(row * k_in_blocks), k_in_blocks)+ };+ let group_global = expert * groups_per_expert + g;+ let xs = unsafe {+ std::slice::from_raw_parts(+ packed_ptr.add(group_global * k_in_blocks * blk_bytes) as *const $blk,+ k_in_blocks,+ )+ };+ let r = $kernel(k, xs, lhs_row);+ unsafe {+ std::ptr::copy_nonoverlapping(+ r.as_ptr(),+ dst_ptr.add(pair * n_out + g * $w),+ $w,+ );+ }+ }+ });+ }+ };+}++#[cfg(target_arch = "aarch64")]+define_indexed_gemv!(+ indexed_gemv_q4k,+ BlockQ4Kx8,+ Q4Kx8,+ super::neon::vec_dot_8_q4k_q8k,+ 8,+ super::k_quants::BlockQ8K,+ QK_K+);+#[cfg(target_arch = "aarch64")]+define_indexed_gemv!(+ indexed_gemv_q6k,+ BlockQ6Kx8,+ Q6Kx8,+ super::neon::vec_dot_8_q6kx8_q8k,+ 8,+ super::k_quants::BlockQ8K,+ QK_K+);+#[cfg(target_arch = "aarch64")]+define_indexed_gemv!(+ indexed_gemv_q8_0,+ BlockQ8_0x4,+ Q8_0x4,+ super::neon::vec_dot_4_q8_0x4_q8_0,+ 4,+ BlockQ8_0,+ QK8_0+);++#[cfg(target_arch = "aarch64")]+const INDEXED_BUCKET_MIN_PAIRS: usize = 32;++#[cfg(target_arch = "aarch64")]+#[allow(clippy::too_many_arguments)]+pub(crate) fn try_indexed_gemv(+ storage: &dyn QuantizedType,+ cache: &PackedCache,+ n_experts: usize,+ n_out: usize,+ k: usize,+ lhs: &[f32],+ n_rows: usize,+ ids: &[u32],+ topk: usize,+ dst: &mut [f32],+) -> Result<bool> {+ #[cfg(target_arch = "aarch64")]+ {+ let features = crate::cpu::features::get();+ if !features.dotprod {+ return Ok(false);+ }+ let Some(kind) = PackedKind::select(storage.dtype(), (1, k, n_experts * n_out)) else {+ return Ok(false);+ };+ // prefill: bucket pairs by expert and run one tiled matmul per expert instead+ // of a gemv per (token, expert) pair+ if ids.len() >= INDEXED_BUCKET_MIN_PAIRS+ && matches!(+ kind,+ PackedKind::Q4Kx8 | PackedKind::Q6Kx8 | PackedKind::Q8_0x4+ )+ {+ let ps = cache.get_or_init(kind, || kind.pack(storage, n_experts * n_out));+ let n_pairs = ids.len();+ let mut buckets: Vec<Vec<usize>> = vec![Vec::new(); n_experts];+ for (pair, &e) in ids.iter().enumerate() {+ buckets[e as usize].push(pair);+ }+ let mut gathered: Vec<f32> = Vec::new();+ let mut scratch: Vec<f32> = Vec::new();+ for (e, bucket) in buckets.iter().enumerate() {+ if bucket.is_empty() {+ continue;+ }+ let cnt = bucket.len();+ let cnt_pad = cnt.next_multiple_of(4);+ gathered.clear();+ gathered.resize(cnt_pad * k, 0.0);+ for (i, &pair) in bucket.iter().enumerate() {+ let row = if n_rows == n_pairs { pair } else { pair / topk };+ gathered[i * k..(i + 1) * k].copy_from_slice(&lhs[row * k..(row + 1) * k]);+ }+ scratch.clear();+ scratch.resize(cnt_pad * n_out, 0.0);+ kind.matmul_expert_range((cnt_pad, k, n_out), &gathered, ps, e, &mut scratch)?;+ for (i, &pair) in bucket.iter().enumerate() {+ dst[pair * n_out..(pair + 1) * n_out]+ .copy_from_slice(&scratch[i * n_out..(i + 1) * n_out]);+ }+ }+ return Ok(true);+ }+ match kind {+ PackedKind::Q4Kx8 if k.is_multiple_of(QK_K) && n_out.is_multiple_of(8) => {+ unsafe {+ indexed_gemv_q4k(+ kind, storage, cache, n_experts, n_out, k, lhs, n_rows, ids, topk, dst,+ )+ };+ Ok(true)+ }+ PackedKind::Q6Kx8 if k.is_multiple_of(QK_K) && n_out.is_multiple_of(8) => {+ unsafe {+ indexed_gemv_q6k(+ kind, storage, cache, n_experts, n_out, k, lhs, n_rows, ids, topk, dst,+ )+ };+ Ok(true)+ }+ PackedKind::Q8_0x4 if k.is_multiple_of(QK8_0) && n_out.is_multiple_of(4) => {+ unsafe {+ indexed_gemv_q8_0(+ kind, storage, cache, n_experts, n_out, k, lhs, n_rows, ids, topk, dst,+ )+ };+ Ok(true)+ }+ _ => Ok(false),+ }+ }+ #[cfg(not(target_arch = "aarch64"))]+ {+ let _ = (+ storage, cache, n_experts, n_out, k, lhs, n_rows, ids, topk, dst,+ );+ Ok(false)+ }+}diff --git a/vendor/candle-core/src/quantized/repack_x86.rs b/vendor/candle-core/src/quantized/repack_x86.rsnew file mode 100644index 0000000..3f6fa1d--- /dev/null+++ b/vendor/candle-core/src/quantized/repack_x86.rs@@ -0,0 +1,1094 @@+//! x86 repacked quantized kernels: weights tiled [n/16][k-blocks][16 rows] so one+//! vpdpbusd (AVX512-VNNI) produces 16 output columns for 4 k-values per instruction.+//! Activations are BlockQ8K rows; q4k/q6k quants are unsigned so vpdpbusd applies+//! directly with the min/offset terms corrected through the q8k bsums, and q8_0+//! weights are stored offset by +128 with the same correction.++use super::k_quants::{BlockQ4K, BlockQ6K, BlockQ8K, BlockQ8_0, QK8_0, QK_K};+use super::{GgmlType, QuantizedType};+use half::f16;++pub(crate) const TILE_N: usize = 16;++// One k-superblock (256 values) for 16 output rows.+#[repr(C)]+pub(crate) struct TileQ4K {+ // per-subblock (8) per-row (16) effective scales/mins, pre-unpacked from the 6-bit fields+ pub(crate) scales: [[f32; TILE_N]; 8],+ pub(crate) mins: [[f32; TILE_N]; 8],+ // nibble-packed: [k8-group (32)][row (16)][4 bytes]; byte j = q[8g+j] | q[8g+j+4] << 4+ pub(crate) qs: [u8; QK_K / 8 * TILE_N * 4],+}++#[repr(C)]+pub(crate) struct TileQ6K {+ pub(crate) scales: [[f32; TILE_N]; 16],+ // raw 6-bit values as u8 (0..63; real value is v - 32): [k4-group (64)][row (16)][4 bytes]+ pub(crate) qs: [u8; QK_K / 4 * TILE_N * 4],+}++// One 32-value block group for 16 rows of q8_0 weights, stored offset by +128.+#[repr(C)]+pub(crate) struct TileQ8_0 {+ pub(crate) d: [f16; TILE_N],+ pub(crate) qs: [u8; QK8_0 * TILE_N],+}++pub(crate) enum PackedX86 {+ Q4K(Vec<TileQ4K>),+ Q6K(Vec<TileQ6K>),+ Q8_0(Vec<TileQ8_0>),+}++pub(crate) fn select(dtype: super::GgmlDType, m: usize, n: usize, k: usize) -> bool {+ use super::GgmlDType as D;+ let Some(lv) = level() else {+ return false;+ };+ if k == 0 || !n.is_multiple_of(TILE_N) || !k.is_multiple_of(QK_K) {+ return false;+ }+ match dtype {+ D::Q4K => true,+ // Keep Q6K GEMV on compact weights; expanded tiles cost bandwidth+ // without reuse across multiple activation rows.+ D::Q6K => m > 1,+ // pure avx2 maddubs overflows on the +128 u8 range; q8_0 needs real dpbusd+ D::Q8_0 => lv != X86Level::Avx2,+ _ => false,+ }+}++fn q4k_blocks(storage: &dyn QuantizedType) -> &[BlockQ4K] {+ let data = storage.storage_size_in_bytes();+ let ptr = storage.as_ptr() as *const BlockQ4K;+ unsafe { std::slice::from_raw_parts(ptr, data / std::mem::size_of::<BlockQ4K>()) }+}+fn q6k_blocks(storage: &dyn QuantizedType) -> &[BlockQ6K] {+ let data = storage.storage_size_in_bytes();+ let ptr = storage.as_ptr() as *const BlockQ6K;+ unsafe { std::slice::from_raw_parts(ptr, data / std::mem::size_of::<BlockQ6K>()) }+}+fn q8_0_blocks(storage: &dyn QuantizedType) -> &[BlockQ8_0] {+ let data = storage.storage_size_in_bytes();+ let ptr = storage.as_ptr() as *const BlockQ8_0;+ unsafe { std::slice::from_raw_parts(ptr, data / std::mem::size_of::<BlockQ8_0>()) }+}++// ggml's 6-bit scale/min unpack for q4k/q5k superblocks.+fn get_scale_min_k4(j: usize, scales: &[u8; 12]) -> (u8, u8) {+ if j < 4 {+ (scales[j] & 63, scales[j + 4] & 63)+ } else {+ (+ (scales[j + 4] & 0x0F) | ((scales[j - 4] >> 6) << 4),+ (scales[j + 4] >> 4) | ((scales[j] >> 6) << 4),+ )+ }+}++pub(crate) fn pack_q4k(storage: &dyn QuantizedType, n: usize, k: usize) -> PackedX86 {+ let blocks = q4k_blocks(storage);+ let kb = k / QK_K;+ let n_tiles = n / TILE_N;+ let mut out: Vec<TileQ4K> = Vec::with_capacity(n_tiles * kb);+ for t in 0..n_tiles {+ for b in 0..kb {+ let mut tile = TileQ4K {+ scales: [[0.0; TILE_N]; 8],+ mins: [[0.0; TILE_N]; 8],+ qs: [0; QK_K / 8 * TILE_N * 4],+ };+ for r in 0..TILE_N {+ let blk = &blocks[(t * TILE_N + r) * kb + b];+ let d = blk.d.to_f32();+ let dmin = blk.dmin.to_f32();+ for sub in 0..8 {+ let (sc, mn) = get_scale_min_k4(sub, &blk.scales);+ tile.scales[sub][r] = d * sc as f32;+ tile.mins[sub][r] = dmin * mn as f32;+ }+ // q4k qs: 32-byte chunks c hold values [c*64, c*64+32) in the low nibbles+ // and [c*64+32, c*64+64) in the high nibbles+ let mut vals = [0u8; QK_K];+ for c in 0..4 {+ for j in 0..32 {+ let byte = blk.qs[c * 32 + j];+ vals[c * 64 + j] = byte & 0x0F;+ vals[c * 64 + 32 + j] = byte >> 4;+ }+ }+ for g in 0..QK_K / 8 {+ for j in 0..4 {+ tile.qs[(g * TILE_N + r) * 4 + j] =+ vals[g * 8 + j] | (vals[g * 8 + j + 4] << 4);+ }+ }+ }+ out.push(tile);+ }+ }+ PackedX86::Q4K(out)+}++pub(crate) fn pack_q6k(storage: &dyn QuantizedType, n: usize, k: usize) -> PackedX86 {+ let blocks = q6k_blocks(storage);+ let kb = k / QK_K;+ let n_tiles = n / TILE_N;+ let mut out: Vec<TileQ6K> = Vec::with_capacity(n_tiles * kb);+ for t in 0..n_tiles {+ for b in 0..kb {+ let mut tile = TileQ6K {+ scales: [[0.0; TILE_N]; 16],+ qs: [0; QK_K / 4 * TILE_N * 4],+ };+ for r in 0..TILE_N {+ let blk = &blocks[(t * TILE_N + r) * kb + b];+ let d = blk.d.to_f32();+ for sub in 0..16 {+ tile.scales[sub][r] = d * blk.scales[sub] as f32;+ }+ // q6k: ql 128 bytes (low 4 bits), qh 64 bytes (high 2 bits), in 128-value halves+ let mut vals = [0u8; QK_K];+ for h in 0..2 {+ let ql = &blk.ql[h * 64..];+ let qh = &blk.qh[h * 32..];+ for j in 0..32 {+ vals[h * 128 + j] = (ql[j] & 0x0F) | ((qh[j] & 3) << 4);+ vals[h * 128 + j + 32] = (ql[j + 32] & 0x0F) | (((qh[j] >> 2) & 3) << 4);+ vals[h * 128 + j + 64] = (ql[j] >> 4) | (((qh[j] >> 4) & 3) << 4);+ vals[h * 128 + j + 96] = (ql[j + 32] >> 4) | (((qh[j] >> 6) & 3) << 4);+ }+ }+ for g in 0..QK_K / 4 {+ for j in 0..4 {+ tile.qs[(g * TILE_N + r) * 4 + j] = vals[g * 4 + j];+ }+ }+ }+ out.push(tile);+ }+ }+ PackedX86::Q6K(out)+}++pub(crate) fn pack_q8_0(storage: &dyn QuantizedType, n: usize, k: usize) -> PackedX86 {+ let blocks = q8_0_blocks(storage);+ let kb = k / QK8_0;+ let n_tiles = n / TILE_N;+ let mut out: Vec<TileQ8_0> = Vec::with_capacity(n_tiles * kb);+ for t in 0..n_tiles {+ for b in 0..kb {+ let mut tile = TileQ8_0 {+ d: [f16::ZERO; TILE_N],+ qs: [0; QK8_0 * TILE_N],+ };+ for r in 0..TILE_N {+ let blk = &blocks[(t * TILE_N + r) * kb + b];+ tile.d[r] = blk.d;+ for g in 0..QK8_0 / 4 {+ for j in 0..4 {+ tile.qs[(g * TILE_N + r) * 4 + j] = (blk.qs[g * 4 + j] as i16 + 128) as u8;+ }+ }+ }+ out.push(tile);+ }+ }+ PackedX86::Q8_0(out)+}++#[cfg(target_arch = "x86_64")]+mod kernels {+ use super::*;+ use core::arch::x86_64::*;++ // acc[m][16] f32 for up to MT activation rows against one 16-row weight tile+ pub(crate) const MAX_MT: usize = 4;++ #[inline(always)]+ unsafe fn bcast4(act: &[i8], off: usize) -> __m512i {+ _mm512_set1_epi32(std::ptr::read_unaligned(act.as_ptr().add(off) as *const i32))+ }++ // q4k: one superblock tile against `mt` q8k activation blocks. Subblocks are+ // processed in pairs with independent integer accumulators to break the dpbusd+ // latency chain when mt is small (decode gemv).+ #[target_feature(enable = "avx512f,avx512vnni,avx512bw")]+ pub(crate) unsafe fn q4k_tile(tile: &TileQ4K, acts: &[&BlockQ8K], acc: &mut [[__m512; 1]]) {+ let mask = _mm512_set1_epi8(0x0F_u8 as i8);+ for sub2 in 0..4 {+ let sa = sub2 * 2;+ let sb = sa + 1;+ let mut isum_a = [_mm512_setzero_si512(); MAX_MT];+ let mut isum_b = [_mm512_setzero_si512(); MAX_MT];+ for g4 in 0..4 {+ let ga = sa * 4 + g4;+ let gb = sb * 4 + g4;+ let wa = _mm512_loadu_si512(tile.qs.as_ptr().add(ga * TILE_N * 4) as *const _);+ let wb = _mm512_loadu_si512(tile.qs.as_ptr().add(gb * TILE_N * 4) as *const _);+ let lo_a = _mm512_and_si512(wa, mask);+ let hi_a = _mm512_and_si512(_mm512_srli_epi16(wa, 4), mask);+ let lo_b = _mm512_and_si512(wb, mask);+ let hi_b = _mm512_and_si512(_mm512_srli_epi16(wb, 4), mask);+ for (m, act) in acts.iter().enumerate() {+ isum_a[m] =+ _mm512_dpbusd_epi32(isum_a[m], lo_a, bcast4(&act.qs, sa * 32 + g4 * 8));+ isum_a[m] =+ _mm512_dpbusd_epi32(isum_a[m], hi_a, bcast4(&act.qs, sa * 32 + g4 * 8 + 4));+ isum_b[m] =+ _mm512_dpbusd_epi32(isum_b[m], lo_b, bcast4(&act.qs, sb * 32 + g4 * 8));+ isum_b[m] =+ _mm512_dpbusd_epi32(isum_b[m], hi_b, bcast4(&act.qs, sb * 32 + g4 * 8 + 4));+ }+ }+ for (sub, isum) in [(sa, &isum_a), (sb, &isum_b)] {+ let scale = _mm512_loadu_ps(tile.scales[sub].as_ptr());+ let min = _mm512_loadu_ps(tile.mins[sub].as_ptr());+ for (m, act) in acts.iter().enumerate() {+ let dall = act.d;+ let f = _mm512_cvtepi32_ps(isum[m]);+ acc[m][0] =+ _mm512_fmadd_ps(f, _mm512_mul_ps(scale, _mm512_set1_ps(dall)), acc[m][0]);+ let bsum = (act.bsums[sub * 2] as i32 + act.bsums[sub * 2 + 1] as i32) as f32;+ acc[m][0] = _mm512_fnmadd_ps(min, _mm512_set1_ps(dall * bsum), acc[m][0]);+ }+ }+ }+ }++ // q6k: values are v-32; vpdpbusd on raw u6 then subtract 32 * bsum per 16-value subblock.+ #[target_feature(enable = "avx512f,avx512vnni,avx512bw")]+ pub(crate) unsafe fn q6k_tile(tile: &TileQ6K, acts: &[&BlockQ8K], acc: &mut [[__m512; 1]]) {+ for sub in 0..16 {+ let mut isum = [_mm512_setzero_si512(); MAX_MT];+ // subblock = 16 k-values = 4 k4-groups+ for g4 in 0..4 {+ let g = sub * 4 + g4;+ let w = _mm512_loadu_si512(tile.qs.as_ptr().add(g * TILE_N * 4) as *const _);+ for (m, act) in acts.iter().enumerate() {+ let a = bcast4(&act.qs, sub * 16 + g4 * 4);+ isum[m] = _mm512_dpbusd_epi32(isum[m], w, a);+ }+ }+ let scale = _mm512_loadu_ps(tile.scales[sub].as_ptr());+ for (m, act) in acts.iter().enumerate() {+ let dall = act.d;+ let bsum = act.bsums[sub] as f32;+ let f = _mm512_sub_ps(_mm512_cvtepi32_ps(isum[m]), _mm512_set1_ps(32.0 * bsum));+ acc[m][0] =+ _mm512_fmadd_ps(f, _mm512_mul_ps(scale, _mm512_set1_ps(dall)), acc[m][0]);+ }+ }+ }++ // q8_0 (+128 offset weights), two 32-value block groups per call with independent+ // integer accumulators so the dpbusd latency chain never serializes at small m.+ #[target_feature(enable = "avx512f,avx512vnni,avx512bw")]+ pub(crate) unsafe fn q8_0_tile2(+ t0: &TileQ8_0,+ t1: &TileQ8_0,+ acts: &[&BlockQ8K],+ sub32: usize,+ acc: &mut [[__m512; 1]],+ ) {+ let mut isum_a = [_mm512_setzero_si512(); MAX_MT];+ let mut isum_b = [_mm512_setzero_si512(); MAX_MT];+ for g4 in 0..8 {+ let wa = _mm512_loadu_si512(t0.qs.as_ptr().add(g4 * TILE_N * 4) as *const _);+ let wb = _mm512_loadu_si512(t1.qs.as_ptr().add(g4 * TILE_N * 4) as *const _);+ for (m, act) in acts.iter().enumerate() {+ isum_a[m] =+ _mm512_dpbusd_epi32(isum_a[m], wa, bcast4(&act.qs, sub32 * 32 + g4 * 4));+ isum_b[m] =+ _mm512_dpbusd_epi32(isum_b[m], wb, bcast4(&act.qs, (sub32 + 1) * 32 + g4 * 4));+ }+ }+ for (blk, isum, tile) in [(sub32, &isum_a, t0), (sub32 + 1, &isum_b, t1)] {+ let dw = _mm512_cvtph_ps(_mm256_loadu_si256(tile.d.as_ptr() as *const _));+ for (m, act) in acts.iter().enumerate() {+ let bsum = (act.bsums[blk * 2] as i32 + act.bsums[blk * 2 + 1] as i32) as f32;+ let f = _mm512_sub_ps(_mm512_cvtepi32_ps(isum[m]), _mm512_set1_ps(128.0 * bsum));+ acc[m][0] = _mm512_fmadd_ps(f, _mm512_mul_ps(dw, _mm512_set1_ps(act.d)), acc[m][0]);+ }+ }+ }+}++// Full matmul over the packed tiles: lhs is m rows already quantized to BlockQ8K,+// parallelized over (m-tiles x n-tiles) on the barrier pool by the caller's chunker.+// Per-unit bodies live in #[target_feature] fns so intrinsics inline under portable+// (target-cpu=generic) release builds; closures cannot carry the attribute.+#[cfg(target_arch = "x86_64")]+#[target_feature(enable = "avx512f,avx512vnni,avx512bw")]+#[allow(clippy::too_many_arguments)]+unsafe fn matmul_unit_512(+ packed: &PackedX86,+ lhs_q: &[BlockQ8K],+ kb: usize,+ nt: usize,+ m0: usize,+ mt: usize,+ dst_ptr: *mut f32,+ n: usize,+) {+ use core::arch::x86_64::*;+ let mut act_refs: [&BlockQ8K; kernels::MAX_MT] = [&lhs_q[m0 * kb]; kernels::MAX_MT];+ let mut acc = [[_mm512_setzero_ps(); 1]; kernels::MAX_MT];+ for b in 0..kb {+ for (i, a) in act_refs.iter_mut().enumerate().take(mt) {+ *a = &lhs_q[(m0 + i) * kb + b];+ }+ match packed {+ PackedX86::Q4K(tiles) => {+ kernels::q4k_tile(&tiles[nt * kb + b], &act_refs[..mt], &mut acc)+ }+ PackedX86::Q6K(tiles) => {+ kernels::q6k_tile(&tiles[nt * kb + b], &act_refs[..mt], &mut acc)+ }+ PackedX86::Q8_0(tiles) => {+ // q8_0 tiles cover 32 k-values each; 8 tiles per q8k superblock+ for s in (0..8).step_by(2) {+ kernels::q8_0_tile2(+ &tiles[nt * (kb * 8) + b * 8 + s],+ &tiles[nt * (kb * 8) + b * 8 + s + 1],+ &act_refs[..mt],+ s,+ &mut acc,+ );+ }+ }+ }+ }+ for (i, row) in acc[..mt].iter().enumerate() {+ _mm512_storeu_ps(dst_ptr.add((m0 + i) * n + nt * TILE_N), row[0]);+ }+}++#[cfg(target_arch = "x86_64")]+#[target_feature(enable = "avx2,fma")]+#[allow(clippy::too_many_arguments)]+unsafe fn matmul_unit_256(+ packed: &PackedX86,+ lhs_q: &[BlockQ8K],+ kb: usize,+ nt: usize,+ m0: usize,+ mt: usize,+ dst_ptr: *mut f32,+ n: usize,+ vnni: bool,+) {+ use core::arch::x86_64::*;+ let mut act_refs: [&BlockQ8K; kernels::MAX_MT] = [&lhs_q[m0 * kb]; kernels::MAX_MT];+ let mut acc = [[_mm256_setzero_ps(); 2]; kernels::MAX_MT];+ for b in 0..kb {+ for (i, a) in act_refs.iter_mut().enumerate().take(mt) {+ *a = &lhs_q[(m0 + i) * kb + b];+ }+ match packed {+ PackedX86::Q4K(tiles) => {+ let t = &tiles[nt * kb + b];+ if vnni {+ kernels256::q4k_tile_avxvnni(t, &act_refs[..mt], &mut acc)+ } else {+ kernels256::q4k_tile_avx2(t, &act_refs[..mt], &mut acc)+ }+ }+ PackedX86::Q6K(tiles) => {+ let t = &tiles[nt * kb + b];+ if vnni {+ kernels256::q6k_tile_avxvnni(t, &act_refs[..mt], &mut acc)+ } else {+ kernels256::q6k_tile_avx2(t, &act_refs[..mt], &mut acc)+ }+ }+ PackedX86::Q8_0(tiles) => {+ for s in (0..8).step_by(2) {+ kernels256::q8_0_tile2_avxvnni(+ &tiles[nt * (kb * 8) + b * 8 + s],+ &tiles[nt * (kb * 8) + b * 8 + s + 1],+ &act_refs[..mt],+ s,+ &mut acc,+ );+ }+ }+ }+ }+ for (i, row) in acc[..mt].iter().enumerate() {+ _mm256_storeu_ps(dst_ptr.add((m0 + i) * n + nt * TILE_N), row[0]);+ _mm256_storeu_ps(dst_ptr.add((m0 + i) * n + nt * TILE_N + 8), row[1]);+ }+}++pub(crate) fn matmul_tiles(+ packed: &PackedX86,+ lhs_q: &[BlockQ8K],+ m: usize,+ k: usize,+ n: usize,+ dst: &mut [f32],+) {+ #[cfg(target_arch = "x86_64")]+ {+ let lv = level().expect("select() gated");+ let kb = k / QK_K;+ let n_tiles = n / TILE_N;+ let mt_step = kernels::MAX_MT;+ let dst_ptr = dst.as_mut_ptr() as usize;+ let total_units = n_tiles * m.div_ceil(mt_step);+ let m_units = m.div_ceil(mt_step);+ crate::utils::barrier_pool().execute_chunked(total_units, |range| {+ let dst_ptr = dst_ptr as *mut f32;+ for unit in range {+ let nt = unit / m_units;+ let m0 = (unit % m_units) * mt_step;+ let mt = (m - m0).min(mt_step);+ unsafe {+ if lv == X86Level::Avx512Vnni {+ matmul_unit_512(packed, lhs_q, kb, nt, m0, mt, dst_ptr, n);+ } else {+ matmul_unit_256(+ packed,+ lhs_q,+ kb,+ nt,+ m0,+ mt,+ dst_ptr,+ n,+ lv == X86Level::AvxVnni,+ );+ }+ }+ }+ });+ }+ #[cfg(not(target_arch = "x86_64"))]+ {+ let _ = (packed, lhs_q, m, k, n, dst);+ unreachable!("x86 repack kernels on non-x86 arch");+ }+}++pub(crate) fn pack(+ dtype: super::GgmlDType,+ storage: &dyn QuantizedType,+ n: usize,+ k: usize,+) -> PackedX86 {+ use super::GgmlDType as D;+ match dtype {+ D::Q4K => pack_q4k(storage, n, k),+ D::Q6K => pack_q6k(storage, n, k),+ D::Q8_0 => pack_q8_0(storage, n, k),+ _ => unreachable!("select() gates dtypes"),+ }+}++// Quantize lhs rows to BlockQ8K, parallelizing only when there are multiple rows.+pub(crate) fn quantize_lhs(lhs: &[f32], m: usize, k: usize) -> Vec<BlockQ8K> {+ let kb = k / QK_K;+ let mut out = vec![BlockQ8K::zeros(); m * kb];+ if m == 1 {+ BlockQ8K::from_float(&lhs[..k], &mut out);+ return out;+ }+ let out_ptr = out.as_mut_ptr() as usize;+ crate::utils::barrier_pool().execute_chunked(m, |range| {+ let out_ptr = out_ptr as *mut BlockQ8K;+ for r in range {+ let row = &lhs[r * k..(r + 1) * k];+ let dst = unsafe { std::slice::from_raw_parts_mut(out_ptr.add(r * kb), kb) };+ BlockQ8K::from_float(row, dst);+ }+ });+ out+}++// Fused multi-projection gemv: one lhs quantization and one barrier region across+// all projections sharing the same activations (qkv, gate/up). 512-bit only; the+// per-unit body reuses matmul_unit_512 so intrinsics inline in portable builds.+pub(crate) fn gemv_fused(+ parts: &[(&PackedX86, usize)],+ lhs_q: &[BlockQ8K],+ m: usize,+ k: usize,+ dsts: &mut [Vec<f32>],+) {+ #[cfg(target_arch = "x86_64")]+ {+ let kb = k / QK_K;+ let mt_step = kernels::MAX_MT;+ let m_units = m.div_ceil(mt_step);+ // (n, dst ptr, unit offset)+ let mut metas: Vec<(usize, usize, usize)> = Vec::with_capacity(parts.len());+ let mut total = 0usize;+ for (i, (_p, n)) in parts.iter().enumerate() {+ metas.push((*n, dsts[i].as_mut_ptr() as usize, total));+ total += (n / TILE_N) * m_units;+ }+ let parts_ref = parts;+ crate::utils::barrier_pool().execute_chunked(total, |range| {+ for unit in range {+ let pi = metas+ .iter()+ .rposition(|(_n, _d, off)| unit >= *off)+ .unwrap();+ let (n, dst_ptr, off) = metas[pi];+ let local = unit - off;+ let nt = local / m_units;+ let m0 = (local % m_units) * mt_step;+ let mt = (m - m0).min(mt_step);+ unsafe {+ matmul_unit_512(+ parts_ref[pi].0,+ lhs_q,+ kb,+ nt,+ m0,+ mt,+ dst_ptr as *mut f32,+ n,+ );+ }+ }+ });+ }+ #[cfg(not(target_arch = "x86_64"))]+ {+ let _ = (parts, lhs_q, m, k, dsts);+ unreachable!("x86 repack kernels on non-x86 arch");+ }+}++// AMX-int8 path for large-m (prefill) q4k matmuls: weights unpacked to i8 in tile+// layout (B tile = 8 rows x 64 bytes = 32k x 16n VNNI-paired), activations repacked+// per 16-row block, tdpbssd accumulating 16x16 i32 per 32-k subblock so the q4k+// per-subblock scales apply exactly.+#[repr(C)]+pub(crate) struct TileQ4KAmx {+ pub(crate) scales: [[f32; TILE_N]; 8],+ pub(crate) mins: [[f32; TILE_N]; 8],+ // [subblock (8)][k4-row (8)][16 cols * 4 k] unsigned 4-bit values as i8+ pub(crate) qs: [u8; QK_K * TILE_N],+}++pub(crate) fn amx_available() -> bool {+ use std::sync::OnceLock;+ static OK: OnceLock<bool> = OnceLock::new();+ *OK.get_or_init(|| {+ // forced downlevel testing must exercise the 256-bit paths end to end+ if level() != Some(X86Level::Avx512Vnni) {+ return false;+ }+ // is_x86_feature_detected!("amx-int8") is unstable; read CPUID leaf 7 directly+ #[cfg(target_arch = "x86_64")]+ let has_amx = {+ let info = core::arch::x86_64::__cpuid_count(7, 0);+ info.edx & (1 << 25) != 0 && info.edx & (1 << 24) != 0+ };+ #[cfg(not(target_arch = "x86_64"))]+ let has_amx = false;+ if !has_amx {+ return false;+ }+ // Linux gates AMX tile state behind a per-process permission request; other+ // OSes would need their own enablement, so AMX stays off there.+ #[cfg(target_os = "linux")]+ {+ const ARCH_REQ_XCOMP_PERM: i32 = 0x1023;+ const XFEATURE_XTILEDATA: u64 = 18;+ let r = unsafe {+ libc::syscall(+ libc::SYS_arch_prctl,+ ARCH_REQ_XCOMP_PERM,+ XFEATURE_XTILEDATA,+ )+ };+ r == 0+ }+ #[cfg(not(target_os = "linux"))]+ false+ })+}++pub(crate) fn pack_q4k_amx(storage: &dyn QuantizedType, n: usize, k: usize) -> Vec<TileQ4KAmx> {+ let blocks = q4k_blocks(storage);+ let kb = k / QK_K;+ let n_tiles = n / TILE_N;+ let mut out: Vec<TileQ4KAmx> = Vec::with_capacity(n_tiles * kb);+ for t in 0..n_tiles {+ for b in 0..kb {+ let mut tile = TileQ4KAmx {+ scales: [[0.0; TILE_N]; 8],+ mins: [[0.0; TILE_N]; 8],+ qs: [0; QK_K * TILE_N],+ };+ for r in 0..TILE_N {+ let blk = &blocks[(t * TILE_N + r) * kb + b];+ let d = blk.d.to_f32();+ let dmin = blk.dmin.to_f32();+ for sub in 0..8 {+ let (sc, mn) = get_scale_min_k4(sub, &blk.scales);+ tile.scales[sub][r] = d * sc as f32;+ tile.mins[sub][r] = dmin * mn as f32;+ }+ let mut vals = [0u8; QK_K];+ for c in 0..4 {+ for j in 0..32 {+ let byte = blk.qs[c * 32 + j];+ vals[c * 64 + j] = byte & 0x0F;+ vals[c * 64 + 32 + j] = byte >> 4;+ }+ }+ for sub in 0..8 {+ for krow in 0..8 {+ for j in 0..4 {+ tile.qs[((sub * 8 + krow) * TILE_N + r) * 4 + j] =+ vals[sub * 32 + krow * 4 + j];+ }+ }+ }+ }+ out.push(tile);+ }+ }+ out+}++#[cfg(target_arch = "x86_64")]+mod amx {+ use super::*;+ use core::arch::x86_64::*;++ #[repr(C, align(64))]+ struct TileCfg {+ palette: u8,+ start_row: u8,+ _rsvd: [u8; 14],+ colsb: [u16; 16],+ rows: [u8; 16],+ }++ // tmm0 = C (16x16 i32), tmm1 = A (16 x 32 i8), tmm2 = B (8 x 64 i8)+ unsafe fn configure() {+ let mut cfg = TileCfg {+ palette: 1,+ start_row: 0,+ _rsvd: [0; 14],+ colsb: [0; 16],+ rows: [0; 16],+ };+ const _: () = assert!(std::mem::size_of::<TileCfg>() == 64);+ // tmm0-3: C (16x16 i32); tmm4-5: A (16x32 i8); tmm6-7: B (8x64 i8)+ for t in 0..4 {+ cfg.colsb[t] = 64;+ cfg.rows[t] = 16;+ }+ for t in 4..6 {+ cfg.colsb[t] = 32;+ cfg.rows[t] = 16;+ }+ for t in 6..8 {+ cfg.colsb[t] = 64;+ cfg.rows[t] = 8;+ }+ core::arch::asm!(+ "ldtilecfg [{cfg}]",+ cfg = in(reg) &cfg,+ options(nostack)+ );+ }++ thread_local! {+ static AMX_READY: std::cell::Cell<bool> = const { std::cell::Cell::new(false) };+ }++ fn ensure_configured() {+ AMX_READY.with(|f| {+ if !f.get() {+ unsafe { configure() };+ f.set(true);+ }+ });+ }++ // One (m16-block, n-tile) product over the full k range.+ // a_pack: [kb][8 subs][16 rows * 32 bytes]; returns via dst rows of n floats.+ // 32m x 32n macro-tile: 4 C tiles per (2 A, 2 B) loads so tdp throughput is not+ // gated on tile loads. a_pack holds 32 m-rows (two 16-row halves interleaved by sub).+ #[target_feature(enable = "avx512f,avx512bw")]+ #[allow(clippy::too_many_arguments)]+ pub(super) unsafe fn q4k_m32(+ tiles: &[TileQ4KAmx],+ nt0: usize,+ n_tiles: usize,+ kb: usize,+ a_pack: &[i8],+ acts: &[&[BlockQ8K]],+ mt: usize,+ dst: *mut f32,+ n: usize,+ ) {+ ensure_configured();+ let two_n = nt0 + 1 < n_tiles;+ let mut c_scratch = [0i32; 4 * 16 * 16];+ let mut facc = [0f32; 32 * 32];+ let a_half = kb * 8 * 16 * 32;+ unsafe {+ for b in 0..kb {+ let t0 = &tiles[nt0 * kb + b];+ let t1 = &tiles[(nt0 + two_n as usize) * kb + b];+ for sub in 0..8 {+ let a0 = a_pack.as_ptr().add((b * 8 + sub) * 16 * 32);+ let a1 = a_pack.as_ptr().add(a_half + (b * 8 + sub) * 16 * 32);+ let b0 = t0.qs.as_ptr().add(sub * 8 * TILE_N * 4);+ let b1 = t1.qs.as_ptr().add(sub * 8 * TILE_N * 4);+ core::arch::asm!(+ "tilezero tmm0",+ "tilezero tmm1",+ "tilezero tmm2",+ "tilezero tmm3",+ "tileloadd tmm4, [{a0} + {sa}*1]",+ "tileloadd tmm6, [{b0} + {sb}*1]",+ "tdpbssd tmm0, tmm4, tmm6",+ "tileloadd tmm7, [{b1} + {sb}*1]",+ "tdpbssd tmm1, tmm4, tmm7",+ "tileloadd tmm5, [{a1} + {sa}*1]",+ "tdpbssd tmm2, tmm5, tmm6",+ "tdpbssd tmm3, tmm5, tmm7",+ "tilestored [{c} + {sc}*1], tmm0",+ "tilestored [{c} + {sc}*1 + 1024], tmm1",+ "tilestored [{c} + {sc}*1 + 2048], tmm2",+ "tilestored [{c} + {sc}*1 + 3072], tmm3",+ a0 = in(reg) a0,+ a1 = in(reg) a1,+ b0 = in(reg) b0,+ b1 = in(reg) b1,+ c = in(reg) c_scratch.as_mut_ptr(),+ sa = in(reg) 32usize,+ sb = in(reg) 64usize,+ sc = in(reg) 64usize,+ options(nostack)+ );+ for half in 0..2 {+ let rows = if half == 0 {+ mt.min(16)+ } else {+ mt.saturating_sub(16)+ };+ if rows == 0 {+ continue;+ }+ for ntl in 0..1 + two_n as usize {+ let tile = if ntl == 0 { t0 } else { t1 };+ let scale = _mm512_loadu_ps(tile.scales[sub].as_ptr());+ let min = _mm512_loadu_ps(tile.mins[sub].as_ptr());+ let cbase = (half * 2 + ntl) * 256;+ for mi in 0..rows {+ let act = &acts[half * 16 + mi][b];+ let dall = act.d;+ let ci = _mm512_loadu_si512(+ c_scratch.as_ptr().add(cbase + mi * 16) as *const _,+ );+ let cf = _mm512_cvtepi32_ps(ci);+ let fo = (half * 16 + mi) * 32 + ntl * 16;+ let f = _mm512_loadu_ps(facc.as_ptr().add(fo));+ let bsum = (act.bsums[sub * 2] as i32+ + act.bsums[sub * 2 + 1] as i32)+ as f32;+ let f = _mm512_fmadd_ps(+ cf,+ _mm512_mul_ps(scale, _mm512_set1_ps(dall)),+ f,+ );+ let f = _mm512_fnmadd_ps(min, _mm512_set1_ps(dall * bsum), f);+ _mm512_storeu_ps(facc.as_mut_ptr().add(fo), f);+ }+ }+ }+ }+ }+ let n_cols = 16 * (1 + two_n as usize);+ for mi in 0..mt {+ std::ptr::copy_nonoverlapping(+ facc.as_ptr().add(mi * 32),+ dst.add(mi * n + nt0 * TILE_N),+ n_cols,+ );+ }+ }+ }+}++// AMX matmul: m >= 32, q4k only. acts are per-row BlockQ8K slices.+pub(crate) fn matmul_amx_q4k(+ tiles: &[TileQ4KAmx],+ lhs_q: &[BlockQ8K],+ m: usize,+ k: usize,+ n: usize,+ dst: &mut [f32],+) {+ #[cfg(target_arch = "x86_64")]+ {+ let kb = k / QK_K;+ let n_tiles = n / TILE_N;+ let m_blocks = m.div_ceil(32);+ let dst_ptr = dst.as_mut_ptr() as usize;+ let lhs_ref = lhs_q;+ crate::utils::barrier_pool().execute_chunked(m_blocks, |range| {+ let dst_ptr = dst_ptr as *mut f32;+ // per-thread activation repack buffer (two 16-row halves), reused across n+ let a_half = kb * 8 * 16 * 32;+ let mut a_pack = vec![0i8; 2 * a_half];+ let mut act_rows: Vec<&[BlockQ8K]> = Vec::with_capacity(32);+ for mb in range {+ let m0 = mb * 32;+ let mt = (m - m0).min(32);+ act_rows.clear();+ for i in 0..mt {+ act_rows.push(&lhs_ref[(m0 + i) * kb..(m0 + i + 1) * kb]);+ }+ a_pack.fill(0);+ for (i, row) in act_rows.iter().enumerate() {+ let base = (i / 16) * a_half;+ let r = i % 16;+ for b in 0..kb {+ let qs = &row[b].qs;+ for sub in 0..8 {+ let dst_off = base + ((b * 8 + sub) * 16 + r) * 32;+ a_pack[dst_off..dst_off + 32]+ .copy_from_slice(&qs[sub * 32..sub * 32 + 32]);+ }+ }+ }+ let mut nt = 0;+ while nt < n_tiles {+ unsafe {+ amx::q4k_m32(+ tiles,+ nt,+ n_tiles,+ kb,+ &a_pack,+ &act_rows,+ mt,+ dst_ptr.add(m0 * n),+ n,+ )+ };+ nt += 2;+ }+ }+ });+ }+ #[cfg(not(target_arch = "x86_64"))]+ {+ let _ = (tiles, lhs_q, m, k, n, dst);+ unreachable!()+ }+}++// 256-bit kernel tier for CPUs without AVX512: AVX-VNNI (vpdpbusd ymm) or pure AVX2+// (maddubs+madd, safe for the u4/u6 x i8 operand ranges). Same tile layout, each+// 16-column group processed as two ymm halves.+#[derive(Clone, Copy, PartialEq)]+pub(crate) enum X86Level {+ Avx512Vnni,+ AvxVnni,+ Avx2,+}++pub(crate) fn level() -> Option<X86Level> {+ use std::sync::OnceLock;+ static LEVEL: OnceLock<Option<X86Level>> = OnceLock::new();+ *LEVEL.get_or_init(|| {+ let force_avx2 = std::env::var("MISTRALRS_FORCE_AVX2").as_deref() == Ok("1");+ let force_vnni = std::env::var("MISTRALRS_FORCE_AVXVNNI").as_deref() == Ok("1");+ if !force_avx2+ && !force_vnni+ && is_x86_feature_detected!("avx512f")+ && is_x86_feature_detected!("avx512vnni")+ {+ return Some(X86Level::Avx512Vnni);+ }+ if !force_avx2+ && (force_vnni || is_x86_feature_detected!("avxvnni"))+ && is_x86_feature_detected!("avx2")+ {+ return Some(X86Level::AvxVnni);+ }+ if is_x86_feature_detected!("avx2") && is_x86_feature_detected!("fma") {+ return Some(X86Level::Avx2);+ }+ None+ })+}++#[cfg(target_arch = "x86_64")]+mod kernels256 {+ use super::kernels::MAX_MT;+ use super::*;+ use core::arch::x86_64::*;++ #[target_feature(enable = "avx2")]+ unsafe fn bcast4_256(act: &[i8], off: usize) -> __m256i {+ _mm256_set1_epi32(std::ptr::read_unaligned(act.as_ptr().add(off) as *const i32))+ }++ #[target_feature(enable = "avx2,avxvnni")]+ unsafe fn dp_vnni(acc: __m256i, w_u8: __m256i, a_i8: __m256i) -> __m256i {+ _mm256_dpbusd_avx_epi32(acc, w_u8, a_i8)+ }++ // dpbusd stand-in for pure AVX2: maddubs (u8 x i8 -> i16 pairs) then madd against ones.+ #[target_feature(enable = "avx2")]+ unsafe fn dp_avx2(acc: __m256i, w_u8: __m256i, a_i8: __m256i) -> __m256i {+ let prod16 = _mm256_maddubs_epi16(w_u8, a_i8);+ _mm256_add_epi32(acc, _mm256_madd_epi16(prod16, _mm256_set1_epi16(1)))+ }++ macro_rules! q4k_tile_256 {+ ($fname:ident, $dp:path, $feat:literal) => {+ #[target_feature(enable = $feat)]+ pub(crate) unsafe fn $fname(+ tile: &TileQ4K,+ acts: &[&BlockQ8K],+ acc: &mut [[__m256; 2]],+ ) {+ let mask = _mm256_set1_epi8(0x0F_u8 as i8);+ for sub in 0..8 {+ let mut isum = [[_mm256_setzero_si256(); 2]; MAX_MT];+ for g4 in 0..4 {+ let g = sub * 4 + g4;+ for h in 0..2 {+ let w = _mm256_loadu_si256(+ tile.qs.as_ptr().add(g * TILE_N * 4 + h * 32) as *const _,+ );+ let lo = _mm256_and_si256(w, mask);+ let hi = _mm256_and_si256(_mm256_srli_epi16(w, 4), mask);+ for (m, act) in acts.iter().enumerate() {+ let a_lo = bcast4_256(&act.qs, sub * 32 + g4 * 8);+ let a_hi = bcast4_256(&act.qs, sub * 32 + g4 * 8 + 4);+ isum[m][h] = $dp(isum[m][h], lo, a_lo);+ isum[m][h] = $dp(isum[m][h], hi, a_hi);+ }+ }+ }+ for h in 0..2 {+ let scale = _mm256_loadu_ps(tile.scales[sub].as_ptr().add(h * 8));+ let min = _mm256_loadu_ps(tile.mins[sub].as_ptr().add(h * 8));+ for (m, act) in acts.iter().enumerate() {+ let dall = act.d;+ let f = _mm256_cvtepi32_ps(isum[m][h]);+ acc[m][h] = _mm256_fmadd_ps(+ f,+ _mm256_mul_ps(scale, _mm256_set1_ps(dall)),+ acc[m][h],+ );+ let bsum =+ (act.bsums[sub * 2] as i32 + act.bsums[sub * 2 + 1] as i32) as f32;+ acc[m][h] =+ _mm256_fnmadd_ps(min, _mm256_set1_ps(dall * bsum), acc[m][h]);+ }+ }+ }+ }+ };+ }+ q4k_tile_256!(q4k_tile_avxvnni, dp_vnni, "avx2,avxvnni,fma");+ q4k_tile_256!(q4k_tile_avx2, dp_avx2, "avx2,fma");++ macro_rules! q6k_tile_256 {+ ($fname:ident, $dp:path, $feat:literal) => {+ #[target_feature(enable = $feat)]+ pub(crate) unsafe fn $fname(+ tile: &TileQ6K,+ acts: &[&BlockQ8K],+ acc: &mut [[__m256; 2]],+ ) {+ for sub in 0..16 {+ let mut isum = [[_mm256_setzero_si256(); 2]; MAX_MT];+ for g4 in 0..4 {+ let g = sub * 4 + g4;+ for h in 0..2 {+ let w = _mm256_loadu_si256(+ tile.qs.as_ptr().add(g * TILE_N * 4 + h * 32) as *const _,+ );+ for (m, act) in acts.iter().enumerate() {+ let a = bcast4_256(&act.qs, sub * 16 + g4 * 4);+ isum[m][h] = $dp(isum[m][h], w, a);+ }+ }+ }+ for h in 0..2 {+ let scale = _mm256_loadu_ps(tile.scales[sub].as_ptr().add(h * 8));+ for (m, act) in acts.iter().enumerate() {+ let dall = act.d;+ let bsum = act.bsums[sub] as f32;+ let f = _mm256_sub_ps(+ _mm256_cvtepi32_ps(isum[m][h]),+ _mm256_set1_ps(32.0 * bsum),+ );+ acc[m][h] = _mm256_fmadd_ps(+ f,+ _mm256_mul_ps(scale, _mm256_set1_ps(dall)),+ acc[m][h],+ );+ }+ }+ }+ }+ };+ }+ q6k_tile_256!(q6k_tile_avxvnni, dp_vnni, "avx2,avxvnni,fma");+ q6k_tile_256!(q6k_tile_avx2, dp_avx2, "avx2,fma");++ // q8_0 needs true dpbusd (u8 range overflows maddubs' i16 pairs), so vnni only.+ #[target_feature(enable = "avx2,avxvnni,fma")]+ pub(crate) unsafe fn q8_0_tile2_avxvnni(+ t0: &TileQ8_0,+ t1: &TileQ8_0,+ acts: &[&BlockQ8K],+ sub32: usize,+ acc: &mut [[__m256; 2]],+ ) {+ let mut isum_a = [[_mm256_setzero_si256(); 2]; MAX_MT];+ let mut isum_b = [[_mm256_setzero_si256(); 2]; MAX_MT];+ for g4 in 0..8 {+ for h in 0..2 {+ let wa =+ _mm256_loadu_si256(t0.qs.as_ptr().add(g4 * TILE_N * 4 + h * 32) as *const _);+ let wb =+ _mm256_loadu_si256(t1.qs.as_ptr().add(g4 * TILE_N * 4 + h * 32) as *const _);+ for (m, act) in acts.iter().enumerate() {+ isum_a[m][h] = _mm256_dpbusd_avx_epi32(+ isum_a[m][h],+ wa,+ bcast4_256(&act.qs, sub32 * 32 + g4 * 4),+ );+ isum_b[m][h] = _mm256_dpbusd_avx_epi32(+ isum_b[m][h],+ wb,+ bcast4_256(&act.qs, (sub32 + 1) * 32 + g4 * 4),+ );+ }+ }+ }+ for (blk, isum, tile) in [(sub32, &isum_a, t0), (sub32 + 1, &isum_b, t1)] {+ for h in 0..2 {+ let dw = _mm256_cvtph_ps(_mm_loadu_si128(tile.d.as_ptr().add(h * 8) as *const _));+ for (m, act) in acts.iter().enumerate() {+ let bsum = (act.bsums[blk * 2] as i32 + act.bsums[blk * 2 + 1] as i32) as f32;+ let f =+ _mm256_sub_ps(_mm256_cvtepi32_ps(isum[m][h]), _mm256_set1_ps(128.0 * bsum));+ acc[m][h] =+ _mm256_fmadd_ps(f, _mm256_mul_ps(dw, _mm256_set1_ps(act.d)), acc[m][h]);+ }+ }+ }+ }+}diff --git a/vendor/candle-core/src/quantized/simd128.rs b/vendor/candle-core/src/quantized/simd128.rsnew file mode 100644index 0000000..4c02f99--- /dev/null+++ b/vendor/candle-core/src/quantized/simd128.rs@@ -0,0 +1,418 @@+use super::k_quants::{BlockQ2K, BlockQ4K, BlockQ4_0, BlockQ6K, BlockQ8K, BlockQ8_0, QK8_0, QK_K};+use byteorder::{ByteOrder, LittleEndian};+use half::f16;++use core::arch::wasm32::*;++#[inline(always)]+pub(crate) fn vec_dot_q4_0_q8_0(n: usize, xs: &[BlockQ4_0], ys: &[BlockQ8_0]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK8_0),+ "vec_dot_q4_0_q8_0: {n} is not divisible by {QK8_0}"+ );+ unsafe {+ let mut acc = f32x4_splat(0.0f32);+ for (x, y) in xs.iter().zip(ys.iter()) {+ let x1234 = v128_load(x.qs.as_ptr() as *const v128);+ let x12 = v128_and(x1234, u8x16_splat(0x0F));+ let x12 = i8x16_sub(x12, i8x16_splat(8));+ let x34 = u8x16_shr(x1234, 4);+ let x34 = i8x16_sub(x34, i8x16_splat(8));++ let x1 = i16x8_extend_low_i8x16(x12);+ let y1 = i16x8_load_extend_i8x8(y.qs.as_ptr());+ let sum_xy = i32x4_dot_i16x8(x1, y1);++ let x2 = i16x8_extend_high_i8x16(x12);+ let y2 = i16x8_load_extend_i8x8(y.qs.as_ptr().add(8));+ let sum_xy = i32x4_add(sum_xy, i32x4_dot_i16x8(x2, y2));++ let x3 = i16x8_extend_low_i8x16(x34);+ let y3 = i16x8_load_extend_i8x8(y.qs.as_ptr().add(16));+ let sum_xy = i32x4_add(sum_xy, i32x4_dot_i16x8(x3, y3));++ let x4 = i16x8_extend_high_i8x16(x34);+ let y4 = i16x8_load_extend_i8x8(y.qs.as_ptr().add(24));+ let sum_xy = i32x4_add(sum_xy, i32x4_dot_i16x8(x4, y4));++ let sum_xy = f32x4_convert_i32x4(sum_xy);++ // f32x4_relaxed_madd is nightly only.+ let d = f32x4_splat(f16::to_f32(x.d) * f16::to_f32(y.d));+ let scaled = f32x4_mul(sum_xy, d);+ acc = f32x4_add(acc, scaled)+ }+ let res = f32x4_extract_lane::<0>(acc)+ + f32x4_extract_lane::<1>(acc)+ + f32x4_extract_lane::<2>(acc)+ + f32x4_extract_lane::<3>(acc);+ res+ }+}++#[inline(always)]+pub(crate) fn vec_dot_q8_0_q8_0(n: usize, xs: &[BlockQ8_0], ys: &[BlockQ8_0]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK8_0),+ "vec_dot_q8_0_q8_0: {n} is not divisible by {QK8_0}"+ );+ unsafe {+ let mut acc = f32x4_splat(0.0f32);+ for (x, y) in xs.iter().zip(ys.iter()) {+ let x1 = i16x8_load_extend_i8x8(x.qs.as_ptr());+ let y1 = i16x8_load_extend_i8x8(y.qs.as_ptr());+ let sum_xy = i32x4_dot_i16x8(x1, y1);++ let x2 = i16x8_load_extend_i8x8(x.qs.as_ptr().add(8));+ let y2 = i16x8_load_extend_i8x8(y.qs.as_ptr().add(8));+ let sum_xy = i32x4_add(sum_xy, i32x4_dot_i16x8(x2, y2));++ let x3 = i16x8_load_extend_i8x8(x.qs.as_ptr().add(16));+ let y3 = i16x8_load_extend_i8x8(y.qs.as_ptr().add(16));+ let sum_xy = i32x4_add(sum_xy, i32x4_dot_i16x8(x3, y3));++ let x4 = i16x8_load_extend_i8x8(x.qs.as_ptr().add(24));+ let y4 = i16x8_load_extend_i8x8(y.qs.as_ptr().add(24));+ let sum_xy = i32x4_add(sum_xy, i32x4_dot_i16x8(x4, y4));++ let sum_xy = f32x4_convert_i32x4(sum_xy);++ // f32x4_relaxed_madd is nightly only.+ let d = f32x4_splat(f16::to_f32(x.d) * f16::to_f32(y.d));+ let scaled = f32x4_mul(sum_xy, d);+ acc = f32x4_add(acc, scaled)+ }+ let res = f32x4_extract_lane::<0>(acc)+ + f32x4_extract_lane::<1>(acc)+ + f32x4_extract_lane::<2>(acc)+ + f32x4_extract_lane::<3>(acc);+ res+ }+}++#[inline(always)]+pub(crate) fn vec_dot_q2k_q8k(n: usize, xs: &[BlockQ2K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q2k_q8k: {n} is not divisible by {QK_K}"+ );+ unsafe {+ let mut sumf = f32x4_splat(0f32);+ for (x, y) in xs.iter().zip(ys.iter()) {+ let mut q2: &[_] = &x.qs;+ let mut q8: &[_] = &y.qs;+ let sc = &x.scales;++ let mut summs = i32x4_splat(0);+ for i in (0..(QK_K / 16)).step_by(4) {+ let bsums = i32x4_load_extend_i16x4(y.bsums.as_ptr().add(i));+ let scales = i32x4_shr(+ i32x4(+ sc[i] as i32,+ sc[i + 1] as i32,+ sc[i + 2] as i32,+ sc[i + 3] as i32,+ ),+ 4,+ );+ summs = i32x4_add(summs, i32x4_mul(bsums, scales))+ }+ let summs = f32x4_convert_i32x4(summs);++ let dall = y.d * x.d.to_f32();+ let dmin = y.d * x.dmin.to_f32();++ let mut isum = i32x4_splat(0);+ let mut is = 0;+ for _ in 0..(QK_K / 128) {+ let mut shift = 0;+ for _ in 0..4 {+ let d = (sc[is] & 0xF) as i32;+ is += 1;+ let mut isuml = i16x8_splat(0);+ for l in (0..16).step_by(8) {+ let q8 = i16x8_load_extend_i8x8(q8.as_ptr().add(l));+ let q2 = i16x8_load_extend_u8x8(q2.as_ptr().add(l));+ let q2 = v128_and(i16x8_shr(q2, shift), i16x8_splat(3));+ isuml = i16x8_add(isuml, i16x8_mul(q2, q8))+ }+ let dd = i32x4_splat(d);+ isum = i32x4_add(isum, i32x4_mul(i32x4_extend_low_i16x8(isuml), dd));+ isum = i32x4_add(isum, i32x4_mul(i32x4_extend_high_i16x8(isuml), dd));+ let d = (sc[is] & 0xF) as i32;+ is += 1;+ let mut isuml = i16x8_splat(0);+ for l in (16..32).step_by(8) {+ let q8 = i16x8_load_extend_i8x8(q8.as_ptr().add(l));+ let q2 = i16x8_load_extend_u8x8(q2.as_ptr().add(l));+ let q2 = v128_and(i16x8_shr(q2, shift), i16x8_splat(3));+ isuml = i16x8_add(isuml, i16x8_mul(q2, q8))+ }+ let dd = i32x4_splat(d);+ isum = i32x4_add(isum, i32x4_mul(i32x4_extend_low_i16x8(isuml), dd));+ isum = i32x4_add(isum, i32x4_mul(i32x4_extend_high_i16x8(isuml), dd));+ shift += 2;+ // adjust the indexing+ q8 = &q8[32..];+ }+ // adjust the indexing+ q2 = &q2[32..];+ }+ let isum = f32x4_convert_i32x4(isum);+ sumf = f32x4_add(+ sumf,+ f32x4_sub(+ f32x4_mul(isum, f32x4_splat(dall)),+ f32x4_mul(summs, f32x4_splat(dmin)),+ ),+ );+ }+ let sumf = f32x4_extract_lane::<0>(sumf)+ + f32x4_extract_lane::<1>(sumf)+ + f32x4_extract_lane::<2>(sumf)+ + f32x4_extract_lane::<3>(sumf);+ sumf+ }+}++#[inline(always)]+pub(crate) fn vec_dot_q4k_q8k(n: usize, xs: &[BlockQ4K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q4k_q8k: {n} is not divisible by {QK_K}"+ );+ const KMASK1: u32 = 0x3f3f3f3f;+ const KMASK2: u32 = 0x0f0f0f0f;+ const KMASK3: u32 = 0x03030303;++ let mut utmp: [u32; 4] = [0; 4];+ let mut scales: [u8; 8] = [0; 8];+ let mut mins: [u8; 8] = [0; 8];++ let mut aux8: [u8; QK_K] = [0; QK_K];+ let mut sums = f32x4_splat(0f32);+ unsafe {+ for (y, x) in ys.iter().zip(xs.iter()) {+ let q4 = &x.qs;+ let q8 = &y.qs;++ for j in 0..QK_K / 64 {+ let q4_1 = v128_load(q4.as_ptr().add(32 * j) as *const v128);+ let q4_2 = v128_load(q4.as_ptr().add(32 * j + 16) as *const v128);+ v128_store(+ aux8.as_mut_ptr().add(64 * j) as *mut v128,+ v128_and(q4_1, u8x16_splat(0x0F)),+ );+ v128_store(+ aux8.as_mut_ptr().add(64 * j + 16) as *mut v128,+ v128_and(q4_2, u8x16_splat(0x0F)),+ );+ v128_store(+ aux8.as_mut_ptr().add(64 * j + 32) as *mut v128,+ u8x16_shr(q4_1, 4),+ );+ v128_store(+ aux8.as_mut_ptr().add(64 * j + 48) as *mut v128,+ u8x16_shr(q4_2, 4),+ );+ }++ LittleEndian::read_u32_into(&x.scales, &mut utmp[0..3]);++ utmp[3] = ((utmp[2] >> 4) & KMASK2) | (((utmp[1] >> 6) & KMASK3) << 4);+ let uaux = utmp[1] & KMASK1;+ utmp[1] = (utmp[2] & KMASK2) | (((utmp[0] >> 6) & KMASK3) << 4);+ utmp[2] = uaux;+ utmp[0] &= KMASK1;++ //extract scales and mins+ LittleEndian::write_u32_into(&utmp[0..2], &mut scales);+ LittleEndian::write_u32_into(&utmp[2..4], &mut mins);++ let mut sumi = i32x4_splat(0);+ for j in (0..QK_K / 16).step_by(4) {+ let bsums = i32x4_load_extend_i16x4(y.bsums.as_ptr().add(j));+ let (m1, m2) = (mins[j / 2] as i32, mins[j / 2 + 1] as i32);+ let mins = i32x4(m1, m1, m2, m2);+ sumi = i32x4_add(sumi, i32x4_mul(bsums, mins));+ }++ let mut aux32 = i32x4_splat(0i32);+ for (scale_i, scale) in scales.iter().enumerate() {+ let scale = i32x4_splat(*scale as i32);+ for j in 0..4 {+ let i = 32 * scale_i + 8 * j;+ let q8 = i16x8_load_extend_i8x8(q8.as_ptr().add(i));+ let aux8 = i16x8_load_extend_u8x8(aux8.as_ptr().add(i));+ let aux16 = i16x8_mul(q8, aux8);+ aux32 = i32x4_add(aux32, i32x4_mul(scale, i32x4_extend_low_i16x8(aux16)));+ aux32 = i32x4_add(aux32, i32x4_mul(scale, i32x4_extend_high_i16x8(aux16)));+ }+ }+ let aux32 = f32x4_convert_i32x4(aux32);+ let d = f32x4_splat(x.d.to_f32() * y.d);+ sums = f32x4_add(sums, f32x4_mul(aux32, d));+ let dmin = x.dmin.to_f32() * y.d;+ let dmin = f32x4_splat(dmin);+ let sumi = f32x4_convert_i32x4(sumi);+ sums = f32x4_sub(sums, f32x4_mul(sumi, dmin));+ }+ let sums = f32x4_extract_lane::<0>(sums)+ + f32x4_extract_lane::<1>(sums)+ + f32x4_extract_lane::<2>(sums)+ + f32x4_extract_lane::<3>(sums);+ sums+ }+}++#[inline(always)]+pub(crate) fn vec_dot_q6k_q8k(n: usize, xs: &[BlockQ6K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q6k_q8k: {n} is not divisible by {QK_K}"+ );+ let mut aux8 = [0i8; QK_K];+ unsafe {+ let mut sums = f32x4_splat(0f32);++ for (x, y) in xs.iter().zip(ys.iter()) {+ let q4 = &x.ql;+ let qh = &x.qh;+ let q8 = &y.qs;+ let mut aux32 = f32x4_splat(0f32);++ for j in (0..QK_K).step_by(128) {+ let aux8 = aux8.as_mut_ptr().add(j);+ let q4 = &q4.as_ptr().add(j / 2);+ let qh = &qh.as_ptr().add(j / 4);+ for l in (0..32).step_by(16) {+ // aux8[l] = (((q4[l] & 0xF) | ((qh[l] & 3) << 4)) as i32 - 32) as i8;+ let a8 = v128_or(+ v128_and(v128_load(q4.add(l) as *const v128), u8x16_splat(0xF)),+ u8x16_shl(+ v128_and(v128_load(qh.add(l) as *const v128), u8x16_splat(3)),+ 4,+ ),+ );+ let a8_low = i16x8_sub(i16x8_extend_low_u8x16(a8), i16x8_splat(32));+ let a8_high = i16x8_sub(i16x8_extend_high_u8x16(a8), i16x8_splat(32));+ v128_store(+ aux8.add(l) as *mut v128,+ i8x16_narrow_i16x8(a8_low, a8_high),+ );++ // aux8[l + 32] =+ // (((q4[l + 32] & 0xF) | (((qh[l] >> 2) & 3) << 4)) as i32 - 32) as i8;+ let a8 = v128_or(+ v128_and(v128_load(q4.add(l + 32) as *const v128), u8x16_splat(0xF)),+ u8x16_shl(+ v128_and(+ u8x16_shr(v128_load(qh.add(l) as *const v128), 2),+ u8x16_splat(3),+ ),+ 4,+ ),+ );+ let a8_low = i16x8_sub(i16x8_extend_low_u8x16(a8), i16x8_splat(32));+ let a8_high = i16x8_sub(i16x8_extend_high_u8x16(a8), i16x8_splat(32));+ v128_store(+ aux8.add(l + 32) as *mut v128,+ i8x16_narrow_i16x8(a8_low, a8_high),+ );++ // aux8[l + 64] = (((q4[l] >> 4) | (((qh[l] >> 4) & 3) << 4)) as i32 - 32) as i8;+ let a8 = v128_or(+ u8x16_shr(v128_load(q4.add(l) as *const v128), 4),+ u8x16_shl(+ v128_and(+ u8x16_shr(v128_load(qh.add(l) as *const v128), 4),+ u8x16_splat(3),+ ),+ 4,+ ),+ );+ let a8_low = i16x8_sub(i16x8_extend_low_u8x16(a8), i16x8_splat(32));+ let a8_high = i16x8_sub(i16x8_extend_high_u8x16(a8), i16x8_splat(32));+ v128_store(+ aux8.add(l + 64) as *mut v128,+ i8x16_narrow_i16x8(a8_low, a8_high),+ );++ // aux8[l + 96] =+ // (((q4[l + 32] >> 4) | (((qh[l] >> 6) & 3) << 4)) as i32 - 32) as i8;+ let a8 = v128_or(+ u8x16_shr(v128_load(q4.add(l + 32) as *const v128), 4),+ u8x16_shl(+ v128_and(+ u8x16_shr(v128_load(qh.add(l) as *const v128), 6),+ u8x16_splat(3),+ ),+ 4,+ ),+ );+ let a8_low = i16x8_sub(i16x8_extend_low_u8x16(a8), i16x8_splat(32));+ let a8_high = i16x8_sub(i16x8_extend_high_u8x16(a8), i16x8_splat(32));+ v128_store(+ aux8.add(l + 96) as *mut v128,+ i8x16_narrow_i16x8(a8_low, a8_high),+ );+ }+ }++ for (j, &scale) in x.scales.iter().enumerate() {+ let scale = f32x4_splat(scale as f32);+ for offset in [0, 8] {+ let aux16 = i16x8_mul(+ i16x8_load_extend_i8x8(q8.as_ptr().add(16 * j + offset)),+ i16x8_load_extend_i8x8(aux8.as_ptr().add(16 * j + offset)),+ );+ aux32 = f32x4_add(+ aux32,+ f32x4_mul(f32x4_convert_i32x4(i32x4_extend_low_i16x8(aux16)), scale),+ );+ aux32 = f32x4_add(+ aux32,+ f32x4_mul(f32x4_convert_i32x4(i32x4_extend_high_i16x8(aux16)), scale),+ );+ }+ }++ let d = f32x4_splat(x.d.to_f32() * y.d);+ sums = f32x4_add(sums, f32x4_mul(aux32, d));+ }+ let sums = f32x4_extract_lane::<0>(sums)+ + f32x4_extract_lane::<1>(sums)+ + f32x4_extract_lane::<2>(sums)+ + f32x4_extract_lane::<3>(sums);+ sums+ }+}++#[inline(always)]+pub(crate) fn vec_dot_q8k_q8k(n: usize, xs: &[BlockQ8K], ys: &[BlockQ8K]) -> f32 {+ debug_assert!(+ n.is_multiple_of(QK_K),+ "vec_dot_q8k_q8k: {n} is not divisible by {QK_K}"+ );+ unsafe {+ let mut acc = f32x4_splat(0.0f32);+ for (xs, ys) in xs.iter().zip(ys.iter()) {+ let x_qs = xs.qs.as_ptr();+ let y_qs = ys.qs.as_ptr();+ let mut sumi = i32x4_splat(0);+ for j in (0..QK_K).step_by(8) {+ let xs = i16x8_load_extend_i8x8(x_qs.add(j));+ let ys = i16x8_load_extend_i8x8(y_qs.add(j));+ let sum_xy = i32x4_dot_i16x8(xs, ys);+ sumi = i32x4_add(sumi, sum_xy)+ }+ let d = f32x4_splat(xs.d * ys.d);+ acc = f32x4_add(acc, f32x4_mul(f32x4_convert_i32x4(sumi), d))+ }+ let res = f32x4_extract_lane::<0>(acc)+ + f32x4_extract_lane::<1>(acc)+ + f32x4_extract_lane::<2>(acc)+ + f32x4_extract_lane::<3>(acc);+ res+ }+}diff --git a/vendor/candle-core/src/quantized/tokenizer.rs b/vendor/candle-core/src/quantized/tokenizer.rsnew file mode 100644index 0000000..9172d95--- /dev/null+++ b/vendor/candle-core/src/quantized/tokenizer.rs@@ -0,0 +1,391 @@+use crate::quantized::gguf_file;+use crate::{Context, Error, Result};+use std::collections::HashSet;+use tokenizers::{+ decoders::{byte_level::ByteLevel as ByteLevelDecoder, DecoderWrapper},+ models::bpe::{Vocab, BPE},+ normalizers::{unicode::NFC, NormalizerWrapper},+ pre_tokenizers::{+ byte_level::ByteLevel as ByteLevelPre,+ sequence::Sequence,+ split::{Split, SplitPattern},+ PreTokenizerWrapper,+ },+ processors::sequence::Sequence as ProcessorSequence,+ processors::{byte_level::ByteLevel as ByteLevelProcessor, PostProcessorWrapper},+ tokenizer::SplitDelimiterBehavior,+ AddedToken, Tokenizer,+};++pub trait TokenizerFromGguf: Sized {+ fn from_gguf(ct: &gguf_file::Content) -> Result<Self>;+}++fn metadata_value<'a>(ct: &'a gguf_file::Content, key: &str) -> Result<&'a gguf_file::Value> {+ ct.metadata+ .get(key)+ .with_context(|| format!("missing GGUF metadata key `{key}`"))+}++fn gguf_value_to_u32(v: &gguf_file::Value) -> Result<u32> {+ use gguf_file::Value::*;+ match v {+ U8(v) => Ok(*v as u32),+ I8(v) => Ok(*v as u32),+ U16(v) => Ok(*v as u32),+ I16(v) => Ok(*v as u32),+ U32(v) => Ok(*v),+ I32(v) => Ok(*v as u32),+ U64(v) => Ok(*v as u32),+ I64(v) => Ok(*v as u32),+ _ => crate::bail!("expected numeric value for token type/id, got {v:?}"),+ }+}++fn value_to_string_array(v: &gguf_file::Value, name: &str) -> Result<Vec<String>> {+ let arr = v+ .to_vec()+ .with_context(|| format!("`{name}` is not an array"))?;+ arr.iter()+ .map(|v| {+ v.to_string()+ .map(|s| s.to_string())+ .with_context(|| format!("`{name}` element is not a string: {v:?}"))+ })+ .collect()+}++fn merges_from_value(v: &gguf_file::Value) -> Result<Vec<(String, String)>> {+ value_to_string_array(v, "tokenizer.ggml.merges")?+ .into_iter()+ .map(|m| {+ m.split_once(' ')+ .map(|(a, b)| (a.to_string(), b.to_string()))+ .ok_or_else(|| Error::msg(format!("invalid merge entry `{m}`")))+ })+ .collect()+}++struct Pipeline {+ normalizer: Option<NormalizerWrapper>,+ pretokenizer: Option<PreTokenizerWrapper>,+ decoder: Option<DecoderWrapper>,+ post_processor: Option<PostProcessorWrapper>,+}++impl Pipeline {+ fn apply(self, tokenizer: &mut Tokenizer) -> Result<()> {+ if let Some(norm) = self.normalizer {+ tokenizer.with_normalizer(Some(norm)).map_err(Error::wrap)?;+ }+ if let Some(pt) = self.pretokenizer {+ tokenizer.with_pre_tokenizer(Some(pt));+ }+ if let Some(dec) = self.decoder {+ tokenizer.with_decoder(Some(dec));+ }+ if let Some(pp) = self.post_processor {+ tokenizer.with_post_processor(Some(pp));+ }+ Ok(())+ }+}++fn pre_tokenizer_sequence(regex: &str, byte_level: ByteLevelPre) -> Result<PreTokenizerWrapper> {+ let split = Split::new(+ SplitPattern::Regex(regex.to_string()),+ SplitDelimiterBehavior::Isolated,+ false,+ )+ .map_err(Error::wrap)?;+ Ok(Sequence::new(vec![split.into(), byte_level.into()]).into())+}++fn pipeline_from_pre(pre: &str) -> Result<Pipeline> {+ const REGEX_QWEN2: &str = r"(?:'[sS]|'[tT]|'[rR][eE]|'[vV][eE]|'[mM]|'[lL][lL]|'[dD])|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+";+ const REGEX_LLAMA3: &str = r"(?:'[sS]|'[tT]|'[rR][eE]|'[vV][eE]|'[mM]|'[lL][lL]|'[dD])|[^\r\n\p{L}\p{N}]?\p{L}+|\p{N}{1,3}| ?[^\s\p{L}\p{N}]+[\r\n]*|\s*[\r\n]+|\s+(?!\S)|\s+";++ Ok(match pre {+ // Matches Qwen2 tokenizer.json settings+ "qwen2" => Pipeline {+ normalizer: Some(NFC.into()),+ pretokenizer: Some(pre_tokenizer_sequence(+ REGEX_QWEN2,+ ByteLevelPre::new(false, false, false),+ )?),+ decoder: Some(ByteLevelDecoder::new(false, false, false).into()),+ post_processor: Some(ByteLevelProcessor::new(false, false, false).into()),+ },+ // Matches Smaug/Llama3 style byte-level BPE+ "smaug-bpe" | "lfm2" | "llama3" => Pipeline {+ normalizer: None,+ pretokenizer: Some(pre_tokenizer_sequence(+ REGEX_LLAMA3,+ ByteLevelPre::new(false, true, false),+ )?),+ decoder: Some(ByteLevelDecoder::new(true, true, true).into()),+ post_processor: Some(ByteLevelProcessor::new(true, false, true).into()),+ },+ // Default GPT-2 style BPE+ _ => Pipeline {+ normalizer: None,+ pretokenizer: Some(ByteLevelPre::default().into()),+ decoder: Some(ByteLevelDecoder::default().into()),+ post_processor: Some(ByteLevelProcessor::default().into()),+ },+ })+}++fn template_processor(+ tokens: &[String],+ bos_id: Option<u32>,+ eos_id: Option<u32>,+ add_bos: bool,+ add_eos: bool,+) -> Option<PostProcessorWrapper> {+ if (!add_bos && !add_eos) || tokens.is_empty() {+ return None;+ }++ let bos = bos_id.and_then(|id| tokens.get(id as usize)).cloned();+ let eos = eos_id.and_then(|id| tokens.get(id as usize)).cloned();++ let mut specials = Vec::new();+ if add_bos {+ let bos_id = bos_id?;+ let bos_tok = bos.clone()?;+ specials.push((bos_tok.clone(), bos_id));+ }+ if add_eos {+ let eos_id = eos_id?;+ let eos_tok = eos.clone()?;+ specials.push((eos_tok.clone(), eos_id));+ }++ let mut single = Vec::new();+ if add_bos {+ single.push(bos.clone()?);+ }+ single.push("$0".to_string());+ if add_eos {+ single.push(eos.clone()?);+ }++ let mut pair = Vec::new();+ if add_bos {+ pair.push(format!("{}:0", bos.clone()?));+ }+ pair.push("$A:0".to_string());+ if add_eos {+ pair.push(format!("{}:0", eos.clone()?));+ }+ if add_bos {+ pair.push(format!("{}:1", bos.clone()?));+ }+ pair.push("$B:1".to_string());+ if add_eos {+ pair.push(format!("{}:1", eos.clone()?));+ }++ let proc = tokenizers::processors::template::TemplateProcessing::builder()+ .try_single(single)+ .ok()?+ .try_pair(pair)+ .ok()?+ .special_tokens(specials)+ .build()+ .ok()?;++ Some(PostProcessorWrapper::Template(proc))+}++impl TokenizerFromGguf for Tokenizer {+ fn from_gguf(ct: &gguf_file::Content) -> Result<Self> {+ let model_kind = metadata_value(ct, "tokenizer.ggml.model")?+ .to_string()?+ .to_lowercase();+ if model_kind != "gpt2" {+ crate::bail!("unsupported tokenizer model `{model_kind}`");+ }++ let tokens = value_to_string_array(+ metadata_value(ct, "tokenizer.ggml.tokens")?,+ "tokenizer.ggml.tokens",+ )?;+ let vocab: Vocab = tokens+ .iter()+ .enumerate()+ .map(|(i, t)| (t.clone(), i as u32))+ .collect();+ let merges = merges_from_value(metadata_value(ct, "tokenizer.ggml.merges")?)?;++ let mut builder = BPE::builder().vocab_and_merges(vocab, merges);++ if let Ok(val) = metadata_value(ct, "tokenizer.ggml.unk_token_id") {+ let token_id = gguf_value_to_u32(val)?;+ if let Some(token) = tokens.get(token_id as usize) {+ builder = builder.unk_token(token.clone());+ }+ }++ if let Ok(val) = metadata_value(ct, "tokenizer.ggml.byte_fallback") {+ builder = builder.byte_fallback(val.to_bool()?);+ }++ if let Ok(val) = metadata_value(ct, "tokenizer.ggml.ignore_merges") {+ builder = builder.ignore_merges(val.to_bool()?);+ }++ let bpe = builder.build().map_err(Error::wrap)?;+ let mut tokenizer = Tokenizer::new(bpe);++ let pre = metadata_value(ct, "tokenizer.ggml.pre")+ .and_then(|v| v.to_string())+ .map(|s| s.to_string())+ .unwrap_or_else(|_| "gpt2".to_string());+ let pipeline = pipeline_from_pre(pre.as_str())?;+ let post_processor_base = pipeline.post_processor.clone();++ let add_bos = metadata_value(ct, "tokenizer.ggml.add_bos_token")+ .and_then(|v| v.to_bool())+ .unwrap_or(false);+ let add_eos = metadata_value(ct, "tokenizer.ggml.add_eos_token")+ .and_then(|v| v.to_bool())+ .unwrap_or(false);+ let bos_id = metadata_value(ct, "tokenizer.ggml.bos_token_id")+ .and_then(gguf_value_to_u32)+ .ok();+ let eos_id = metadata_value(ct, "tokenizer.ggml.eos_token_id")+ .and_then(gguf_value_to_u32)+ .ok();++ pipeline.apply(&mut tokenizer)?;++ // Compose existing post-processor with a template-based one if needed+ let template_pp = template_processor(&tokens, bos_id, eos_id, add_bos, add_eos);+ if template_pp.is_some() || post_processor_base.is_some() {+ let mut steps = Vec::new();+ if let Some(pp) = post_processor_base {+ steps.push(pp);+ }+ if let Some(tp) = template_pp {+ steps.push(tp);+ }+ let pp = if steps.len() == 1 {+ steps.pop().unwrap()+ } else {+ ProcessorSequence::new(steps).into()+ };+ tokenizer.with_post_processor(Some(pp));+ }++ // Mark special tokens so decode(skip_special_tokens = true) behaves as expected+ if let Ok(gguf_file::Value::Array(arr)) = metadata_value(ct, "tokenizer.ggml.token_type") {+ let mut specials = Vec::new();+ for (idx, v) in arr.iter().enumerate() {+ let ty = gguf_value_to_u32(v)?;+ // Aligns with llama_token_type: treat non-normal/non-byte tokens as special.+ let is_special = matches!(ty, 2..=5);+ if is_special {+ if let Some(tok) = tokens.get(idx) {+ specials.push(AddedToken::from(tok.clone(), true));+ }+ }+ }+ if !specials.is_empty() {+ tokenizer+ .add_special_tokens(specials)+ .map_err(Error::wrap)?;+ }+ }++ let mut explicit_specials = HashSet::new();+ for key in [+ "tokenizer.ggml.bos_token_id",+ "tokenizer.ggml.eos_token_id",+ "tokenizer.ggml.pad_token_id",+ "tokenizer.ggml.sep_token_id",+ "tokenizer.ggml.unk_token_id",+ ] {+ if let Ok(val) = metadata_value(ct, key) {+ explicit_specials.insert(gguf_value_to_u32(val)?);+ }+ }+ if !explicit_specials.is_empty() {+ let specials: Vec<_> = explicit_specials+ .into_iter()+ .filter_map(|id| tokens.get(id as usize))+ .map(|tok| AddedToken::from(tok.clone(), true))+ .collect();+ if !specials.is_empty() {+ tokenizer+ .add_special_tokens(specials)+ .map_err(Error::wrap)?;+ }+ }++ Ok(tokenizer)+ }+}++#[cfg(test)]+mod tests {+ use super::*;+ use tokenizers::tokenizer::{OffsetReferential, OffsetType, PreTokenizedString, PreTokenizer};++ fn splits(pre: &str, input: &str) -> Vec<String> {+ let pipeline = pipeline_from_pre(pre).unwrap();+ let mut pre_tokenized = PreTokenizedString::from(input);+ pipeline+ .pretokenizer+ .unwrap()+ .pre_tokenize(&mut pre_tokenized)+ .unwrap();+ pre_tokenized+ .get_splits(OffsetReferential::Original, OffsetType::Byte)+ .into_iter()+ .map(|(s, _, _)| s.to_string())+ .collect()+ }++ // Sample made to test negative lookahead like (`\s+(?!\S)`).+ // If someone changes the backend to one that does not support backtracking regex the tests below will fail.+ const SAMPLE: &str = "Hello'Ve world 123\n\n foo";++ #[test]+ fn qwen2_pretokenizer_splits() {+ assert_eq!(+ splits("qwen2", SAMPLE),+ [+ "Hello",+ "'Ve",+ "\u{120}world",+ "\u{120}",+ "1",+ "2",+ "3",+ "\u{10a}\u{10a}",+ "\u{120}",+ "\u{120}foo"+ ]+ );+ }++ #[test]+ fn llama3_pretokenizer_splits() {+ // Differs from qwen2 only in digit grouping: `\p{N}{1,3}` vs `\p{N}`.+ assert_eq!(+ splits("llama3", SAMPLE),+ [+ "Hello",+ "'Ve",+ "\u{120}world",+ "\u{120}",+ "123",+ "\u{10a}\u{10a}",+ "\u{120}",+ "\u{120}foo"+ ]+ );+ }+}diff --git a/vendor/candle-core/src/quantized/utils.rs b/vendor/candle-core/src/quantized/utils.rsnew file mode 100644index 0000000..9dd9c09--- /dev/null+++ b/vendor/candle-core/src/quantized/utils.rs@@ -0,0 +1,562 @@+pub(super) fn nearest_int(v: f32) -> i32 {+ v.round() as i32+}++/// Validates that the input and output are the right size and returns an iterator which maps each+/// input region `xs` to its corresponding output block in `ys`. Each output region is guaranteed+/// to be `T::BLCK_SIZE` long.+pub(super) fn group_for_quantization<'a, 'b, T: super::k_quants::GgmlType>(+ xs: &'b [f32],+ ys: &'a mut [T],+) -> Vec<(&'a mut T, &'b [f32])> {+ let block_size = T::BLCK_SIZE;+ let dtype = T::DTYPE;++ let expected_blocks = xs.len() / block_size;+ let actual_blocks = ys.len();++ // Validate that the input is the right size+ debug_assert_eq!(+ expected_blocks,+ actual_blocks,+ "quantize {dtype:?}: expected {expected_blocks} blocks but only {actual_blocks} were provided!");++ ys.iter_mut().zip(xs.chunks_exact(block_size)).collect()+}++/// Validates that the input and output are the right size and returns an iterator which maps each+/// input block `xs` to its corresponding output region in `ys`. Each output region is guaranteed+/// to be `T::BLCK_SIZE` long.+pub(super) fn group_for_dequantization<'a, 'b, T: super::k_quants::GgmlType>(+ xs: &'a [T],+ ys: &'b mut [f32],+) -> Vec<(&'a T, &'b mut [f32])> {+ let block_size = T::BLCK_SIZE;+ let dtype = T::DTYPE;++ let actual_output_len = ys.len();+ let expected_output_len = xs.len() * block_size;+ // Validate that the output is the right size+ debug_assert_eq!(+ expected_output_len,+ actual_output_len,+ "dequantize {dtype:?}: ys (len = {actual_output_len}) does not match the expected length of {expected_output_len}!"+ );++ // Zip the blocks and outputs together+ xs.iter().zip(ys.chunks_exact_mut(block_size)).collect()+}++pub(super) fn get_scale_min_k4(j: usize, q: &[u8]) -> (u8, u8) {+ if j < 4 {+ let d = q[j] & 63;+ let m = q[j + 4] & 63;+ (d, m)+ } else {+ let d = (q[j + 4] & 0xF) | ((q[j - 4] >> 6) << 4);+ let m = (q[j + 4] >> 4) | ((q[j] >> 6) << 4);+ (d, m)+ }+}++pub(super) unsafe fn make_qx_quants(+ n: usize,+ nmax: i32,+ x: *const f32,+ ls: *mut i8,+ rmse_type: i32,+ qw: *const f32,+) -> f32 {+ let mut max = 0f32;+ let mut amax = 0f32;+ for i in 0..n {+ let x = *x.add(i);+ let ax = x.abs();+ if ax > amax {+ amax = ax;+ max = x;+ }+ }+ if amax == 0. {+ // all zero+ for i in 0..n {+ *ls.add(i) = 0;+ }+ return 0.;+ }+ let mut iscale = -(nmax as f32) / max;+ if rmse_type == 0 {+ for i in 0..n {+ let x = *x.add(i);+ let l = nearest_int(iscale * x);+ *ls.add(i) = (nmax + l.clamp(-nmax, nmax - 1)) as i8;+ }+ return 1.0 / iscale;+ }+ let weight_type = rmse_type % 2;+ let mut sumlx = 0f32;+ let mut suml2 = 0f32;+ for i in 0..n {+ let x = *x.add(i);+ let l = nearest_int(iscale * x);+ let l = l.clamp(-nmax, nmax - 1);+ *ls.add(i) = (l + nmax) as i8;+ let w = if !qw.is_null() {+ *qw.add(i)+ } else if weight_type == 1 {+ x * x+ } else {+ 1.0+ };+ let l = l as f32;+ sumlx += w * x * l;+ suml2 += w * l * l;+ }+ let mut scale = sumlx / suml2;+ let mut best = scale * sumlx;+ for _itry in 0..3 {+ let iscale = 1.0 / scale;+ let mut slx = 0f32;+ let mut sl2 = 0f32;+ let mut changed = false;+ for i in 0..n {+ let x = *x.add(i);+ let l = nearest_int(iscale * x);+ let l = l.clamp(-nmax, nmax - 1);+ if l + nmax != *ls.add(i) as i32 {+ changed = true;+ }+ let w = if !qw.is_null() {+ *qw.add(i)+ } else if weight_type == 1 {+ x * x+ } else {+ 1.0+ };+ let l = l as f32;+ slx += w * x * l;+ sl2 += w * l * l;+ }+ if !changed || sl2 == 0.0 || slx * slx <= best * sl2 {+ break;+ }+ for i in 0..n {+ let x = *x.add(i);+ let l = nearest_int(iscale * x);+ *ls.add(i) = (nmax + l.clamp(-nmax, nmax - 1)) as i8;+ }+ sumlx = slx;+ suml2 = sl2;+ scale = sumlx / suml2;+ best = scale * sumlx;+ }+ for _itry in 0..5 {+ let mut n_changed = 0;+ for i in 0..n {+ let x = *x.add(i);+ let w = if !qw.is_null() {+ *qw.add(i)+ } else if weight_type == 1 {+ x * x+ } else {+ 1.0+ };+ let l = *ls.add(i) as i32 - nmax;+ let mut slx = sumlx - w * x * l as f32;+ if slx > 0. {+ let mut sl2 = suml2 - w * l as f32 * l as f32;+ let new_l = nearest_int(x * sl2 / slx);+ let new_l = new_l.clamp(-nmax, nmax - 1);+ if new_l != l {+ slx += w * x * new_l as f32;+ sl2 += w * new_l as f32 * new_l as f32;+ if sl2 > 0. && slx * slx * suml2 > sumlx * sumlx * sl2 {+ *ls.add(i) = (nmax + new_l) as i8;+ sumlx = slx;+ suml2 = sl2;+ scale = sumlx / suml2;+ best = scale * sumlx;+ n_changed += 1;+ }+ }+ }+ }+ if n_changed == 0 {+ break;+ }+ }+ if rmse_type < 3 {+ return scale;+ }+ for is in -4..4 {+ if is == 0 {+ continue;+ }+ iscale = -(nmax as f32 + 0.1f32 * is as f32) / max;+ let mut sumlx = 0.;+ let mut suml2 = 0.;+ for i in 0..n {+ let x = *x.add(i);+ let l = nearest_int(iscale * x);+ let l = l.clamp(-nmax, nmax - 1);+ let w = if !qw.is_null() {+ *qw.add(i)+ } else if weight_type == 1 {+ x * x+ } else {+ 1.0+ };+ let l = l as f32;+ sumlx += w * x * l;+ suml2 += w * l * l;+ }+ if suml2 > 0. && sumlx * sumlx > best * suml2 {+ for i in 0..n {+ let x = *x.add(i);+ let l = nearest_int(iscale * x);+ *ls.add(i) = (nmax + l.clamp(-nmax, nmax - 1)) as i8;+ }+ scale = sumlx / suml2;+ best = scale * sumlx;+ }+ }+ scale+}++// https://github.com/ggerganov/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/k_quants.c#L224+pub(super) fn make_qkx1_quants(nmax: i32, ntry: usize, x: &[f32]) -> (f32, f32) {+ let n = x.len();+ let mut l = vec![0; n];+ // Get min/max+ let min = *x+ .iter()+ .take(n)+ .min_by(|a, b| a.total_cmp(b))+ .unwrap_or(&x[0]);+ let max = *x.iter().max_by(|a, b| a.total_cmp(b)).unwrap_or(&x[0]);++ // If min == max, all values are the same => nothing to do here+ if max == min {+ return (0.0, 0.0);+ }++ // Ensure min <= 0.0+ let mut min = min.min(0.);++ // Compute scale and inverse scale+ let mut iscale = nmax as f32 / (max - min);+ let mut scale = 1.0 / iscale;++ for _ in 0..ntry {+ let mut sumlx = 0.0;+ let mut suml2 = 0;+ let mut did_change = false;++ for (i, value) in x.iter().enumerate().take(n) {+ let li = nearest_int(iscale * (value - min)).clamp(0, nmax);+ let clamped_li = li as u8;+ if clamped_li != l[i] {+ l[i] = clamped_li;+ did_change = true;+ }+ sumlx += (value - min) * li as f32;+ suml2 += li * li;+ }+ scale = sumlx / suml2 as f32;++ let sum: f32 = x+ .iter()+ .take(n)+ .zip(l.iter().take(n))+ .map(|(xi, &li)| xi - scale * li as f32)+ .sum();++ min = sum / n as f32;+ if min > 0.0 {+ min = 0.0;+ }+ iscale = 1.0 / scale;+ if !did_change {+ break;+ }+ }+ (scale, -min)+}++// https://github.com/ggerganov/llama.cpp/blob/8183159cf3def112f6d1fe94815fce70e1bffa12/k_quants.c#L165+pub(super) fn make_q3_quants(x: &[f32], nmax: i32, do_rmse: bool) -> f32 {+ let n = x.len();+ let mut l = vec![0i8; n];++ let mut max = 0.0;+ let mut amax = 0.0;+ for &xi in x.iter().take(n) {+ let ax = xi.abs();+ if ax > amax {+ amax = ax;+ max = xi;+ }+ }++ if amax == 0.0 {+ return 0.0;+ }++ let iscale = -(nmax as f32) / max;+ if do_rmse {+ let mut sumlx = 0.0;+ let mut suml2 = 0.0;+ for i in 0..n {+ let li = (iscale * x[i]).round() as i32;+ let li = li.clamp(-nmax, nmax - 1);+ l[i] = li as i8;+ let w = x[i] * x[i];+ sumlx += w * x[i] * li as f32;+ suml2 += w * (li * li) as f32;+ }+ for _ in 0..5 {+ let mut n_changed = 0;+ for i in 0..n {+ let w = x[i] * x[i];+ let mut slx = sumlx - w * x[i] * l[i] as f32;+ if slx > 0.0 {+ let mut sl2 = suml2 - w * (l[i] as i32 * l[i] as i32) as f32;+ let mut new_l = (x[i] * sl2 / slx).round() as i32;+ new_l = new_l.clamp(-nmax, nmax - 1);+ if new_l != l[i] as i32 {+ slx += w * x[i] * new_l as f32;+ sl2 += w * (new_l * new_l) as f32;+ if sl2 > 0.0 && slx * slx * suml2 > sumlx * sumlx * sl2 {+ l[i] = new_l as i8;+ sumlx = slx;+ suml2 = sl2;+ n_changed += 1;+ }+ }+ }+ }+ if n_changed == 0 {+ break;+ }+ }+ for li in l.iter_mut() {+ *li += nmax as i8;+ }+ return sumlx / suml2;+ }+ for i in 0..n {+ let li = (iscale * x[i]).round() as i32;+ l[i] = (li.clamp(-nmax, nmax - 1) + nmax) as i8;+ }+ 1.0 / iscale+}++// https://github.com/ggerganov/llama.cpp/blob/678d7994f4da0af3d29046be99950ac999ee9762/ggml/src/ggml-quants.c#L744+/// (scale, min)+pub(super) fn make_qkx3_quants(+ nmax: i32,+ x: &[f32],+ weights: Option<&[f32]>,+ rmin: f32,+ rdelta: f32,+ nstep: usize,+ use_mad: bool,+) -> (f32, f32) {+ let n = x.len();+ let mut l: [u8; 32] = [0; 32];+ let mut l_aux: [u8; 32] = [0; 32];++ let mut min_val = x[0];+ let mut max_val = x[0];+ let mut sum_w = match weights {+ Some(w) => w[0],+ None => x[0] * x[0],+ };+ let mut sum_x = sum_w * x[0];++ for i in 1..n {+ if x[i] < min_val {+ min_val = x[i];+ }+ if x[i] > max_val {+ max_val = x[i];+ }+ let w = match weights {+ Some(w) => w[i],+ None => x[i] * x[i],+ };+ sum_w += w;+ sum_x += w * x[i];+ }++ if min_val > 0.0 {+ min_val = 0.0;+ }++ if max_val <= min_val {+ return (0.0, -min_val);+ }++ let mut iscale = nmax as f32 / (max_val - min_val);+ let mut scale = 1.0 / iscale;+ let mut best_mad = 0.0;++ for i in 0..n {+ let l_val = nearest_int(iscale * (x[i] - min_val)).clamp(0, nmax) as u8;+ l[i] = l_val;+ let diff = scale * (l_val as f32) + min_val - x[i];+ let diff = if use_mad { diff.abs() } else { diff * diff };+ let w = match weights {+ Some(w) => w[i],+ None => x[i] * x[i],+ };+ best_mad += w * diff;+ }++ if nstep < 1 {+ return (scale, -min_val);+ }++ for is in 0..=nstep {+ iscale = (rmin + rdelta * is as f32 + nmax as f32) / (max_val - min_val);+ let (mut sum_l, mut sum_l2, mut sum_xl) = (0.0, 0.0, 0.0);++ for i in 0..n {+ let l_val = nearest_int(iscale * (x[i] - min_val)).clamp(0, nmax) as u8;+ l_aux[i] = l_val;+ let w = match weights {+ Some(w) => w[i],+ None => x[i] * x[i],+ };+ sum_l += w * l_val as f32;+ sum_l2 += w * (l_val as f32).powi(2);+ sum_xl += w * l_val as f32 * x[i];+ }++ let d = sum_w * sum_l2 - sum_l * sum_l;+ if d > 0.0 {+ let mut this_scale = (sum_w * sum_xl - sum_x * sum_l) / d;+ let mut this_min = (sum_l2 * sum_x - sum_l * sum_xl) / d;++ if this_min > 0.0 {+ this_min = 0.0;+ this_scale = sum_xl / sum_l2;+ }++ let mut mad = 0.0;+ for i in 0..n {+ let diff = this_scale * (l_aux[i] as f32) + this_min - x[i];+ let diff = if use_mad { diff.abs() } else { diff * diff };+ let w = match weights {+ Some(w) => w[i],+ None => x[i] * x[i],+ };+ mad += w * diff;+ }++ if mad < best_mad {+ l.copy_from_slice(&l_aux);+ best_mad = mad;+ scale = this_scale;+ min_val = this_min;+ }+ }+ }++ (scale, -min_val)+}++// https://github.com/ggerganov/llama.cpp/blob/678d7994f4da0af3d29046be99950ac999ee9762/ggml/src/ggml-quants.c#L827+pub(super) fn make_qp_quants(+ n: usize,+ nmax: u8,+ x: &[f32],+ l: &mut [u8],+ quant_weights: &[f32],+) -> f32 {+ assert_eq!(x.len(), n);+ assert_eq!(l.len(), n);+ assert_eq!(quant_weights.len(), n);++ let max = x.iter().copied().fold(0.0, f32::max);+ if max == 0.0 {+ l.iter_mut().for_each(|li| *li = 0);+ return 0.0;+ }++ let mut iscale = nmax as f32 / max;+ for (xi, li) in x.iter().zip(l.iter_mut()) {+ *li = nearest_int(iscale * xi) as u8;+ }++ let scale = 1.0 / iscale;+ let mut best_mse = x+ .iter()+ .zip(l.iter())+ .zip(quant_weights.iter())+ .map(|((&xi, &li), &w)| {+ let diff = xi - scale * li as f32;+ w * diff * diff+ })+ .sum::<f32>();++ for is in -4..=4 {+ if is == 0 {+ continue;+ }+ let iscale_is = (0.1 * is as f32 + nmax as f32) / max;+ let scale_is = 1.0 / iscale_is;++ let mse = x+ .iter()+ .zip(quant_weights.iter())+ .map(|(&xi, &w)| {+ let mut li = nearest_int(iscale_is * xi) as u8;+ li = li.min(nmax);+ let diff = xi - scale_is * li as f32;+ w * diff * diff+ })+ .sum::<f32>();++ if mse < best_mse {+ best_mse = mse;+ iscale = iscale_is;+ }+ }++ let mut sumlx = 0.0;+ let mut suml2 = 0.0;+ for ((xi, li), &w) in x.iter().zip(l.iter_mut()).zip(quant_weights.iter()) {+ let mut li_new = (iscale * xi).round() as u8;+ li_new = li_new.min(nmax);+ *li = li_new;+ sumlx += w * xi * li_new as f32;+ suml2 += w * (li_new as f32).powi(2);+ }++ for _ in 0..5 {+ let mut n_changed = 0;+ for ((xi, li), &w) in x.iter().zip(l.iter_mut()).zip(quant_weights.iter()) {+ let mut slx = sumlx - w * xi * *li as f32;+ let mut sl2 = suml2 - w * (*li as f32).powi(2);+ if slx > 0.0 && sl2 > 0.0 {+ let new_li = (nearest_int(xi * sl2 / slx) as u8).min(nmax);+ if new_li != *li {+ slx += w * xi * new_li as f32;+ sl2 += w * (new_li as f32).powi(2);+ if slx.powi(2) * suml2 > sumlx.powi(2) * sl2 {+ *li = new_li;+ sumlx = slx;+ suml2 = sl2;+ n_changed += 1;+ }+ }+ }+ }+ if n_changed == 0 {+ break;+ }+ }++ sumlx / suml2+}diff --git a/vendor/candle-core/src/safetensors.rs b/vendor/candle-core/src/safetensors.rsnew file mode 100644index 0000000..b289059--- /dev/null+++ b/vendor/candle-core/src/safetensors.rs@@ -0,0 +1,652 @@+//! Module to load `safetensor` files into CPU/GPU memory.+//!+//! There are multiple ways to load tensors from safetensor files:+//! - `load` function for loading directly into memory and returning a HashMap of tensors+//! - `MmapedSafetensors` for memory mapping files and avoiding full allocation+//! - `SliceSafetensors` for working with in-memory buffers+//! - `BufferedSafetensors` for owning a buffer of data+//!+//! Tensors can also be serialized to safetensor format using the `save` function or+//! `Tensor::save_safetensors` method.+//!+use crate::op::BackpropOp;+use crate::storage::Storage;+use crate::tensor::from_storage;+use crate::{DType, Device, Error, Result, Tensor, WithDType};+use safetensors::tensor as st;+use safetensors::tensor::SafeTensors;+use std::borrow::Cow;+use std::collections::HashMap;+use std::path::Path;++impl From<DType> for st::Dtype {+ fn from(value: DType) -> Self {+ match value {+ DType::U8 => st::Dtype::U8,+ DType::U32 => st::Dtype::U32,+ DType::I16 => st::Dtype::I16,+ DType::I32 => st::Dtype::I32,+ DType::I64 => st::Dtype::I64,+ DType::BF16 => st::Dtype::BF16,+ DType::F16 => st::Dtype::F16,+ DType::F32 => st::Dtype::F32,+ DType::F64 => st::Dtype::F64,+ DType::F8E4M3 => st::Dtype::F8_E4M3,+ DType::F6E2M3 => st::Dtype::F6_E2M3,+ DType::F6E3M2 => st::Dtype::F6_E3M2,+ DType::F4 => st::Dtype::F4,+ DType::F8E8M0 => st::Dtype::F8_E8M0,+ }+ }+}++impl TryFrom<st::Dtype> for DType {+ type Error = Error;+ fn try_from(value: st::Dtype) -> Result<Self> {+ match value {+ st::Dtype::U8 => Ok(DType::U8),+ st::Dtype::U32 => Ok(DType::U32),+ st::Dtype::I16 => Ok(DType::I16),+ st::Dtype::I32 => Ok(DType::I32),+ st::Dtype::I64 => Ok(DType::I64),+ st::Dtype::BF16 => Ok(DType::BF16),+ st::Dtype::F16 => Ok(DType::F16),+ st::Dtype::F32 => Ok(DType::F32),+ st::Dtype::F64 => Ok(DType::F64),+ st::Dtype::F8_E4M3 => Ok(DType::F8E4M3),+ st::Dtype::F6_E2M3 => Ok(DType::F6E2M3),+ st::Dtype::F6_E3M2 => Ok(DType::F6E3M2),+ st::Dtype::F4 => Ok(DType::F4),+ st::Dtype::F8_E8M0 => Ok(DType::F8E8M0),+ dtype => Err(Error::UnsupportedSafeTensorDtype(dtype)),+ }+ }+}++impl st::View for Tensor {+ fn dtype(&self) -> st::Dtype {+ self.dtype().into()+ }+ fn shape(&self) -> &[usize] {+ self.shape().dims()+ }++ fn data(&self) -> Cow<'_, [u8]> {+ // This copies data from GPU to CPU.+ // TODO: Avoid the unwrap here.+ Cow::Owned(convert_back(self).unwrap())+ }++ fn data_len(&self) -> usize {+ let n: usize = self.shape().elem_count();+ let bytes_per_element = self.dtype().size_in_bytes();+ n * bytes_per_element+ }+}++impl st::View for &Tensor {+ fn dtype(&self) -> st::Dtype {+ (*self).dtype().into()+ }+ fn shape(&self) -> &[usize] {+ self.dims()+ }++ fn data(&self) -> Cow<'_, [u8]> {+ // This copies data from GPU to CPU.+ // TODO: Avoid the unwrap here.+ Cow::Owned(convert_back(self).unwrap())+ }++ fn data_len(&self) -> usize {+ let n: usize = self.dims().iter().product();+ let bytes_per_element = (*self).dtype().size_in_bytes();+ n * bytes_per_element+ }+}++impl Tensor {+ pub fn save_safetensors<P: AsRef<Path>>(&self, name: &str, filename: P) -> Result<()> {+ let data = [(name, self.clone())];+ Ok(st::serialize_to_file(data, None, filename.as_ref())?)+ }+}++fn convert_slice<T: WithDType>(data: &[u8], shape: &[usize], device: &Device) -> Result<Tensor> {+ let size_in_bytes = T::DTYPE.size_in_bytes();+ let elem_count = data.len() / size_in_bytes;+ if (data.as_ptr() as usize).is_multiple_of(size_in_bytes) {+ // SAFETY This is safe because we just checked that this+ // was correctly aligned.+ let data: &[T] =+ unsafe { std::slice::from_raw_parts(data.as_ptr() as *const T, elem_count) };+ Tensor::from_slice(data, shape, device)+ } else {+ // XXX: We need to specify `T` here, otherwise the compiler will infer u8 because of the following cast+ // Making this vector too small to fit a full f16/f32/f64 weights, resulting in out-of-bounds access+ let mut c: Vec<T> = Vec::with_capacity(elem_count);+ // SAFETY: We just created c, so the allocated memory is necessarily+ // contiguous and non overlapping with the view's data.+ // We're downgrading the `c` pointer from T to u8, which removes alignment+ // constraints.+ unsafe {+ std::ptr::copy_nonoverlapping(data.as_ptr(), c.as_mut_ptr() as *mut u8, data.len());+ c.set_len(elem_count)+ }+ Tensor::from_slice(&c, shape, device)+ }+}++fn convert_slice_with_cast<T: Sized + Copy, U: WithDType, F: Fn(T) -> Result<U>>(+ data: &[u8],+ shape: &[usize],+ device: &Device,+ conv: F,+) -> Result<Tensor> {+ let size_in_bytes = std::mem::size_of::<T>();+ let elem_count = data.len() / size_in_bytes;+ if (data.as_ptr() as usize).is_multiple_of(size_in_bytes) {+ // SAFETY This is safe because we just checked that this+ // was correctly aligned.+ let data: &[T] =+ unsafe { std::slice::from_raw_parts(data.as_ptr() as *const T, elem_count) };+ let data = data.iter().map(|t| conv(*t)).collect::<Result<Vec<_>>>()?;+ Tensor::from_vec(data, shape, device)+ } else {+ // XXX: We need to specify `T` here, otherwise the compiler will infer u8 because of the following cast+ // Making this vector too small to fit a full f16/f32/f64 weights, resulting in out-of-bounds access+ let mut c: Vec<T> = Vec::with_capacity(elem_count);+ // SAFETY: We just created c, so the allocated memory is necessarily+ // contiguous and non overlapping with the view's data.+ // We're downgrading the `c` pointer from T to u8, which removes alignment+ // constraints.+ unsafe {+ std::ptr::copy_nonoverlapping(data.as_ptr(), c.as_mut_ptr() as *mut u8, data.len());+ c.set_len(elem_count)+ }+ let c = c.into_iter().map(conv).collect::<Result<Vec<_>>>()?;+ Tensor::from_vec(c, shape, device)+ }+}++fn convert_with_cast_<T: Sized + Copy, U: WithDType, F: Fn(T) -> Result<U>>(+ view: &st::TensorView<'_>,+ device: &Device,+ conv: F,+) -> Result<Tensor> {+ convert_slice_with_cast::<T, U, F>(view.data(), view.shape(), device, conv)+}++fn convert_<T: WithDType>(view: &st::TensorView<'_>, device: &Device) -> Result<Tensor> {+ convert_slice::<T>(view.data(), view.shape(), device)+}++fn convert_back_<T: WithDType>(mut vs: Vec<T>) -> Vec<u8> {+ let size_in_bytes = T::DTYPE.size_in_bytes();+ let length = vs.len() * size_in_bytes;+ let capacity = vs.capacity() * size_in_bytes;+ let ptr = vs.as_mut_ptr() as *mut u8;+ // Don't run the destructor for Vec<T>+ std::mem::forget(vs);+ // SAFETY:+ //+ // Every T is larger than u8, so there is no issue regarding alignment.+ // This re-interpret the Vec<T> as a Vec<u8>.+ unsafe { Vec::from_raw_parts(ptr, length, capacity) }+}++pub trait Load {+ fn load(&self, device: &Device) -> Result<Tensor>;+}++impl Load for st::TensorView<'_> {+ fn load(&self, device: &Device) -> Result<Tensor> {+ convert(self, device)+ }+}++impl Tensor {+ pub fn from_raw_buffer(+ data: &[u8],+ dtype: DType,+ shape: &[usize],+ device: &Device,+ ) -> Result<Self> {+ match dtype {+ DType::U8 => convert_slice::<u8>(data, shape, device),+ DType::U32 => convert_slice::<u32>(data, shape, device),+ DType::I16 => convert_slice::<i16>(data, shape, device),+ DType::I32 => convert_slice::<i32>(data, shape, device),+ DType::I64 => convert_slice::<i64>(data, shape, device),+ DType::BF16 => convert_slice::<half::bf16>(data, shape, device),+ DType::F16 => convert_slice::<half::f16>(data, shape, device),+ DType::F32 => convert_slice::<f32>(data, shape, device),+ DType::F64 => convert_slice::<f64>(data, shape, device),+ DType::F8E4M3 => convert_slice::<float8::F8E4M3>(data, shape, device),+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ // For dummy types, create storage with raw bytes+ let storage = match device {+ Device::Cpu => {+ let cpu_storage = match dtype {+ DType::F6E2M3 => crate::cpu_backend::CpuStorage::F6E2M3(data.to_vec()),+ DType::F6E3M2 => crate::cpu_backend::CpuStorage::F6E3M2(data.to_vec()),+ DType::F4 => crate::cpu_backend::CpuStorage::F4(data.to_vec()),+ DType::F8E8M0 => crate::cpu_backend::CpuStorage::F8E8M0(data.to_vec()),+ _ => unreachable!(),+ };+ Storage::Cpu(cpu_storage)+ }+ #[cfg(feature = "cuda")]+ Device::Cuda(device) => {+ let mut slice = unsafe { device.alloc::<u8>(data.len())? };+ device.memcpy_htod(data, &mut slice)?;++ let slice = match dtype {+ DType::F6E2M3 => crate::cuda_backend::CudaStorageSlice::F6E2M3(slice),+ DType::F6E3M2 => crate::cuda_backend::CudaStorageSlice::F6E3M2(slice),+ DType::F4 => crate::cuda_backend::CudaStorageSlice::F4(slice),+ DType::F8E8M0 => crate::cuda_backend::CudaStorageSlice::F8E8M0(slice),+ _ => unreachable!(),+ };+ let storage = crate::cuda_backend::CudaStorage {+ slice,+ device: device.clone(),+ };+ Storage::Cuda(storage)+ }+ #[cfg(not(feature = "cuda"))]+ Device::Cuda(_) => {+ return Err(Error::Msg("CUDA support not compiled".to_string()));+ }+ #[cfg(feature = "metal")]+ Device::Metal(device) => {+ let buffer = device+ .new_buffer_builder()+ .with_data(data)+ .with_label("safetensors_view")+ .build()?;++ let storage = crate::metal_backend::MetalStorage::new(+ buffer,+ device.clone(),+ data.len(),+ dtype,+ );+ Storage::Metal(storage)+ }+ #[cfg(not(feature = "metal"))]+ Device::Metal(_) => {+ return Err(Error::Msg("Metal support not compiled".to_string()));+ }+ };++ let op = BackpropOp::none();+ Ok(from_storage(storage, shape, op, false))+ }+ }+ }+}++fn convert(view: &st::TensorView<'_>, device: &Device) -> Result<Tensor> {+ match view.dtype() {+ st::Dtype::U8 => convert_::<u8>(view, device),+ st::Dtype::U16 => {+ let conv = |x| Ok(u32::from(x));+ convert_with_cast_::<u16, u32, _>(view, device, conv)+ }+ st::Dtype::U32 => convert_::<u32>(view, device),+ st::Dtype::I16 => convert_::<i16>(view, device),+ st::Dtype::I32 => convert_::<i32>(view, device),+ st::Dtype::I64 => convert_::<i64>(view, device),+ st::Dtype::BF16 => convert_::<half::bf16>(view, device),+ st::Dtype::F16 => convert_::<half::f16>(view, device),+ st::Dtype::F32 => convert_::<f32>(view, device),+ st::Dtype::F64 => convert_::<f64>(view, device),+ st::Dtype::F8_E4M3 => convert_::<float8::F8E4M3>(view, device),+ st::Dtype::F6_E2M3 | st::Dtype::F6_E3M2 | st::Dtype::F4 | st::Dtype::F8_E8M0 => {+ // For dummy types, we need to handle loading by creating a dummy tensor+ // Since these types don't have actual data representation, we'll create+ // a tensor that indicates it's a dummy type+ convert_dummy(view, device)+ }+ dtype => Err(Error::UnsupportedSafeTensorDtype(dtype)),+ }+}++fn convert_dummy(view: &st::TensorView<'_>, device: &Device) -> Result<Tensor> {+ // For dummy types, we'll create the appropriate storage variant that preserves+ // both the raw data and the correct dtype+ let (dtype, _dtype_name) = match view.dtype() {+ st::Dtype::F6_E2M3 => (DType::F6E2M3, "F6_E2M3 (MX6)"),+ st::Dtype::F6_E3M2 => (DType::F6E3M2, "F6_E3M2 (MX6)"),+ st::Dtype::F4 => (DType::F4, "F4 (MX4)"),+ st::Dtype::F8_E8M0 => (DType::F8E8M0, "F8_E8M0"),+ _ => unreachable!("convert_dummy called with non-dummy dtype"),+ };++ // Load the raw bytes+ let data = view.data();+ let shape = view.shape();++ // Create storage with the appropriate dummy type variant+ let storage = match device {+ Device::Cpu => {+ let cpu_storage = match dtype {+ DType::F6E2M3 => crate::cpu_backend::CpuStorage::F6E2M3(data.to_vec()),+ DType::F6E3M2 => crate::cpu_backend::CpuStorage::F6E3M2(data.to_vec()),+ DType::F4 => crate::cpu_backend::CpuStorage::F4(data.to_vec()),+ DType::F8E8M0 => crate::cpu_backend::CpuStorage::F8E8M0(data.to_vec()),+ _ => unreachable!(),+ };+ Storage::Cpu(cpu_storage)+ }+ #[cfg(feature = "cuda")]+ Device::Cuda(device) => {+ let mut slice = unsafe { device.alloc::<u8>(data.len())? };+ device.memcpy_htod(data, &mut slice)?;++ let slice = match dtype {+ DType::F6E2M3 => crate::cuda_backend::CudaStorageSlice::F6E2M3(slice),+ DType::F6E3M2 => crate::cuda_backend::CudaStorageSlice::F6E3M2(slice),+ DType::F4 => crate::cuda_backend::CudaStorageSlice::F4(slice),+ DType::F8E8M0 => crate::cuda_backend::CudaStorageSlice::F8E8M0(slice),+ _ => unreachable!(),+ };+ let storage = crate::cuda_backend::CudaStorage {+ slice,+ device: device.clone(),+ };+ Storage::Cuda(storage)+ }+ #[cfg(not(feature = "cuda"))]+ Device::Cuda(_) => {+ return Err(Error::Msg("CUDA support not compiled".to_string()));+ }+ #[cfg(feature = "metal")]+ Device::Metal(device) => {+ let buffer = device+ .new_buffer_builder()+ .with_data(data)+ .with_label("safetensors_load")+ .build()?;++ let storage =+ crate::metal_backend::MetalStorage::new(buffer, device.clone(), data.len(), dtype);+ Storage::Metal(storage)+ }+ #[cfg(not(feature = "metal"))]+ Device::Metal(_) => {+ return Err(Error::Msg("Metal support not compiled".to_string()));+ }+ };++ // Create tensor with correct dtype+ let op = BackpropOp::none();+ Ok(from_storage(storage, shape, op, false))+}++fn convert_back(tensor: &Tensor) -> Result<Vec<u8>> {+ // TODO: This makes an unnecessary copy when the tensor is on the cpu.+ let tensor = tensor.flatten_all()?;+ match tensor.dtype() {+ DType::U8 => Ok(convert_back_::<u8>(tensor.to_vec1()?)),+ DType::U32 => Ok(convert_back_::<u32>(tensor.to_vec1()?)),+ DType::I16 => Ok(convert_back_::<i16>(tensor.to_vec1()?)),+ DType::I32 => Ok(convert_back_::<i32>(tensor.to_vec1()?)),+ DType::I64 => Ok(convert_back_::<i64>(tensor.to_vec1()?)),+ DType::F16 => Ok(convert_back_::<half::f16>(tensor.to_vec1()?)),+ DType::BF16 => Ok(convert_back_::<half::bf16>(tensor.to_vec1()?)),+ DType::F32 => Ok(convert_back_::<f32>(tensor.to_vec1()?)),+ DType::F64 => Ok(convert_back_::<f64>(tensor.to_vec1()?)),+ DType::F8E4M3 => Ok(convert_back_::<float8::F8E4M3>(tensor.to_vec1()?)),+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ Err(Error::Msg("Internal error: dtype mismatch in storage".to_string()).bt())+ }+ }+}++pub fn load<P: AsRef<Path>>(filename: P, device: &Device) -> Result<HashMap<String, Tensor>> {+ let data = std::fs::read(filename.as_ref())?;+ load_buffer(&data[..], device)+}++pub fn load_buffer(data: &[u8], device: &Device) -> Result<HashMap<String, Tensor>> {+ let st = safetensors::SafeTensors::deserialize(data)?;+ st.tensors()+ .into_iter()+ .map(|(name, view)| Ok((name, view.load(device)?)))+ .collect()+}++pub fn save<K: AsRef<str> + Ord + std::fmt::Display, P: AsRef<Path>>(+ tensors: &HashMap<K, Tensor>,+ filename: P,+) -> Result<()> {+ Ok(st::serialize_to_file(tensors, None, filename.as_ref())?)+}++#[derive(yoke::Yokeable)]+struct SafeTensors_<'a>(SafeTensors<'a>);++pub struct MmapedSafetensors {+ safetensors: Vec<yoke::Yoke<SafeTensors_<'static>, memmap2::Mmap>>,+ routing: Option<HashMap<String, usize>>,+}++impl MmapedSafetensors {+ /// Creates a wrapper around a memory mapped file and deserialize the safetensors header.+ ///+ /// # Safety+ ///+ /// The unsafe is inherited from [`memmap2::MmapOptions`].+ pub unsafe fn new<P: AsRef<Path>>(p: P) -> Result<Self> {+ let p = p.as_ref();+ let file = std::fs::File::open(p).map_err(|e| Error::from(e).with_path(p))?;+ let file = memmap2::MmapOptions::new()+ .map(&file)+ .map_err(|e| Error::from(e).with_path(p))?;+ let safetensors = yoke::Yoke::<SafeTensors_<'static>, memmap2::Mmap>::try_attach_to_cart(+ file,+ |data: &[u8]| {+ let st = safetensors::SafeTensors::deserialize(data)+ .map_err(|e| Error::from(e).with_path(p))?;+ Ok::<_, Error>(SafeTensors_(st))+ },+ )?;+ Ok(Self {+ safetensors: vec![safetensors],+ routing: None,+ })+ }++ /// Creates a wrapper around multiple memory mapped file and deserialize the safetensors headers.+ ///+ /// If a tensor name appears in multiple files, the last entry is returned.+ ///+ /// # Safety+ ///+ /// The unsafe is inherited from [`memmap2::MmapOptions`].+ pub unsafe fn multi<P: AsRef<Path>>(paths: &[P]) -> Result<Self> {+ let mut routing = HashMap::new();+ let mut safetensors = vec![];+ for (index, p) in paths.iter().enumerate() {+ let p = p.as_ref();+ let file = std::fs::File::open(p).map_err(|e| Error::from(e).with_path(p))?;+ let file = memmap2::MmapOptions::new()+ .map(&file)+ .map_err(|e| Error::from(e).with_path(p))?;+ let data = yoke::Yoke::<SafeTensors_<'static>, memmap2::Mmap>::try_attach_to_cart(+ file,+ |data: &[u8]| {+ let st = safetensors::SafeTensors::deserialize(data)+ .map_err(|e| Error::from(e).with_path(p))?;+ Ok::<_, Error>(SafeTensors_(st))+ },+ )?;+ for k in data.get().0.names() {+ routing.insert(k.to_string(), index);+ }+ safetensors.push(data)+ }+ Ok(Self {+ safetensors,+ routing: Some(routing),+ })+ }++ pub fn load(&self, name: &str, dev: &Device) -> Result<Tensor> {+ self.get(name)?.load(dev)+ }++ pub fn tensors(&self) -> Vec<(String, st::TensorView<'_>)> {+ let mut tensors = vec![];+ for safetensors in self.safetensors.iter() {+ tensors.push(safetensors.get().0.tensors())+ }+ tensors.into_iter().flatten().collect()+ }++ pub fn get(&self, name: &str) -> Result<st::TensorView<'_>> {+ let index = match &self.routing {+ None => 0,+ Some(routing) => {+ let index = routing.get(name).ok_or_else(|| {+ Error::CannotFindTensor {+ path: name.to_string(),+ }+ .bt()+ })?;+ *index+ }+ };+ Ok(self.safetensors[index].get().0.tensor(name)?)+ }+}++pub struct SliceSafetensors<'a> {+ safetensors: SafeTensors<'a>,+}++impl<'a> SliceSafetensors<'a> {+ /// Creates a wrapper around a binary buffer and deserialize the safetensors header.+ pub fn new(buffer: &'a [u8]) -> Result<Self> {+ let safetensors = safetensors::SafeTensors::deserialize(buffer)?;+ Ok(Self { safetensors })+ }++ pub fn load(&self, name: &str, dev: &Device) -> Result<Tensor> {+ self.safetensors.tensor(name)?.load(dev)+ }++ pub fn tensors(&self) -> Vec<(String, st::TensorView<'_>)> {+ self.safetensors.tensors()+ }++ pub fn get(&self, name: &str) -> Result<st::TensorView<'_>> {+ Ok(self.safetensors.tensor(name)?)+ }+}++pub struct BufferedSafetensors {+ safetensors: yoke::Yoke<SafeTensors_<'static>, Vec<u8>>,+}++impl BufferedSafetensors {+ /// Creates a wrapper around a binary buffer and deserialize the safetensors header.+ pub fn new(buffer: Vec<u8>) -> Result<Self> {+ let safetensors = yoke::Yoke::<SafeTensors_<'static>, Vec<u8>>::try_attach_to_cart(+ buffer,+ |data: &[u8]| {+ let st = safetensors::SafeTensors::deserialize(data)?;+ Ok::<_, Error>(SafeTensors_(st))+ },+ )?;+ Ok(Self { safetensors })+ }++ pub fn load(&self, name: &str, dev: &Device) -> Result<Tensor> {+ self.get(name)?.load(dev)+ }++ pub fn tensors(&self) -> Vec<(String, st::TensorView<'_>)> {+ self.safetensors.get().0.tensors()+ }++ pub fn get(&self, name: &str) -> Result<st::TensorView<'_>> {+ Ok(self.safetensors.get().0.tensor(name)?)+ }+}++pub struct MmapedFile {+ path: std::path::PathBuf,+ inner: memmap2::Mmap,+}++impl MmapedFile {+ /// Creates a wrapper around a memory mapped file from which you can retrieve+ /// tensors using [`MmapedFile::deserialize`]+ ///+ /// # Safety+ ///+ /// The unsafe is inherited from [`memmap2::MmapOptions`].+ pub unsafe fn new<P: AsRef<Path>>(p: P) -> Result<Self> {+ let p = p.as_ref();+ let file = std::fs::File::open(p).map_err(|e| Error::from(e).with_path(p))?;+ let inner = memmap2::MmapOptions::new()+ .map(&file)+ .map_err(|e| Error::from(e).with_path(p))?;+ Ok(Self {+ inner,+ path: p.to_path_buf(),+ })+ }++ pub fn deserialize(&self) -> Result<SafeTensors<'_>> {+ let st = safetensors::SafeTensors::deserialize(&self.inner)+ .map_err(|e| Error::from(e).with_path(&self.path))?;+ Ok(st)+ }+}++#[cfg(test)]+mod tests {+ use super::*;+ use std::collections::HashMap;++ #[test]+ fn save_single_tensor() {+ let t = Tensor::zeros((2, 2), DType::F32, &Device::Cpu).unwrap();+ t.save_safetensors("t", "t.safetensors").unwrap();+ let bytes = std::fs::read("t.safetensors").unwrap();+ assert_eq!(bytes, b"@\0\0\0\0\0\0\0{\"t\":{\"dtype\":\"F32\",\"shape\":[2,2],\"data_offsets\":[0,16]}} \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0");+ std::fs::remove_file("t.safetensors").unwrap();+ }++ #[test]+ fn save_load_multiple_tensors() {+ let t = Tensor::zeros((2, 2), DType::F32, &Device::Cpu).unwrap();+ let u = Tensor::zeros((1, 2), DType::F32, &Device::Cpu).unwrap();+ let map: HashMap<_, _> = [("t", t), ("u", u)].into_iter().collect();+ save(&map, "multi.safetensors").unwrap();++ let weights = load("multi.safetensors", &Device::Cpu).unwrap();+ assert_eq!(weights.get("t").unwrap().dims(), &[2, 2]);+ assert_eq!(weights.get("u").unwrap().dims(), &[1, 2]);+ let bytes = std::fs::read("multi.safetensors").unwrap();+ assert_eq!(bytes, b"x\0\0\0\0\0\0\0{\"t\":{\"dtype\":\"F32\",\"shape\":[2,2],\"data_offsets\":[0,16]},\"u\":{\"dtype\":\"F32\",\"shape\":[1,2],\"data_offsets\":[16,24]}} \0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0");+ std::fs::remove_file("multi.safetensors").unwrap();+ }++ #[test]+ fn load_u8() {+ let bytes = b"8\0\0\0\0\0\0\0{\"x\":{\"dtype\":\"U8\",\"shape\":[2],\"data_offsets\":[0,2]}} \x01\x03";+ std::fs::write("test_u8.safetensors", bytes).unwrap();+ let weights = load("test_u8.safetensors", &Device::Cpu).unwrap();+ let tensor = weights.get("x").unwrap();+ assert_eq!(tensor.dims(), &[2]);+ assert_eq!(tensor.dtype(), DType::U8);+ let data: Vec<u8> = tensor.to_vec1().unwrap();+ assert_eq!(data, vec![1, 3]);+ std::fs::remove_file("test_u8.safetensors").unwrap();+ }+}diff --git a/vendor/candle-core/src/scalar.rs b/vendor/candle-core/src/scalar.rsnew file mode 100644index 0000000..5c512c0--- /dev/null+++ b/vendor/candle-core/src/scalar.rs@@ -0,0 +1,115 @@+//! TensorScalar Enum and Trait+//!+use crate::{DType, Result, Tensor, WithDType};+use float8::F8E4M3 as f8e4m3;+use half::{bf16, f16};++#[derive(Debug, Clone, Copy, PartialEq)]+pub enum Scalar {+ U8(u8),+ U32(u32),+ I16(i16),+ I32(i32),+ I64(i64),+ BF16(bf16),+ F16(f16),+ F32(f32),+ F64(f64),+ F8E4M3(f8e4m3),+}++impl<T: WithDType> From<T> for Scalar {+ fn from(value: T) -> Self {+ value.to_scalar()+ }+}++impl Scalar {+ pub fn zero(dtype: DType) -> Self {+ match dtype {+ DType::U8 => Scalar::U8(0),+ DType::U32 => Scalar::U32(0),+ DType::I16 => Scalar::I16(0),+ DType::I32 => Scalar::I32(0),+ DType::I64 => Scalar::I64(0),+ DType::BF16 => Scalar::BF16(bf16::ZERO),+ DType::F16 => Scalar::F16(f16::ZERO),+ DType::F32 => Scalar::F32(0.0),+ DType::F64 => Scalar::F64(0.0),+ DType::F8E4M3 => Scalar::F8E4M3(f8e4m3::ZERO),+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ panic!("Cannot create zero scalar for dummy type {dtype:?}")+ }+ }+ }++ pub fn one(dtype: DType) -> Self {+ match dtype {+ DType::U8 => Scalar::U8(1),+ DType::U32 => Scalar::U32(1),+ DType::I16 => Scalar::I16(1),+ DType::I32 => Scalar::I32(1),+ DType::I64 => Scalar::I64(1),+ DType::BF16 => Scalar::BF16(bf16::ONE),+ DType::F16 => Scalar::F16(f16::ONE),+ DType::F32 => Scalar::F32(1.0),+ DType::F64 => Scalar::F64(1.0),+ DType::F8E4M3 => Scalar::F8E4M3(f8e4m3::ONE),+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ panic!("Cannot create one scalar for dummy type {dtype:?}")+ }+ }+ }++ pub fn dtype(&self) -> DType {+ match self {+ Scalar::U8(_) => DType::U8,+ Scalar::U32(_) => DType::U32,+ Scalar::I16(_) => DType::I16,+ Scalar::I32(_) => DType::I32,+ Scalar::I64(_) => DType::I64,+ Scalar::BF16(_) => DType::BF16,+ Scalar::F16(_) => DType::F16,+ Scalar::F32(_) => DType::F32,+ Scalar::F64(_) => DType::F64,+ Scalar::F8E4M3(_) => DType::F8E4M3,+ }+ }++ pub fn to_f64(&self) -> f64 {+ match self {+ Scalar::U8(v) => *v as f64,+ Scalar::U32(v) => *v as f64,+ Scalar::I16(v) => *v as f64,+ Scalar::I32(v) => *v as f64,+ Scalar::I64(v) => *v as f64,+ Scalar::BF16(v) => v.to_f64(),+ Scalar::F16(v) => v.to_f64(),+ Scalar::F32(v) => *v as f64,+ Scalar::F64(v) => *v,+ Scalar::F8E4M3(v) => v.to_f64(),+ }+ }+}++pub enum TensorScalar {+ Tensor(Tensor),+ Scalar(Tensor),+}++pub trait TensorOrScalar {+ fn to_tensor_scalar(self) -> Result<TensorScalar>;+}++impl TensorOrScalar for &Tensor {+ fn to_tensor_scalar(self) -> Result<TensorScalar> {+ Ok(TensorScalar::Tensor(self.clone()))+ }+}++impl<T: WithDType> TensorOrScalar for T {+ fn to_tensor_scalar(self) -> Result<TensorScalar> {+ let scalar = Tensor::new(self, &crate::Device::Cpu)?;+ Ok(TensorScalar::Scalar(scalar))+ }+}diff --git a/vendor/candle-core/src/shape.rs b/vendor/candle-core/src/shape.rsnew file mode 100644index 0000000..114c948--- /dev/null+++ b/vendor/candle-core/src/shape.rs@@ -0,0 +1,642 @@+//! The shape of a tensor is a tuple with the size of each of its dimensions.+#![allow(clippy::redundant_closure_call)]+use crate::{Error, Result};++#[derive(Clone, PartialEq, Eq)]+pub struct Shape(Vec<usize>);++pub const SCALAR: Shape = Shape(vec![]);++impl std::fmt::Debug for Shape {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ write!(f, "{:?}", self.dims())+ }+}++impl<const C: usize> From<&[usize; C]> for Shape {+ fn from(dims: &[usize; C]) -> Self {+ Self(dims.to_vec())+ }+}++impl From<&[usize]> for Shape {+ fn from(dims: &[usize]) -> Self {+ Self(dims.to_vec())+ }+}++impl From<&Shape> for Shape {+ fn from(shape: &Shape) -> Self {+ Self(shape.0.to_vec())+ }+}++impl From<()> for Shape {+ fn from(_: ()) -> Self {+ Self(vec![])+ }+}++impl From<usize> for Shape {+ fn from(d1: usize) -> Self {+ Self(vec![d1])+ }+}++macro_rules! impl_from_tuple {+ ($tuple:ty, $($index:tt),+) => {+ impl From<$tuple> for Shape {+ fn from(d: $tuple) -> Self {+ Self(vec![$(d.$index,)+])+ }+ }+ }+}++impl_from_tuple!((usize,), 0);+impl_from_tuple!((usize, usize), 0, 1);+impl_from_tuple!((usize, usize, usize), 0, 1, 2);+impl_from_tuple!((usize, usize, usize, usize), 0, 1, 2, 3);+impl_from_tuple!((usize, usize, usize, usize, usize), 0, 1, 2, 3, 4);+impl_from_tuple!((usize, usize, usize, usize, usize, usize), 0, 1, 2, 3, 4, 5);++impl From<Vec<usize>> for Shape {+ fn from(dims: Vec<usize>) -> Self {+ Self(dims)+ }+}++macro_rules! extract_dims {+ ($fn_name:ident, $cnt:tt, $dims:expr, $out_type:ty) => {+ pub fn $fn_name(dims: &[usize]) -> Result<$out_type> {+ if dims.len() != $cnt {+ Err(Error::UnexpectedNumberOfDims {+ expected: $cnt,+ got: dims.len(),+ shape: Shape::from(dims),+ }+ .bt())+ } else {+ Ok($dims(dims))+ }+ }++ impl Shape {+ pub fn $fn_name(&self) -> Result<$out_type> {+ $fn_name(self.0.as_slice())+ }+ }++ impl crate::Tensor {+ pub fn $fn_name(&self) -> Result<$out_type> {+ self.shape().$fn_name()+ }+ }++ impl std::convert::TryInto<$out_type> for Shape {+ type Error = crate::Error;+ fn try_into(self) -> std::result::Result<$out_type, Self::Error> {+ self.$fn_name()+ }+ }+ };+}++impl Shape {+ pub fn from_dims(dims: &[usize]) -> Self {+ Self(dims.to_vec())+ }++ /// The rank is the number of dimensions, 0 for a scalar value, 1 for a vector, etc.+ pub fn rank(&self) -> usize {+ self.0.len()+ }++ pub fn into_dims(self) -> Vec<usize> {+ self.0+ }++ /// The dimensions as a slice of `usize`.+ pub fn dims(&self) -> &[usize] {+ &self.0+ }++ /// The dimension size for a specified dimension index.+ pub fn dim<D: Dim>(&self, dim: D) -> Result<usize> {+ let dim = dim.to_index(self, "dim")?;+ Ok(self.dims()[dim])+ }++ /// The total number of elements, this is the product of all dimension sizes.+ pub fn elem_count(&self) -> usize {+ self.0.iter().product()+ }++ /// The strides given in number of elements for a contiguous n-dimensional+ /// arrays using this shape.+ pub(crate) fn stride_contiguous(&self) -> Vec<usize> {+ let mut stride: Vec<_> = self+ .0+ .iter()+ .rev()+ .scan(1, |prod, u| {+ let prod_pre_mult = *prod;+ *prod *= u;+ Some(prod_pre_mult)+ })+ .collect();+ stride.reverse();+ stride+ }++ /// Returns true if the strides are C contiguous (aka row major).+ pub fn is_contiguous(&self, stride: &[usize]) -> bool {+ if self.0.len() != stride.len() {+ return false;+ }+ let mut acc = 1;+ for (&stride, &dim) in stride.iter().zip(self.0.iter()).rev() {+ if dim > 1 && stride != acc {+ return false;+ }+ acc *= dim;+ }+ true+ }++ /// Returns true if the strides are Fortran contiguous (aka column major).+ pub fn is_fortran_contiguous(&self, stride: &[usize]) -> bool {+ if self.0.len() != stride.len() {+ return false;+ }+ let mut acc = 1;+ for (&stride, &dim) in stride.iter().zip(self.0.iter()) {+ if dim > 1 && stride != acc {+ return false;+ }+ acc *= dim;+ }+ true+ }++ /// Modifies the shape by adding a list of additional dimensions at the end of the existing+ /// dimensions.+ pub fn extend(mut self, additional_dims: &[usize]) -> Self {+ self.0.extend(additional_dims);+ self+ }++ /// Check whether the two shapes are compatible for broadcast, and if it is the case return the+ /// broadcasted shape. This is to be used for binary pointwise ops.+ pub fn broadcast_shape_binary_op(&self, rhs: &Self, op: &'static str) -> Result<Shape> {+ let lhs = self;+ let lhs_dims = lhs.dims();+ let rhs_dims = rhs.dims();+ let lhs_ndims = lhs_dims.len();+ let rhs_ndims = rhs_dims.len();+ let bcast_ndims = usize::max(lhs_ndims, rhs_ndims);+ let mut bcast_dims = vec![0; bcast_ndims];+ for (idx, bcast_value) in bcast_dims.iter_mut().enumerate() {+ let rev_idx = bcast_ndims - idx;+ let l_value = if lhs_ndims < rev_idx {+ 1+ } else {+ lhs_dims[lhs_ndims - rev_idx]+ };+ let r_value = if rhs_ndims < rev_idx {+ 1+ } else {+ rhs_dims[rhs_ndims - rev_idx]+ };+ *bcast_value = if l_value == r_value {+ l_value+ } else if l_value == 1 {+ r_value+ } else if r_value == 1 {+ l_value+ } else {+ Err(Error::ShapeMismatchBinaryOp {+ lhs: lhs.clone(),+ rhs: rhs.clone(),+ op,+ }+ .bt())?+ }+ }+ Ok(Shape::from(bcast_dims))+ }++ pub(crate) fn broadcast_shape_matmul(&self, rhs: &Self) -> Result<(Shape, Shape)> {+ let lhs = self;+ let lhs_dims = lhs.dims();+ let rhs_dims = rhs.dims();+ if lhs_dims.len() < 2 || rhs_dims.len() < 2 {+ crate::bail!("only 2d matrixes are supported {lhs:?} {rhs:?}")+ }+ let (m, lhs_k) = (lhs_dims[lhs_dims.len() - 2], lhs_dims[lhs_dims.len() - 1]);+ let (rhs_k, n) = (rhs_dims[rhs_dims.len() - 2], rhs_dims[rhs_dims.len() - 1]);+ if lhs_k != rhs_k {+ crate::bail!("different inner dimensions in broadcast matmul {lhs:?} {rhs:?}")+ }++ let lhs_b = Self::from(&lhs_dims[..lhs_dims.len() - 2]);+ let rhs_b = Self::from(&rhs_dims[..rhs_dims.len() - 2]);+ let bcast = lhs_b.broadcast_shape_binary_op(&rhs_b, "broadcast_matmul")?;+ let bcast_dims = bcast.dims();++ let bcast_lhs = [bcast_dims, &[m, lhs_k]].concat();+ let bcast_rhs = [bcast_dims, &[rhs_k, n]].concat();+ Ok((Shape::from(bcast_lhs), Shape::from(bcast_rhs)))+ }+}++pub trait Dim {+ fn to_index(&self, shape: &Shape, op: &'static str) -> Result<usize>;+ fn to_index_plus_one(&self, shape: &Shape, op: &'static str) -> Result<usize>;+}++impl Dim for usize {+ fn to_index(&self, shape: &Shape, op: &'static str) -> Result<usize> {+ let dim = *self;+ if dim >= shape.dims().len() {+ Err(Error::DimOutOfRange {+ shape: shape.clone(),+ dim: dim as i32,+ op,+ }+ .bt())?+ } else {+ Ok(dim)+ }+ }++ fn to_index_plus_one(&self, shape: &Shape, op: &'static str) -> Result<usize> {+ let dim = *self;+ if dim > shape.dims().len() {+ Err(Error::DimOutOfRange {+ shape: shape.clone(),+ dim: dim as i32,+ op,+ }+ .bt())?+ } else {+ Ok(dim)+ }+ }+}++#[derive(Debug, Copy, Clone, PartialEq, Eq, Hash)]+pub enum D {+ Minus1,+ Minus2,+ Minus(usize),+}++impl D {+ fn out_of_range(&self, shape: &Shape, op: &'static str) -> Error {+ let dim = match self {+ Self::Minus1 => -1,+ Self::Minus2 => -2,+ Self::Minus(u) => -(*u as i32),+ };+ Error::DimOutOfRange {+ shape: shape.clone(),+ dim,+ op,+ }+ .bt()+ }+}++impl Dim for D {+ fn to_index(&self, shape: &Shape, op: &'static str) -> Result<usize> {+ let rank = shape.rank();+ match self {+ Self::Minus1 if rank >= 1 => Ok(rank - 1),+ Self::Minus2 if rank >= 2 => Ok(rank - 2),+ Self::Minus(u) if *u > 0 && rank >= *u => Ok(rank - *u),+ _ => Err(self.out_of_range(shape, op)),+ }+ }++ fn to_index_plus_one(&self, shape: &Shape, op: &'static str) -> Result<usize> {+ let rank = shape.rank();+ match self {+ Self::Minus1 => Ok(rank),+ Self::Minus2 if rank >= 1 => Ok(rank - 1),+ Self::Minus(u) if *u > 0 && rank + 1 >= *u => Ok(rank + 1 - *u),+ _ => Err(self.out_of_range(shape, op)),+ }+ }+}++pub trait Dims: Sized {+ fn to_indexes_internal(self, shape: &Shape, op: &'static str) -> Result<Vec<usize>>;++ fn to_indexes(self, shape: &Shape, op: &'static str) -> Result<Vec<usize>> {+ let dims = self.to_indexes_internal(shape, op)?;+ for (i, &dim) in dims.iter().enumerate() {+ if dims[..i].contains(&dim) {+ Err(Error::DuplicateDimIndex {+ shape: shape.clone(),+ dims: dims.clone(),+ op,+ }+ .bt())?+ }+ if dim >= shape.rank() {+ Err(Error::DimOutOfRange {+ shape: shape.clone(),+ dim: dim as i32,+ op,+ }+ .bt())?+ }+ }+ Ok(dims)+ }+}++impl Dims for Vec<usize> {+ fn to_indexes_internal(self, _: &Shape, _: &'static str) -> Result<Vec<usize>> {+ Ok(self)+ }+}++impl<const N: usize> Dims for [usize; N] {+ fn to_indexes_internal(self, _: &Shape, _: &'static str) -> Result<Vec<usize>> {+ Ok(self.to_vec())+ }+}++impl Dims for &[usize] {+ fn to_indexes_internal(self, _: &Shape, _: &'static str) -> Result<Vec<usize>> {+ Ok(self.to_vec())+ }+}++impl Dims for () {+ fn to_indexes_internal(self, _: &Shape, _: &'static str) -> Result<Vec<usize>> {+ Ok(vec![])+ }+}++impl<D: Dim + Sized> Dims for D {+ fn to_indexes_internal(self, shape: &Shape, op: &'static str) -> Result<Vec<usize>> {+ let dim = self.to_index(shape, op)?;+ Ok(vec![dim])+ }+}++impl<D: Dim> Dims for (D,) {+ fn to_indexes_internal(self, shape: &Shape, op: &'static str) -> Result<Vec<usize>> {+ let dim = self.0.to_index(shape, op)?;+ Ok(vec![dim])+ }+}++impl<D1: Dim, D2: Dim> Dims for (D1, D2) {+ fn to_indexes_internal(self, shape: &Shape, op: &'static str) -> Result<Vec<usize>> {+ let d0 = self.0.to_index(shape, op)?;+ let d1 = self.1.to_index(shape, op)?;+ Ok(vec![d0, d1])+ }+}++impl<D1: Dim, D2: Dim, D3: Dim> Dims for (D1, D2, D3) {+ fn to_indexes_internal(self, shape: &Shape, op: &'static str) -> Result<Vec<usize>> {+ let d0 = self.0.to_index(shape, op)?;+ let d1 = self.1.to_index(shape, op)?;+ let d2 = self.2.to_index(shape, op)?;+ Ok(vec![d0, d1, d2])+ }+}++impl<D1: Dim, D2: Dim, D3: Dim, D4: Dim> Dims for (D1, D2, D3, D4) {+ fn to_indexes_internal(self, shape: &Shape, op: &'static str) -> Result<Vec<usize>> {+ let d0 = self.0.to_index(shape, op)?;+ let d1 = self.1.to_index(shape, op)?;+ let d2 = self.2.to_index(shape, op)?;+ let d3 = self.3.to_index(shape, op)?;+ Ok(vec![d0, d1, d2, d3])+ }+}++impl<D1: Dim, D2: Dim, D3: Dim, D4: Dim, D5: Dim> Dims for (D1, D2, D3, D4, D5) {+ fn to_indexes_internal(self, shape: &Shape, op: &'static str) -> Result<Vec<usize>> {+ let d0 = self.0.to_index(shape, op)?;+ let d1 = self.1.to_index(shape, op)?;+ let d2 = self.2.to_index(shape, op)?;+ let d3 = self.3.to_index(shape, op)?;+ let d4 = self.4.to_index(shape, op)?;+ Ok(vec![d0, d1, d2, d3, d4])+ }+}++impl<D1: Dim, D2: Dim, D3: Dim, D4: Dim, D5: Dim, D6: Dim> Dims for (D1, D2, D3, D4, D5, D6) {+ fn to_indexes_internal(self, shape: &Shape, op: &'static str) -> Result<Vec<usize>> {+ let d0 = self.0.to_index(shape, op)?;+ let d1 = self.1.to_index(shape, op)?;+ let d2 = self.2.to_index(shape, op)?;+ let d3 = self.3.to_index(shape, op)?;+ let d4 = self.4.to_index(shape, op)?;+ let d5 = self.5.to_index(shape, op)?;+ Ok(vec![d0, d1, d2, d3, d4, d5])+ }+}++extract_dims!(dims0, 0, |_: &[usize]| (), ());+extract_dims!(dims1, 1, |d: &[usize]| d[0], usize);+extract_dims!(dims2, 2, |d: &[usize]| (d[0], d[1]), (usize, usize));+extract_dims!(+ dims3,+ 3,+ |d: &[usize]| (d[0], d[1], d[2]),+ (usize, usize, usize)+);+extract_dims!(+ dims4,+ 4,+ |d: &[usize]| (d[0], d[1], d[2], d[3]),+ (usize, usize, usize, usize)+);+extract_dims!(+ dims5,+ 5,+ |d: &[usize]| (d[0], d[1], d[2], d[3], d[4]),+ (usize, usize, usize, usize, usize)+);++pub trait ShapeWithOneHole {+ fn into_shape(self, el_count: usize) -> Result<Shape>;+}++impl<S: Into<Shape>> ShapeWithOneHole for S {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ let shape = self.into();+ if shape.elem_count() != el_count {+ return Err(Error::ShapeMismatch {+ buffer_size: el_count,+ shape,+ }+ .bt());+ }+ Ok(shape)+ }+}++impl ShapeWithOneHole for ((),) {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ Ok(el_count.into())+ }+}++fn hole_size(el_count: usize, prod_d: usize, s: &dyn std::fmt::Debug) -> Result<usize> {+ if prod_d == 0 {+ crate::bail!("cannot reshape tensor of {el_count} elements to {s:?}")+ }+ if !el_count.is_multiple_of(prod_d) {+ crate::bail!("cannot reshape tensor with {el_count} elements to {s:?}")+ }+ Ok(el_count / prod_d)+}++impl ShapeWithOneHole for ((), usize) {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ let ((), d1) = self;+ Ok((hole_size(el_count, d1, &self)?, d1).into())+ }+}++impl ShapeWithOneHole for (usize, ()) {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ let (d1, ()) = self;+ Ok((d1, hole_size(el_count, d1, &self)?).into())+ }+}++impl ShapeWithOneHole for ((), usize, usize) {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ let ((), d1, d2) = self;+ Ok((hole_size(el_count, d1 * d2, &self)?, d1, d2).into())+ }+}++impl ShapeWithOneHole for (usize, (), usize) {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ let (d1, (), d2) = self;+ Ok((d1, hole_size(el_count, d1 * d2, &self)?, d2).into())+ }+}++impl ShapeWithOneHole for (usize, usize, ()) {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ let (d1, d2, ()) = self;+ Ok((d1, d2, hole_size(el_count, d1 * d2, &self)?).into())+ }+}++impl ShapeWithOneHole for ((), usize, usize, usize) {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ let ((), d1, d2, d3) = self;+ let d = hole_size(el_count, d1 * d2 * d3, &self)?;+ Ok((d, d1, d2, d3).into())+ }+}++impl ShapeWithOneHole for (usize, (), usize, usize) {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ let (d1, (), d2, d3) = self;+ let d = hole_size(el_count, d1 * d2 * d3, &self)?;+ Ok((d1, d, d2, d3).into())+ }+}++impl ShapeWithOneHole for (usize, usize, (), usize) {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ let (d1, d2, (), d3) = self;+ let d = hole_size(el_count, d1 * d2 * d3, &self)?;+ Ok((d1, d2, d, d3).into())+ }+}++impl ShapeWithOneHole for (usize, usize, usize, ()) {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ let (d1, d2, d3, ()) = self;+ let d = hole_size(el_count, d1 * d2 * d3, &self)?;+ Ok((d1, d2, d3, d).into())+ }+}++impl ShapeWithOneHole for ((), usize, usize, usize, usize) {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ let ((), d1, d2, d3, d4) = self;+ let d = hole_size(el_count, d1 * d2 * d3 * d4, &self)?;+ Ok((d, d1, d2, d3, d4).into())+ }+}++impl ShapeWithOneHole for (usize, (), usize, usize, usize) {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ let (d1, (), d2, d3, d4) = self;+ let d = hole_size(el_count, d1 * d2 * d3 * d4, &self)?;+ Ok((d1, d, d2, d3, d4).into())+ }+}++impl ShapeWithOneHole for (usize, usize, (), usize, usize) {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ let (d1, d2, (), d3, d4) = self;+ let d = hole_size(el_count, d1 * d2 * d3 * d4, &self)?;+ Ok((d1, d2, d, d3, d4).into())+ }+}++impl ShapeWithOneHole for (usize, usize, usize, (), usize) {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ let (d1, d2, d3, (), d4) = self;+ let d = hole_size(el_count, d1 * d2 * d3 * d4, &self)?;+ Ok((d1, d2, d3, d, d4).into())+ }+}++impl ShapeWithOneHole for (usize, usize, usize, usize, ()) {+ fn into_shape(self, el_count: usize) -> Result<Shape> {+ let (d1, d2, d3, d4, ()) = self;+ let d = hole_size(el_count, d1 * d2 * d3 * d4, &self)?;+ Ok((d1, d2, d3, d4, d).into())+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ #[test]+ fn stride() {+ let shape = Shape::from(());+ assert_eq!(shape.stride_contiguous(), Vec::<usize>::new());+ let shape = Shape::from(42);+ assert_eq!(shape.stride_contiguous(), [1]);+ let shape = Shape::from((42, 1337));+ assert_eq!(shape.stride_contiguous(), [1337, 1]);+ let shape = Shape::from((299, 792, 458));+ assert_eq!(shape.stride_contiguous(), [458 * 792, 458, 1]);+ }++ #[test]+ fn test_from_tuple() {+ let shape = Shape::from((2,));+ assert_eq!(shape.dims(), &[2]);+ let shape = Shape::from((2, 3));+ assert_eq!(shape.dims(), &[2, 3]);+ let shape = Shape::from((2, 3, 4));+ assert_eq!(shape.dims(), &[2, 3, 4]);+ let shape = Shape::from((2, 3, 4, 5));+ assert_eq!(shape.dims(), &[2, 3, 4, 5]);+ let shape = Shape::from((2, 3, 4, 5, 6));+ assert_eq!(shape.dims(), &[2, 3, 4, 5, 6]);+ let shape = Shape::from((2, 3, 4, 5, 6, 7));+ assert_eq!(shape.dims(), &[2, 3, 4, 5, 6, 7]);+ }+}diff --git a/vendor/candle-core/src/sort.rs b/vendor/candle-core/src/sort.rsnew file mode 100644index 0000000..c49da7f--- /dev/null+++ b/vendor/candle-core/src/sort.rs@@ -0,0 +1,298 @@+use crate::{Result, Tensor};+use rayon::prelude::*;++#[derive(Debug, Clone, Copy)]+struct ArgSort {+ asc: bool,+ last_dim: usize,+}++impl ArgSort {+ fn asort<T: crate::WithDType>(&self, vs: &[T], layout: &crate::Layout) -> Result<Vec<u32>> {+ let vs = match layout.contiguous_offsets() {+ None => crate::bail!("input has to be contiguous"),+ Some((o1, o2)) => &vs[o1..o2],+ };+ #[allow(clippy::uninit_vec)]+ // Safety: indexes are set later in the parallelized section.+ let mut sort_indexes = unsafe {+ let el_count = layout.shape().elem_count();+ let mut v = Vec::with_capacity(el_count);+ v.set_len(el_count);+ v+ };+ if self.asc {+ sort_indexes+ .par_chunks_exact_mut(self.last_dim)+ .zip(vs.par_chunks_exact(self.last_dim))+ .for_each(|(indexes, vs)| {+ indexes+ .iter_mut()+ .enumerate()+ .for_each(|(i, v)| *v = i as u32);+ indexes.sort_by(|&i, &j| {+ vs[i as usize]+ .partial_cmp(&vs[j as usize])+ .unwrap_or(std::cmp::Ordering::Greater)+ })+ });+ } else {+ sort_indexes+ .par_chunks_exact_mut(self.last_dim)+ .zip(vs.par_chunks_exact(self.last_dim))+ .for_each(|(indexes, vs)| {+ indexes+ .iter_mut()+ .enumerate()+ .for_each(|(i, v)| *v = i as u32);+ indexes.sort_by(|&j, &i| {+ vs[i as usize]+ .partial_cmp(&vs[j as usize])+ .unwrap_or(std::cmp::Ordering::Greater)+ })+ });+ }+ Ok(sort_indexes)+ }+}++#[cfg(feature = "cuda")]+mod cuda {+ use super::*;+ use crate::cuda_backend::cudarc::driver::{+ CudaSlice, DeviceRepr, LaunchConfig, ValidAsZeroBits,+ };+ use crate::cuda_backend::{kernel_name, kernels, CudaStorageSlice as S, WrapErr};+ use crate::{CudaDevice, WithDType};++ impl crate::cuda_backend::Map1Any for ArgSort {+ fn f<T: DeviceRepr + WithDType + ValidAsZeroBits, W: Fn(CudaSlice<T>) -> S>(+ &self,+ src: &CudaSlice<T>,+ dev: &CudaDevice,+ layout: &crate::Layout,+ _wrap: W,+ ) -> Result<S> {+ use cudarc::driver::PushKernelArg;++ let slice = match layout.contiguous_offsets() {+ None => crate::bail!("input has to be contiguous"),+ Some((o1, o2)) => src.slice(o1..o2),+ };+ let elem_count = layout.shape().elem_count();+ let dst = unsafe { dev.alloc::<u32>(elem_count)? };+ let func = if self.asc {+ dev.get_or_load_func(&kernel_name::<T>("asort_asc"), &kernels::SORT)?+ } else {+ dev.get_or_load_func(&kernel_name::<T>("asort_desc"), &kernels::SORT)?+ };+ let ncols = self.last_dim;+ let nrows = elem_count / ncols;+ let ncols_pad = next_power_of_2(ncols);+ // Limit block dim to 1024 threads, which is the maximum on modern CUDA gpus.+ let block_dim = ncols_pad.min(1024);+ let cfg = LaunchConfig {+ grid_dim: (nrows as u32, 1, 1),+ block_dim: (block_dim as u32, 1, 1),+ shared_mem_bytes: (ncols_pad * std::mem::size_of::<u32>()) as u32,+ };+ let stream = dev.cuda_stream();+ let mut builder = stream.launch_builder(&func);+ let ncols = ncols as i32;+ let ncols_pad = ncols_pad as i32;+ builder.arg(&slice).arg(&dst).arg(&ncols).arg(&ncols_pad);+ unsafe { builder.launch(cfg) }.w()?;+ Ok(S::U32(dst))+ }+ }+}++impl crate::CustomOp1 for ArgSort {+ fn name(&self) -> &'static str {+ "argsort"+ }++ fn cpu_fwd(+ &self,+ storage: &crate::CpuStorage,+ layout: &crate::Layout,+ ) -> Result<(crate::CpuStorage, crate::Shape)> {+ let sort_indexes = match storage {+ crate::CpuStorage::U8(vs) => self.asort(vs, layout)?,+ crate::CpuStorage::U32(vs) => self.asort(vs, layout)?,+ crate::CpuStorage::I16(vs) => self.asort(vs, layout)?,+ crate::CpuStorage::I32(vs) => self.asort(vs, layout)?,+ crate::CpuStorage::I64(vs) => self.asort(vs, layout)?,+ crate::CpuStorage::BF16(vs) => self.asort(vs, layout)?,+ crate::CpuStorage::F16(vs) => self.asort(vs, layout)?,+ crate::CpuStorage::F32(vs) => self.asort(vs, layout)?,+ crate::CpuStorage::F64(vs) => self.asort(vs, layout)?,+ crate::CpuStorage::F8E4M3(vs) => self.asort(vs, layout)?,+ // Dummy types don't support sorting+ crate::CpuStorage::F6E2M3(_) => {+ return Err(+ crate::Error::UnsupportedDTypeForOp(crate::DType::F6E2M3, "argsort").bt(),+ )+ }+ crate::CpuStorage::F6E3M2(_) => {+ return Err(+ crate::Error::UnsupportedDTypeForOp(crate::DType::F6E3M2, "argsort").bt(),+ )+ }+ crate::CpuStorage::F4(_) => {+ return Err(crate::Error::UnsupportedDTypeForOp(crate::DType::F4, "argsort").bt())+ }+ crate::CpuStorage::F8E8M0(_) => {+ return Err(+ crate::Error::UnsupportedDTypeForOp(crate::DType::F8E8M0, "argsort").bt(),+ )+ }+ };+ let sort_indexes = crate::CpuStorage::U32(sort_indexes);+ Ok((sort_indexes, layout.shape().into()))+ }++ #[cfg(feature = "cuda")]+ fn cuda_fwd(+ &self,+ storage: &crate::CudaStorage,+ layout: &crate::Layout,+ ) -> Result<(crate::CudaStorage, crate::Shape)> {+ use crate::backend::BackendStorage;+ use crate::cuda_backend::Map1Any;+ let dev = storage.device();+ let slice = self.map(&storage.slice, dev, layout)?;+ let dst = crate::cuda_backend::CudaStorage {+ slice,+ device: dev.clone(),+ };+ Ok((dst, layout.shape().clone()))+ }++ #[cfg(feature = "metal")]+ fn metal_fwd(+ &self,+ storage: &crate::MetalStorage,+ layout: &crate::Layout,+ ) -> Result<(crate::MetalStorage, crate::Shape)> {+ use crate::backend::BackendStorage;+ use crate::DType;++ let name = {+ if self.asc {+ match storage.dtype() {+ DType::BF16 => "asort_asc_bf16",+ DType::F16 => "asort_asc_f16",+ DType::F32 => "asort_asc_f32",+ DType::F64 => "asort_asc_f64",+ DType::U8 => "asort_asc_u8",+ DType::U32 => "asort_asc_u32",+ DType::I16 => "asort_asc_i16",+ DType::I32 => "asort_asc_i32",+ DType::I64 => "asort_asc_i64",+ DType::F8E4M3 => crate::bail!("Metal device does not yet support F8E4M3."),+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ return Err(+ crate::Error::UnsupportedDTypeForOp(storage.dtype(), "argsort").bt(),+ )+ }+ }+ } else {+ match storage.dtype() {+ DType::BF16 => "asort_desc_bf16",+ DType::F16 => "asort_desc_f16",+ DType::F32 => "asort_desc_f32",+ DType::F64 => "asort_desc_f64",+ DType::U8 => "asort_desc_u8",+ DType::U32 => "asort_desc_u32",+ DType::I16 => "asort_desc_i16",+ DType::I32 => "asort_desc_i32",+ DType::I64 => "asort_desc_i64",+ DType::F8E4M3 => crate::bail!("Metal device does not yet support F8E4M3."),+ DType::F6E2M3 | DType::F6E3M2 | DType::F4 | DType::F8E8M0 => {+ return Err(+ crate::Error::UnsupportedDTypeForOp(storage.dtype(), "argsort").bt(),+ )+ }+ }+ }+ };+ let device = storage.device();+ let kernels = device.kernels();+ let command_encoder = device.command_encoder()?;+ let el = layout.shape().elem_count();+ let ncols = self.last_dim;+ let nrows = el / ncols;+ let src = crate::metal_backend::buffer_o(storage.buffer(), layout, storage.dtype());+ let dst = device+ .new_buffer_builder()+ .with_size_for(el, DType::U32)+ .with_label("asort")+ .build()?;+ let mut ncols_pad = 1;+ while ncols_pad < ncols {+ ncols_pad *= 2;+ }+ candle_metal_kernels::call_arg_sort(+ device.metal_device(),+ &command_encoder,+ kernels,+ name,+ nrows,+ ncols,+ ncols_pad,+ src,+ &dst,+ )+ .map_err(crate::Error::wrap)?;+ let dst = crate::MetalStorage::new(dst, device.clone(), el, DType::U32);+ Ok((dst, layout.shape().clone()))+ }+}++#[allow(unused)]+fn next_power_of_2(x: usize) -> usize {+ let mut n = 1;+ while n < x {+ n *= 2+ }+ n+}++impl Tensor {+ /// Returns the indices that sort the tensor along the last dimension.+ ///+ /// If `asc` is `true`, sorting is in ascending order. Otherwise sorting is performed in+ /// descending order. The sort is unstable so there is no guarantees on the final order when it+ /// comes to ties.+ pub fn arg_sort_last_dim(&self, asc: bool) -> Result<Tensor> {+ if !self.is_contiguous() {+ return Err(crate::Error::RequiresContiguous {+ op: "arg_sort_last_dim",+ });+ }+ let last_dim = match self.dims().last() {+ None => crate::bail!("empty last-dim in arg-sort"),+ Some(last_dim) => *last_dim,+ };+ // No need for a backward pass for arg sort.+ self.apply_op1_no_bwd(&ArgSort { asc, last_dim })+ }++ /// Sorts the tensor along the last dimension, returns the sorted tensor together with the+ /// sorted indexes.+ ///+ /// If `asc` is `true`, sorting is in ascending order. Otherwise sorting is performed in+ /// descending order. The sort is unstable so there is no guarantees on the final order when it+ /// comes to ties.+ pub fn sort_last_dim(&self, asc: bool) -> Result<(Tensor, Tensor)> {+ if !self.is_contiguous() {+ return Err(crate::Error::RequiresContiguous {+ op: "sort_last_dim",+ });+ }+ let asort = self.arg_sort_last_dim(asc)?;+ let sorted = self.gather(&asort, crate::D::Minus1)?;+ Ok((sorted, asort))+ }+}diff --git a/vendor/candle-core/src/storage.rs b/vendor/candle-core/src/storage.rsnew file mode 100644index 0000000..9f57631--- /dev/null+++ b/vendor/candle-core/src/storage.rs@@ -0,0 +1,855 @@+use crate::backend::BackendStorage;+use crate::custom_op::{all_distinct, InplaceOpN, Src};+use crate::op::{self, CmpOp, ReduceOp};+use crate::scalar::Scalar;+use crate::{CpuStorage, CudaStorage, DType, Device, Error, Layout, MetalStorage, Result, Shape};+use crate::{CustomOp1, CustomOp2, CustomOp3};+use parking_lot::{RwLockReadGuard, RwLockWriteGuard};++// We do not want to implement Clone on Storage as cloning may fail because of+// out of memory. Instead try_clone should be used.+#[derive(Debug)]+pub enum Storage {+ Cpu(CpuStorage),+ Cuda(CudaStorage),+ Metal(MetalStorage),+}++pub type StorageRef<'a> = RwLockReadGuard<'a, Storage>;+pub type StorageMutRef<'a> = RwLockWriteGuard<'a, Storage>;++impl Storage {+ pub fn try_clone(&self, layout: &Layout) -> Result<Self> {+ match self {+ Self::Cpu(storage) => Ok(Self::Cpu(storage.clone())),+ Self::Cuda(storage) => {+ let storage = storage.try_clone(layout)?;+ Ok(Self::Cuda(storage))+ }+ Self::Metal(storage) => {+ let storage = storage.try_clone(layout)?;+ Ok(Self::Metal(storage))+ }+ }+ }++ pub fn device(&self) -> Device {+ match self {+ Self::Cpu(_) => Device::Cpu,+ Self::Cuda(storage) => Device::Cuda(storage.device().clone()),+ Self::Metal(storage) => Device::Metal(storage.device().clone()),+ }+ }++ pub fn dtype(&self) -> DType {+ match self {+ Self::Cpu(storage) => storage.dtype(),+ Self::Cuda(storage) => storage.dtype(),+ Self::Metal(storage) => storage.dtype(),+ }+ }++ pub(crate) fn same_device(&self, rhs: &Self, op: &'static str) -> Result<()> {+ let lhs_device = self.device();+ let rhs_device = rhs.device();+ let lhs = lhs_device.location();+ let rhs = rhs_device.location();+ let same_device = if self.device().is_metal() {+ // On metal, we require the device to be exactly the same rather than+ // having the same location. In cuda this is not necessary as all CudaDevice on the+ // same GPU will use the same cuda stream.+ lhs_device.same_device(&rhs_device)+ } else {+ lhs == rhs+ };+ if !same_device {+ Err(Error::DeviceMismatchBinaryOp { lhs, rhs, op }.bt())+ } else {+ Ok(())+ }+ }++ pub(crate) fn same_dtype(&self, rhs: &Self, op: &'static str) -> Result<()> {+ let lhs = self.dtype();+ let rhs = rhs.dtype();+ if lhs != rhs {+ Err(Error::DTypeMismatchBinaryOp { lhs, rhs, op }.bt())+ } else {+ Ok(())+ }+ }++ pub(crate) fn const_set(&mut self, v: Scalar, l: &Layout) -> Result<()> {+ match self {+ Storage::Cpu(storage) => storage.const_set(v, l),+ Storage::Cuda(storage) => storage.const_set(v, l),+ Storage::Metal(storage) => storage.const_set(v, l),+ }+ }++ pub(crate) fn affine(&self, layout: &Layout, mul: f64, add: f64) -> Result<Self> {+ match self {+ Storage::Cpu(storage) => {+ let storage = storage.affine(layout, mul, add)?;+ Ok(Self::Cpu(storage))+ }+ Self::Cuda(storage) => {+ let storage = storage.affine(layout, mul, add)?;+ Ok(Self::Cuda(storage))+ }+ Self::Metal(storage) => {+ let storage = storage.affine(layout, mul, add)?;+ Ok(Self::Metal(storage))+ }+ }+ }++ pub(crate) fn powf(&self, layout: &Layout, alpha: f64) -> Result<Self> {+ match self {+ Storage::Cpu(storage) => {+ let storage = storage.powf(layout, alpha)?;+ Ok(Self::Cpu(storage))+ }+ Self::Cuda(storage) => {+ let storage = storage.powf(layout, alpha)?;+ Ok(Self::Cuda(storage))+ }+ Self::Metal(storage) => {+ let storage = storage.powf(layout, alpha)?;+ Ok(Self::Metal(storage))+ }+ }+ }++ pub(crate) fn elu(&self, layout: &Layout, alpha: f64) -> Result<Self> {+ match self {+ Storage::Cpu(storage) => {+ let storage = storage.elu(layout, alpha)?;+ Ok(Self::Cpu(storage))+ }+ Self::Cuda(storage) => {+ let storage = storage.elu(layout, alpha)?;+ Ok(Self::Cuda(storage))+ }+ Self::Metal(storage) => {+ let storage = storage.elu(layout, alpha)?;+ Ok(Self::Metal(storage))+ }+ }+ }++ pub(crate) fn cmp(+ &self,+ op: CmpOp,+ rhs: &Self,+ lhs_layout: &Layout,+ rhs_layout: &Layout,+ ) -> Result<Self> {+ self.same_device(rhs, "cmp")?;+ self.same_dtype(rhs, "cmp")?;+ match (self, rhs) {+ (Storage::Cpu(lhs), Storage::Cpu(rhs)) => {+ let storage = lhs.cmp(op, rhs, lhs_layout, rhs_layout)?;+ Ok(Self::Cpu(storage))+ }+ (Self::Cuda(lhs), Self::Cuda(rhs)) => {+ let storage = lhs.cmp(op, rhs, lhs_layout, rhs_layout)?;+ Ok(Self::Cuda(storage))+ }+ (Self::Metal(lhs), Self::Metal(rhs)) => {+ let storage = lhs.cmp(op, rhs, lhs_layout, rhs_layout)?;+ Ok(Self::Metal(storage))+ }+ (lhs, rhs) => {+ // Should not happen because of the same device check above but we're defensive+ // anyway.+ Err(Error::DeviceMismatchBinaryOp {+ lhs: lhs.device().location(),+ rhs: rhs.device().location(),+ op: "cmp",+ }+ .bt())+ }+ }+ }++ pub(crate) fn reduce_op(&self, op: ReduceOp, layout: &Layout, s: &[usize]) -> Result<Self> {+ match self {+ Storage::Cpu(storage) => {+ let storage = storage.reduce_op(op, layout, s)?;+ Ok(Self::Cpu(storage))+ }+ Self::Cuda(storage) => {+ let storage = storage.reduce_op(op, layout, s)?;+ Ok(Self::Cuda(storage))+ }+ Self::Metal(storage) => {+ let storage = storage.reduce_op(op, layout, s)?;+ Ok(Self::Metal(storage))+ }+ }+ }++ pub(crate) fn to_dtype(&self, layout: &Layout, dtype: DType) -> Result<Self> {+ match self {+ Storage::Cpu(storage) => {+ let storage = storage.to_dtype(layout, dtype)?;+ Ok(Self::Cpu(storage))+ }+ Self::Cuda(storage) => {+ let storage = storage.to_dtype(layout, dtype)?;+ Ok(Self::Cuda(storage))+ }+ Self::Metal(storage) => {+ let storage = storage.to_dtype(layout, dtype)?;+ Ok(Self::Metal(storage))+ }+ }+ }++ pub(crate) fn apply_op1(&self, l: &Layout, c: &dyn CustomOp1) -> Result<(Self, Shape)> {+ match self {+ Self::Cpu(storage) => {+ let (storage, shape) = c.cpu_fwd(storage, l)?;+ Ok((Self::Cpu(storage), shape))+ }+ Self::Cuda(storage) => {+ let (storage, shape) = c.cuda_fwd(storage, l)?;+ Ok((Self::Cuda(storage), shape))+ }+ Self::Metal(storage) => {+ let (storage, shape) = c.metal_fwd(storage, l)?;+ Ok((Self::Metal(storage), shape))+ }+ }+ }++ pub(crate) fn apply_op2(+ &self,+ l1: &Layout,+ t2: &Self,+ l2: &Layout,+ c: &dyn CustomOp2,+ ) -> Result<(Self, Shape)> {+ self.same_device(t2, c.name())?;+ match (self, t2) {+ (Self::Cpu(s1), Self::Cpu(s2)) => {+ let (s, shape) = c.cpu_fwd(s1, l1, s2, l2)?;+ Ok((Self::Cpu(s), shape))+ }+ (Self::Cuda(s1), Self::Cuda(s2)) => {+ let (s, shape) = c.cuda_fwd(s1, l1, s2, l2)?;+ Ok((Self::Cuda(s), shape))+ }+ (Self::Metal(s1), Self::Metal(s2)) => {+ let (s, shape) = c.metal_fwd(s1, l1, s2, l2)?;+ Ok((Self::Metal(s), shape))+ }+ _ => unreachable!(),+ }+ }++ pub(crate) fn apply_op3(+ &self,+ l1: &Layout,+ t2: &Self,+ l2: &Layout,+ t3: &Self,+ l3: &Layout,+ c: &dyn CustomOp3,+ ) -> Result<(Self, Shape)> {+ self.same_device(t2, c.name())?;+ self.same_device(t3, c.name())?;+ match (self, t2, t3) {+ (Self::Cpu(s1), Self::Cpu(s2), Self::Cpu(s3)) => {+ let (s, shape) = c.cpu_fwd(s1, l1, s2, l2, s3, l3)?;+ Ok((Self::Cpu(s), shape))+ }+ (Self::Cuda(s1), Self::Cuda(s2), Self::Cuda(s3)) => {+ let (s, shape) = c.cuda_fwd(s1, l1, s2, l2, s3, l3)?;+ Ok((Self::Cuda(s), shape))+ }+ (Self::Metal(s1), Self::Metal(s2), Self::Metal(s3)) => {+ let (s, shape) = c.metal_fwd(s1, l1, s2, l2, s3, l3)?;+ Ok((Self::Metal(s), shape))+ }+ _ => unreachable!(),+ }+ }++ /// Applies an custom in-place op for the `self` tensor.+ ///+ /// [`Src::Aliased`] use the same underlying storage as `self`, while [`Src::Distinct`] does not.+ /// If there are aliases present the aliased forward function on `InplaceOpN` is called. Otherwise+ /// the normal forward function can be used.+ pub(crate) fn inplace_op<const N: usize, C: InplaceOpN<N>>(+ &mut self,+ dst_l: &Layout,+ srcs: [(Src<'_, Storage>, &Layout); N],+ c: &C,+ ) -> Result<()> {+ // Aliased sources are `self`. Only distinct needs checking.+ for (s, _) in srcs.iter() {+ if let Src::Distinct(s) = s {+ self.same_device(s, c.name())?;+ }+ }+ macro_rules! inplace_dispatch {+ ($dst:expr, $variant:ident, $fwd:ident, $fwd_aliased:ident) => {{+ // Extract underlying storage variant on same device+ let operands: [(Src<'_, _>, &Layout); N] = std::array::from_fn(|i| {+ let (s, l) = srcs[i];+ let s = match s {+ Src::Distinct(Storage::$variant(s)) => Src::Distinct(s),+ Src::Aliased(rel) => Src::Aliased(rel),+ Src::Distinct(_) => {+ unreachable!("same_device above rejects mismatched backends")+ }+ };+ (s, l)+ });++ match all_distinct(&operands) {+ Some(distinct) => c.$fwd($dst, dst_l, distinct),+ None => c.$fwd_aliased($dst, dst_l, operands),+ }+ }};+ }++ match self {+ Storage::Cpu(dst) => inplace_dispatch!(dst, Cpu, cpu_fwd, cpu_fwd_aliased),+ Storage::Cuda(dst) => inplace_dispatch!(dst, Cuda, cuda_fwd, cuda_fwd_aliased),+ Storage::Metal(dst) => inplace_dispatch!(dst, Metal, metal_fwd, metal_fwd_aliased),+ }+ }++ pub(crate) fn unary_impl<B: op::UnaryOpT>(&self, layout: &Layout) -> Result<Self> {+ match self {+ Storage::Cpu(storage) => {+ let storage = storage.unary_impl::<B>(layout)?;+ Ok(Self::Cpu(storage))+ }+ Self::Cuda(storage) => {+ let storage = storage.unary_impl::<B>(layout)?;+ Ok(Self::Cuda(storage))+ }+ Self::Metal(storage) => {+ let storage = storage.unary_impl::<B>(layout)?;+ Ok(Self::Metal(storage))+ }+ }+ }++ pub(crate) fn binary_impl<B: op::BinaryOpT>(+ &self,+ rhs: &Self,+ lhs_layout: &Layout,+ rhs_layout: &Layout,+ ) -> Result<Self> {+ self.same_device(rhs, B::NAME)?;+ self.same_dtype(rhs, B::NAME)?;+ match (self, rhs) {+ (Storage::Cpu(lhs), Storage::Cpu(rhs)) => {+ let storage = lhs.binary_impl::<B>(rhs, lhs_layout, rhs_layout)?;+ Ok(Self::Cpu(storage))+ }+ (Self::Cuda(lhs), Self::Cuda(rhs)) => {+ let storage = lhs.binary_impl::<B>(rhs, lhs_layout, rhs_layout)?;+ Ok(Self::Cuda(storage))+ }+ (Self::Metal(lhs), Self::Metal(rhs)) => {+ let storage = lhs.binary_impl::<B>(rhs, lhs_layout, rhs_layout)?;+ Ok(Self::Metal(storage))+ }+ (lhs, rhs) => {+ // Should not happen because of the same device check above but we're defensive+ // anyway.+ Err(Error::DeviceMismatchBinaryOp {+ lhs: lhs.device().location(),+ rhs: rhs.device().location(),+ op: B::NAME,+ }+ .bt())+ }+ }+ }++ pub(crate) fn conv1d(+ &self,+ l: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &crate::conv::ParamsConv1D,+ ) -> Result<Self> {+ self.same_device(kernel, "conv1d")?;+ self.same_dtype(kernel, "conv1d")?;+ match (self, &kernel) {+ (Storage::Cpu(inp), Storage::Cpu(kernel)) => {+ let s = inp.conv1d(l, kernel, kernel_l, params)?;+ Ok(Self::Cpu(s))+ }+ (Storage::Cuda(inp), Storage::Cuda(kernel)) => {+ let s = inp.conv1d(l, kernel, kernel_l, params)?;+ Ok(Self::Cuda(s))+ }+ (Storage::Metal(inp), Storage::Metal(kernel)) => {+ let s = inp.conv1d(l, kernel, kernel_l, params)?;+ Ok(Self::Metal(s))+ }+ (lhs, rhs) => Err(Error::DeviceMismatchBinaryOp {+ lhs: lhs.device().location(),+ rhs: rhs.device().location(),+ op: "conv1d",+ }+ .bt()),+ }+ }++ pub(crate) fn conv_transpose1d(+ &self,+ l: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &crate::conv::ParamsConvTranspose1D,+ ) -> Result<Self> {+ self.same_device(kernel, "conv-transpose1d")?;+ self.same_dtype(kernel, "conv-transpose1d")?;+ match (self, &kernel) {+ (Storage::Cpu(inp), Storage::Cpu(kernel)) => {+ let s = inp.conv_transpose1d(l, kernel, kernel_l, params)?;+ Ok(Self::Cpu(s))+ }+ (Storage::Cuda(inp), Storage::Cuda(kernel)) => {+ let s = inp.conv_transpose1d(l, kernel, kernel_l, params)?;+ Ok(Self::Cuda(s))+ }+ (Storage::Metal(inp), Storage::Metal(kernel)) => {+ let s = inp.conv_transpose1d(l, kernel, kernel_l, params)?;+ Ok(Self::Metal(s))+ }+ (lhs, rhs) => Err(Error::DeviceMismatchBinaryOp {+ lhs: lhs.device().location(),+ rhs: rhs.device().location(),+ op: "conv-transpose1d",+ }+ .bt()),+ }+ }++ pub(crate) fn conv2d(+ &self,+ l: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &crate::conv::ParamsConv2D,+ ) -> Result<Self> {+ self.same_device(kernel, "conv2d")?;+ self.same_dtype(kernel, "conv2d")?;+ match (self, &kernel) {+ (Storage::Cpu(inp), Storage::Cpu(kernel)) => {+ let s = inp.conv2d(l, kernel, kernel_l, params)?;+ Ok(Self::Cpu(s))+ }+ (Storage::Cuda(inp), Storage::Cuda(kernel)) => {+ let s = inp.conv2d(l, kernel, kernel_l, params)?;+ Ok(Self::Cuda(s))+ }+ (Storage::Metal(inp), Storage::Metal(kernel)) => {+ let s = inp.conv2d(l, kernel, kernel_l, params)?;+ Ok(Self::Metal(s))+ }+ (lhs, rhs) => Err(Error::DeviceMismatchBinaryOp {+ lhs: lhs.device().location(),+ rhs: rhs.device().location(),+ op: "conv2d",+ }+ .bt()),+ }+ }++ pub(crate) fn conv_transpose2d(+ &self,+ l: &Layout,+ kernel: &Self,+ kernel_l: &Layout,+ params: &crate::conv::ParamsConvTranspose2D,+ ) -> Result<Self> {+ self.same_device(kernel, "conv_transpose2d")?;+ self.same_dtype(kernel, "conv_transpose2d")?;+ match (self, &kernel) {+ (Storage::Cpu(inp), Storage::Cpu(kernel)) => {+ let s = inp.conv_transpose2d(l, kernel, kernel_l, params)?;+ Ok(Self::Cpu(s))+ }+ (Storage::Cuda(inp), Storage::Cuda(kernel)) => {+ let s = inp.conv_transpose2d(l, kernel, kernel_l, params)?;+ Ok(Self::Cuda(s))+ }+ (Storage::Metal(inp), Storage::Metal(kernel)) => {+ let s = inp.conv_transpose2d(l, kernel, kernel_l, params)?;+ Ok(Self::Metal(s))+ }+ (lhs, rhs) => Err(Error::DeviceMismatchBinaryOp {+ lhs: lhs.device().location(),+ rhs: rhs.device().location(),+ op: "conv_transpose2d",+ }+ .bt()),+ }+ }++ pub(crate) fn avg_pool2d(+ &self,+ layout: &Layout,+ kernel_size: (usize, usize),+ stride: (usize, usize),+ ) -> Result<Self> {+ match self {+ Storage::Cpu(storage) => {+ let storage = storage.avg_pool2d(layout, kernel_size, stride)?;+ Ok(Self::Cpu(storage))+ }+ Self::Cuda(storage) => {+ let storage = storage.avg_pool2d(layout, kernel_size, stride)?;+ Ok(Self::Cuda(storage))+ }+ Self::Metal(storage) => {+ let storage = storage.avg_pool2d(layout, kernel_size, stride)?;+ Ok(Self::Metal(storage))+ }+ }+ }++ pub(crate) fn max_pool2d(+ &self,+ layout: &Layout,+ kernel_size: (usize, usize),+ stride: (usize, usize),+ ) -> Result<Self> {+ match self {+ Storage::Cpu(storage) => {+ let storage = storage.max_pool2d(layout, kernel_size, stride)?;+ Ok(Self::Cpu(storage))+ }+ Self::Cuda(storage) => {+ let storage = storage.max_pool2d(layout, kernel_size, stride)?;+ Ok(Self::Cuda(storage))+ }+ Self::Metal(storage) => {+ let storage = storage.max_pool2d(layout, kernel_size, stride)?;+ Ok(Self::Metal(storage))+ }+ }+ }++ pub(crate) fn upsample_nearest1d(&self, layout: &Layout, sz: usize) -> Result<Self> {+ match self {+ Storage::Cpu(storage) => {+ let storage = storage.upsample_nearest1d(layout, sz)?;+ Ok(Self::Cpu(storage))+ }+ Self::Cuda(storage) => {+ let storage = storage.upsample_nearest1d(layout, sz)?;+ Ok(Self::Cuda(storage))+ }+ Self::Metal(storage) => {+ let storage = storage.upsample_nearest1d(layout, sz)?;+ Ok(Self::Metal(storage))+ }+ }+ }++ pub(crate) fn upsample_nearest2d(&self, layout: &Layout, h: usize, w: usize) -> Result<Self> {+ match self {+ Storage::Cpu(storage) => {+ let storage = storage.upsample_nearest2d(layout, h, w)?;+ Ok(Self::Cpu(storage))+ }+ Self::Cuda(storage) => {+ let storage = storage.upsample_nearest2d(layout, h, w)?;+ Ok(Self::Cuda(storage))+ }+ Self::Metal(storage) => {+ let storage = storage.upsample_nearest2d(layout, h, w)?;+ Ok(Self::Metal(storage))+ }+ }+ }++ pub(crate) fn upsample_bilinear2d(+ &self,+ layout: &Layout,+ h: usize,+ w: usize,+ align_corners: bool,+ scale_h: Option<f64>,+ scale_w: Option<f64>,+ ) -> Result<Self> {+ match self {+ Storage::Cpu(storage) => {+ let storage =+ storage.upsample_bilinear2d(layout, h, w, align_corners, scale_h, scale_w)?;+ Ok(Self::Cpu(storage))+ }+ Self::Cuda(storage) => {+ let storage =+ storage.upsample_bilinear2d(layout, h, w, align_corners, scale_h, scale_w)?;+ Ok(Self::Cuda(storage))+ }+ Self::Metal(storage) => {+ let storage =+ storage.upsample_bilinear2d(layout, h, w, align_corners, scale_h, scale_w)?;+ Ok(Self::Metal(storage))+ }+ }+ }++ pub(crate) fn where_cond(+ &self,+ layout: &Layout,+ t: &Self,+ layout_t: &Layout,+ f: &Self,+ layout_f: &Layout,+ ) -> Result<Self> {+ self.same_device(t, "where")?;+ self.same_device(f, "where")?;+ t.same_dtype(f, "where")?;+ match (self, t, f) {+ (Storage::Cpu(cond), Storage::Cpu(t), Storage::Cpu(f)) => {+ let storage = cond.where_cond(layout, t, layout_t, f, layout_f)?;+ Ok(Self::Cpu(storage))+ }+ (Self::Cuda(cond), Self::Cuda(t), Self::Cuda(f)) => {+ let storage = cond.where_cond(layout, t, layout_t, f, layout_f)?;+ Ok(Self::Cuda(storage))+ }+ (Self::Metal(cond), Self::Metal(t), Self::Metal(f)) => {+ let storage = cond.where_cond(layout, t, layout_t, f, layout_f)?;+ Ok(Self::Metal(storage))+ }+ (_, lhs, rhs) => Err(Error::DeviceMismatchBinaryOp {+ lhs: lhs.device().location(),+ rhs: rhs.device().location(),+ op: "where",+ }+ .bt()),+ }+ }++ pub(crate) fn gather(+ &self,+ l: &Layout,+ indexes: &Self,+ indexes_l: &Layout,+ d: usize,+ ) -> Result<Self> {+ self.same_device(indexes, "index-add")?;+ match (self, indexes) {+ (Self::Cpu(s), Self::Cpu(indexes)) => {+ let storage = s.gather(l, indexes, indexes_l, d)?;+ Ok(Self::Cpu(storage))+ }+ (Self::Cuda(s), Self::Cuda(indexes)) => {+ let storage = s.gather(l, indexes, indexes_l, d)?;+ Ok(Self::Cuda(storage))+ }+ (Self::Metal(s), Self::Metal(indexes)) => {+ let storage = s.gather(l, indexes, indexes_l, d)?;+ Ok(Self::Metal(storage))+ }+ _ => unreachable!(),+ }+ }++ pub(crate) fn scatter_set(+ &mut self,+ l: &Layout,+ indexes: &Self,+ indexes_l: &Layout,+ source: &Self,+ source_l: &Layout,+ d: usize,+ ) -> Result<()> {+ self.same_device(indexes, "scatter-set")?;+ self.same_device(source, "scatter-set")?;+ match (self, indexes, source) {+ (Self::Cpu(s), Self::Cpu(indexes), Self::Cpu(source)) => {+ s.scatter_set(l, indexes, indexes_l, source, source_l, d)?;+ }+ (Self::Cuda(s), Self::Cuda(indexes), Self::Cuda(source)) => {+ s.scatter_set(l, indexes, indexes_l, source, source_l, d)?;+ }+ (Self::Metal(s), Self::Metal(indexes), Self::Metal(source)) => {+ s.scatter_set(l, indexes, indexes_l, source, source_l, d)?;+ }+ _ => unreachable!(),+ }+ Ok(())+ }++ pub(crate) fn scatter_add(+ &mut self,+ l: &Layout,+ indexes: &Self,+ indexes_l: &Layout,+ source: &Self,+ source_l: &Layout,+ d: usize,+ ) -> Result<()> {+ self.same_device(indexes, "scatter-add")?;+ self.same_device(source, "scatter-add")?;+ match (self, indexes, source) {+ (Self::Cpu(s), Self::Cpu(indexes), Self::Cpu(source)) => {+ s.scatter_add_set(l, indexes, indexes_l, source, source_l, d)?;+ }+ (Self::Cuda(s), Self::Cuda(indexes), Self::Cuda(source)) => {+ s.scatter_add_set(l, indexes, indexes_l, source, source_l, d)?;+ }+ (Self::Metal(s), Self::Metal(indexes), Self::Metal(source)) => {+ s.scatter_add_set(l, indexes, indexes_l, source, source_l, d)?;+ }+ _ => unreachable!(),+ }+ Ok(())+ }++ pub(crate) fn index_add(+ &self,+ l: &Layout,+ indexes: &Self,+ indexes_l: &Layout,+ source: &Self,+ source_l: &Layout,+ d: usize,+ ) -> Result<Self> {+ self.same_device(indexes, "index-add")?;+ self.same_device(source, "index-add")?;+ match (self, indexes, source) {+ (Self::Cpu(s), Self::Cpu(indexes), Self::Cpu(source)) => {+ let storage = s.index_add(l, indexes, indexes_l, source, source_l, d)?;+ Ok(Self::Cpu(storage))+ }+ (Self::Cuda(s), Self::Cuda(indexes), Self::Cuda(source)) => {+ let storage = s.index_add(l, indexes, indexes_l, source, source_l, d)?;+ Ok(Self::Cuda(storage))+ }+ (Self::Metal(s), Self::Metal(indexes), Self::Metal(source)) => {+ let storage = s.index_add(l, indexes, indexes_l, source, source_l, d)?;+ Ok(Self::Metal(storage))+ }+ _ => unreachable!(),+ }+ }++ pub(crate) fn index_select(+ &self,+ rhs: &Self,+ lhs_l: &Layout,+ rhs_l: &Layout,+ d: usize,+ ) -> Result<Self> {+ self.same_device(rhs, "index-select")?;+ match (self, rhs) {+ (Self::Cpu(lhs), Self::Cpu(rhs)) => {+ let storage = lhs.index_select(rhs, lhs_l, rhs_l, d)?;+ Ok(Self::Cpu(storage))+ }+ (Self::Cuda(lhs), Self::Cuda(rhs)) => {+ let storage = lhs.index_select(rhs, lhs_l, rhs_l, d)?;+ Ok(Self::Cuda(storage))+ }+ (Self::Metal(lhs), Self::Metal(rhs)) => {+ let storage = lhs.index_select(rhs, lhs_l, rhs_l, d)?;+ Ok(Self::Metal(storage))+ }+ (lhs, rhs) => Err(Error::DeviceMismatchBinaryOp {+ lhs: lhs.device().location(),+ rhs: rhs.device().location(),+ op: "index-select",+ }+ .bt()),+ }+ }++ pub(crate) fn matmul(+ &self,+ rhs: &Self,+ bmnk: (usize, usize, usize, usize),+ lhs_layout: &Layout,+ rhs_layout: &Layout,+ ) -> Result<Self> {+ self.same_device(rhs, "matmul")?;+ self.same_dtype(rhs, "matmul")?;+ match (self, rhs) {+ (Self::Cpu(lhs), Self::Cpu(rhs)) => {+ let storage = lhs.matmul(rhs, bmnk, lhs_layout, rhs_layout)?;+ Ok(Self::Cpu(storage))+ }+ (Self::Cuda(lhs), Self::Cuda(rhs)) => {+ let storage = lhs.matmul(rhs, bmnk, lhs_layout, rhs_layout)?;+ Ok(Self::Cuda(storage))+ }+ (Self::Metal(lhs), Self::Metal(rhs)) => {+ let storage = lhs.matmul(rhs, bmnk, lhs_layout, rhs_layout)?;+ Ok(Self::Metal(storage))+ }+ (lhs, rhs) => Err(Error::DeviceMismatchBinaryOp {+ lhs: lhs.device().location(),+ rhs: rhs.device().location(),+ op: "matmul",+ }+ .bt()),+ }+ }++ // self, the source can be strided whereas dst is contiguous.+ pub(crate) fn copy_strided_src(+ &self,+ dst: &mut Self,+ dst_offset: usize,+ src_l: &Layout,+ ) -> Result<()> {+ match (self, dst) {+ (Self::Cpu(src), Self::Cpu(dst)) => src.copy_strided_src(dst, dst_offset, src_l),+ (Self::Cuda(src), Self::Cuda(dst)) => Ok(src.copy_strided_src(dst, dst_offset, src_l)?),+ (Self::Metal(src), Self::Metal(dst)) => {+ Ok(src.copy_strided_src(dst, dst_offset, src_l)?)+ }+ (lhs, rhs) => Err(Error::DeviceMismatchBinaryOp {+ lhs: lhs.device().location(),+ rhs: rhs.device().location(),+ op: "copy",+ }+ .bt()),+ }+ }++ #[allow(clippy::too_many_arguments)]+ pub(crate) fn copy2d(+ &self,+ dst: &mut Self,+ d1: usize,+ d2: usize,+ src_s: usize,+ dst_s: usize,+ src_o: usize,+ dst_o: usize,+ ) -> Result<()> {+ match (self, dst) {+ (Self::Cpu(src), Self::Cpu(dst)) => src.copy2d(dst, d1, d2, src_s, dst_s, src_o, dst_o),+ (Self::Cuda(src), Self::Cuda(dst)) => {+ Ok(src.copy2d(dst, d1, d2, src_s, dst_s, src_o, dst_o)?)+ }+ (Self::Metal(src), Self::Metal(dst)) => {+ Ok(src.copy2d(dst, d1, d2, src_s, dst_s, src_o, dst_o)?)+ }+ (lhs, rhs) => Err(Error::DeviceMismatchBinaryOp {+ lhs: lhs.device().location(),+ rhs: rhs.device().location(),+ op: "copy2d",+ }+ .bt()),+ }+ }+}diff --git a/vendor/candle-core/src/streaming.rs b/vendor/candle-core/src/streaming.rsnew file mode 100644index 0000000..f4c0a9f--- /dev/null+++ b/vendor/candle-core/src/streaming.rs@@ -0,0 +1,208 @@+//! StreamTensror useful for streaming ops.+//!+use crate::{Result, Shape, Tensor};++pub trait Dim: crate::shape::Dim + Copy {}+impl<T: crate::shape::Dim + Copy> Dim for T {}++/// A stream tensor is used in streaming module. It can either contain an actual tensor or be+/// empty.+#[derive(Clone)]+pub struct StreamTensor(Option<Tensor>);++impl std::fmt::Debug for StreamTensor {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ match &self.0 {+ Some(t) => write!(f, "{:?}", t.shape()),+ None => write!(f, "Empty"),+ }+ }+}++impl std::convert::From<Option<Tensor>> for StreamTensor {+ fn from(value: Option<Tensor>) -> Self {+ Self(value)+ }+}++impl std::convert::From<Tensor> for StreamTensor {+ fn from(value: Tensor) -> Self {+ Self(Some(value))+ }+}++impl std::convert::From<()> for StreamTensor {+ fn from(_value: ()) -> Self {+ Self(None)+ }+}++impl StreamTensor {+ pub fn empty() -> Self {+ Self(None)+ }++ pub fn from_tensor(tensor: Tensor) -> Self {+ Self(Some(tensor))+ }++ pub fn shape(&self) -> Option<&Shape> {+ self.0.as_ref().map(|t| t.shape())+ }++ pub fn cat2<D: Dim>(&self, rhs: &Self, dim: D) -> Result<Self> {+ let xs = match (&self.0, &rhs.0) {+ (Some(lhs), Some(rhs)) => {+ let xs = Tensor::cat(&[lhs, rhs], dim)?;+ Some(xs)+ }+ (Some(xs), None) | (None, Some(xs)) => Some(xs.clone()),+ (None, None) => None,+ };+ Ok(Self(xs))+ }++ pub fn seq_len<D: Dim>(&self, dim: D) -> Result<usize> {+ match &self.0 {+ None => Ok(0),+ Some(v) => v.dim(dim),+ }+ }++ pub fn reset(&mut self) {+ self.0 = None+ }++ pub fn narrow<D: Dim>(&self, dim: D, offset: usize, len: usize) -> Result<StreamTensor> {+ let t = match &self.0 {+ None => None,+ Some(t) => {+ let seq_len = t.dim(dim)?;+ if seq_len <= offset {+ None+ } else {+ let t = t.narrow(dim, offset, usize::min(len, seq_len - offset))?;+ Some(t)+ }+ }+ };+ Ok(Self(t))+ }++ /// Splits the Streaming Tensor on the time axis `dim` with the first `lhs_len` elements+ /// returned in the first output and the remaining in the second output.+ pub fn split<D: Dim>(&self, dim: D, lhs_len: usize) -> Result<(Self, Self)> {+ match &self.0 {+ None => Ok((Self::empty(), Self::empty())),+ Some(t) => {+ let seq_len = t.dim(dim)?;+ let lhs_len = usize::min(seq_len, lhs_len);+ if lhs_len == 0 {+ Ok((Self::empty(), t.clone().into()))+ } else {+ let lhs = Self::from_tensor(t.narrow(dim, 0, lhs_len)?);+ let rhs_len = seq_len - lhs_len;+ let rhs = if rhs_len == 0 {+ Self::empty()+ } else {+ Self::from_tensor(t.narrow(dim, lhs_len, rhs_len)?)+ };+ Ok((lhs, rhs))+ }+ }+ }+ }++ pub fn as_option(&self) -> Option<&Tensor> {+ self.0.as_ref()+ }++ pub fn apply<M: crate::Module>(&self, m: &M) -> Result<Self> {+ match &self.0 {+ None => Ok(Self::empty()),+ Some(t) => Ok(Self::from_tensor(t.apply(m)?)),+ }+ }+}++/// Streaming modules take as input a stream tensor and return a stream tensor. They may perform+/// some internal buffering so that enough data has been received for the module to be able to+/// perform some operations.+pub trait StreamingModule {+ // TODO: Should we also have a flush method?+ fn step(&mut self, xs: &StreamTensor) -> Result<StreamTensor>;+ fn reset_state(&mut self);+}++#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]+pub enum BinOp {+ Add,+ Mul,+ Sub,+ Div,+}++#[derive(Debug, Clone)]+pub struct StreamingBinOp {+ prev_lhs: StreamTensor,+ prev_rhs: StreamTensor,+ pub op: BinOp,+ pub dim: crate::D,+}++impl StreamingBinOp {+ pub fn new(op: BinOp, dim: crate::D) -> Self {+ Self {+ prev_lhs: StreamTensor::empty(),+ prev_rhs: StreamTensor::empty(),+ op,+ dim,+ }+ }++ pub fn reset_state(&mut self) {+ self.prev_lhs.reset();+ self.prev_rhs.reset();+ }++ pub fn forward(&self, lhs: &Tensor, rhs: &Tensor) -> Result<Tensor> {+ match self.op {+ BinOp::Add => Tensor::add(lhs, rhs),+ BinOp::Mul => Tensor::mul(lhs, rhs),+ BinOp::Sub => Tensor::sub(lhs, rhs),+ BinOp::Div => Tensor::div(lhs, rhs),+ }+ }++ pub fn step(&mut self, lhs: &StreamTensor, rhs: &StreamTensor) -> Result<StreamTensor> {+ let lhs = StreamTensor::cat2(&self.prev_lhs, lhs, self.dim)?;+ let rhs = StreamTensor::cat2(&self.prev_rhs, rhs, self.dim)?;+ let lhs_len = lhs.seq_len(self.dim)?;+ let rhs_len = rhs.seq_len(self.dim)?;+ let common_len = usize::min(lhs_len, rhs_len);+ let (lhs, prev_lhs) = lhs.split(self.dim, common_len)?;+ let (rhs, prev_rhs) = rhs.split(self.dim, common_len)?;+ let ys = match (lhs.0, rhs.0) {+ (Some(lhs), Some(rhs)) => {+ let ys = self.forward(&lhs, &rhs)?;+ StreamTensor::from_tensor(ys)+ }+ (None, None) => StreamTensor::empty(),+ (lhs, rhs) => crate::bail!("INTERNAL ERROR inconsistent lhs and rhs {lhs:?} {rhs:?}"),+ };+ self.prev_lhs = prev_lhs;+ self.prev_rhs = prev_rhs;+ Ok(ys)+ }+}++/// Simple wrapper that doesn't do any buffering.+pub struct Map<T: crate::Module>(T);++impl<T: crate::Module> StreamingModule for Map<T> {+ fn reset_state(&mut self) {}++ fn step(&mut self, xs: &StreamTensor) -> Result<StreamTensor> {+ xs.apply(&self.0)+ }+}diff --git a/vendor/candle-core/src/strided_index.rs b/vendor/candle-core/src/strided_index.rsnew file mode 100644index 0000000..97764b3--- /dev/null+++ b/vendor/candle-core/src/strided_index.rs@@ -0,0 +1,100 @@+use crate::Layout;++/// An iterator over offset position for items of an N-dimensional arrays stored in a+/// flat buffer using some potential strides.+#[derive(Debug)]+pub struct StridedIndex<'a> {+ next_storage_index: Option<usize>,+ multi_index: Vec<usize>,+ dims: &'a [usize],+ stride: &'a [usize],+ remaining: usize,+}++impl<'a> StridedIndex<'a> {+ pub(crate) fn new(dims: &'a [usize], stride: &'a [usize], start_offset: usize) -> Self {+ let elem_count: usize = dims.iter().product();+ let next_storage_index = if elem_count == 0 {+ None+ } else {+ // This applies to the scalar case.+ Some(start_offset)+ };+ StridedIndex {+ next_storage_index,+ multi_index: vec![0; dims.len()],+ dims,+ stride,+ remaining: elem_count,+ }+ }++ pub(crate) fn from_layout(l: &'a Layout) -> Self {+ Self::new(l.dims(), l.stride(), l.start_offset())+ }+}++impl Iterator for StridedIndex<'_> {+ type Item = usize;++ #[inline]+ fn next(&mut self) -> Option<Self::Item> {+ let storage_index = self.next_storage_index?;+ let mut updated = false;+ let mut next_storage_index = storage_index;+ for ((multi_i, max_i), stride_i) in self+ .multi_index+ .iter_mut()+ .zip(self.dims.iter())+ .zip(self.stride.iter())+ .rev()+ {+ let next_i = *multi_i + 1;+ if next_i < *max_i {+ *multi_i = next_i;+ updated = true;+ next_storage_index += stride_i;+ break;+ } else {+ next_storage_index -= *multi_i * stride_i;+ *multi_i = 0+ }+ }+ self.remaining -= 1;+ self.next_storage_index = if updated {+ Some(next_storage_index)+ } else {+ None+ };+ Some(storage_index)+ }++ #[inline]+ fn size_hint(&self) -> (usize, Option<usize>) {+ (self.remaining, Some(self.remaining))+ }+}++impl ExactSizeIterator for StridedIndex<'_> {+ fn len(&self) -> usize {+ self.remaining+ }+}++#[derive(Debug)]+pub enum StridedBlocks<'a> {+ SingleBlock {+ start_offset: usize,+ len: usize,+ },+ UniformBlocks {+ start_offset: usize,+ block_len: usize,+ count: usize,+ src_stride: usize,+ },+ MultipleBlocks {+ block_start_index: StridedIndex<'a>,+ block_len: usize,+ },+}diff --git a/vendor/candle-core/src/tensor.rs b/vendor/candle-core/src/tensor.rsnew file mode 100644index 0000000..0b8a0a2--- /dev/null+++ b/vendor/candle-core/src/tensor.rs@@ -0,0 +1,3152 @@+//! Tensors are N-dimensional matrixes of elements using a single data type.+#![allow(clippy::redundant_closure_call)]+use crate::backend::{BackendDevice, BackendStorage};+use crate::op::{BackpropOp, BinaryOp, CmpOp, Op, ReduceOp, UnaryOp};+use crate::scalar::TensorOrScalar;+use crate::shape::{Dim, Dims, ShapeWithOneHole};+use crate::storage::{StorageMutRef, StorageRef};+use crate::{bail, storage::Storage, DType, Device, Error, Layout, Result, Shape};+use parking_lot::RwLock;+use std::sync::Arc;++/// Unique identifier for tensors.+#[derive(Clone, Copy, Debug, PartialEq, Eq, Hash)]+pub struct TensorId(usize);++impl TensorId {+ fn new() -> Self {+ // https://users.rust-lang.org/t/idiomatic-rust-way-to-generate-unique-id/33805+ use std::sync::atomic;+ static COUNTER: atomic::AtomicUsize = atomic::AtomicUsize::new(1);+ Self(COUNTER.fetch_add(1, atomic::Ordering::Relaxed))+ }+}++pub struct Tensor_ {+ id: TensorId,+ // As we provide inner mutability on the tensor content, the alternatives are:+ // - Using a mutex, this would have the highest cost when retrieving the storage but would+ // prevent errors when concurrent access takes place. Mutex would also be subject to+ // deadlocks for example using the current code if the same tensor is used twice by a single+ // binary op.+ // - Using a refcell unsafe cell would have some intermediary cost, borrow checking would be+ // verified dynamically, but the resulting tensors would not be send or sync.+ // - Using an unsafe cell would have the lowest cost but undefined behavior on concurrent+ // accesses.+ // Ideally, we would use Arc<Storage> for tensors on which we don't plan on modifying the data+ // and Arc<Mutex<Storage>> for tensors where the data could be modified, e.g. variables but+ // that's tricky to encode in the current setup.+ storage: Arc<RwLock<Storage>>,+ layout: Layout,+ op: BackpropOp,+ is_variable: bool,+ dtype: DType,+ device: Device,+}++impl AsRef<Tensor> for Tensor {+ fn as_ref(&self) -> &Tensor {+ self+ }+}++// Tensors are refcounted so that cloning is cheap when building the op graph.+// Storages are also refcounted independently so that its possible to avoid+// copying the storage for operations that only modify the shape or stride.+#[derive(Clone)]+/// The core struct for manipulating tensors.+///+/// ```rust+/// use candle_core::{Tensor, DType, Device};+///+/// let a = Tensor::arange(0f32, 6f32, &Device::Cpu)?.reshape((2, 3))?;+/// let b = Tensor::arange(0f32, 12f32, &Device::Cpu)?.reshape((3, 4))?;+///+/// let c = a.matmul(&b)?;+/// # Ok::<(), candle_core::Error>(())+/// ```+///+/// Tensors are reference counted with [`Arc`] so cloning them is cheap.+pub struct Tensor(Arc<Tensor_>);++impl std::ops::Deref for Tensor {+ type Target = Tensor_;++ fn deref(&self) -> &Self::Target {+ self.0.as_ref()+ }+}++macro_rules! unary_op {+ ($fn_name:ident, $op_name:ident) => {+ pub fn $fn_name(&self) -> Result<Self> {+ let shape = self.shape();+ if shape.elem_count() == 0 {+ return Ok(self.clone());+ }+ let storage = self+ .storage()+ .unary_impl::<crate::op::$op_name>(self.layout())?;+ let op = BackpropOp::new1(self, |s| Op::Unary(s, UnaryOp::$op_name));+ Ok(from_storage(storage, shape.clone(), op, false))+ }+ };+}++macro_rules! binary_op {+ ($fn_name:ident, $op_name:ident) => {+ pub fn $fn_name(&self, rhs: &Self) -> Result<Self> {+ let shape = self.same_shape_binary_op(rhs, stringify!($fn_name))?;+ if shape.elem_count() == 0 {+ return Ok(self.clone());+ }+ let storage = self.storage().binary_impl::<crate::op::$op_name>(+ &*rhs.storage(),+ self.layout(),+ rhs.layout(),+ )?;+ let op = BackpropOp::new2(self, rhs, |t1, t2| Op::Binary(t1, t2, BinaryOp::$op_name));+ Ok(from_storage(storage, shape.clone(), op, false))+ }+ };+}++macro_rules! binary_op_scalar {+ ($fn_name:ident, $op_name:ident) => {+ pub fn $fn_name<T: TensorOrScalar>(&self, rhs: T) -> Result<Self> {+ let rhs = match rhs.to_tensor_scalar()? {+ crate::scalar::TensorScalar::Tensor(rhs) => rhs,+ crate::scalar::TensorScalar::Scalar(rhs) => rhs+ .to_dtype(self.dtype())?+ .to_device(self.device())?+ .broadcast_as(self.shape())?,+ };+ let shape = self.same_shape_binary_op(&rhs, stringify!($fn_name))?;+ if self.elem_count() == 0 {+ return Ok(self.clone());+ }+ let storage = self.storage().binary_impl::<crate::op::$op_name>(+ &*rhs.storage(),+ self.layout(),+ rhs.layout(),+ )?;+ let op = BackpropOp::new2(self, &rhs, |t1, t2| Op::Binary(t1, t2, BinaryOp::$op_name));+ Ok(from_storage(storage, shape.clone(), op, false))+ }+ };+}++macro_rules! broadcast_binary_op {+ ($fn_name:ident, $inner_fn_name:ident) => {+ pub fn $fn_name(&self, rhs: &Self) -> Result<Self> {+ let lhs = self;+ let shape = lhs+ .shape()+ .broadcast_shape_binary_op(rhs.shape(), stringify!($fn_name))?;+ let l_broadcast = shape != *lhs.shape();+ let r_broadcast = shape != *rhs.shape();+ match (l_broadcast, r_broadcast) {+ (true, true) => lhs+ .broadcast_as(&shape)?+ .$inner_fn_name(&rhs.broadcast_as(&shape)?),+ (false, true) => lhs.$inner_fn_name(&rhs.broadcast_as(&shape)?),+ (true, false) => lhs.broadcast_as(&shape)?.$inner_fn_name(rhs),+ (false, false) => lhs.$inner_fn_name(rhs),+ }+ }+ };+}++/// Creates a fresh tensor structure based on a storage and a shape, this uses contiguous strides.+pub(crate) fn from_storage<S: Into<Shape>>(+ storage: Storage,+ shape: S,+ op: BackpropOp,+ is_variable: bool,+) -> Tensor {+ let dtype = storage.dtype();+ let device = storage.device();+ let tensor_ = Tensor_ {+ id: TensorId::new(),+ storage: Arc::new(RwLock::new(storage)),+ layout: Layout::contiguous(shape),+ op,+ is_variable,+ dtype,+ device,+ };+ Tensor(Arc::new(tensor_))+}++impl Tensor {+ pub(crate) fn ones_impl<S: Into<Shape>>(+ shape: S,+ dtype: DType,+ device: &Device,+ is_variable: bool,+ ) -> Result<Self> {+ let none = BackpropOp::none();+ let shape = shape.into();+ let mut storage = unsafe { device.alloc_uninit(&shape, dtype)? };+ let layout = Layout::contiguous(shape.clone());+ storage.const_set(crate::scalar::Scalar::one(dtype), &layout)?;+ Ok(from_storage(storage, shape, none, is_variable))+ }++ /// Creates a new tensor filled with ones.+ ///+ /// ```rust+ /// use candle_core::{Tensor, DType, Device};+ /// let a = Tensor::ones((2, 3), DType::F32, &Device::Cpu)?;+ /// let b = Tensor::from_slice(&[1.0f32, 1.0, 1.0, 1.0, 1.0, 1.0], (2, 3), &Device::Cpu)?;+ /// // a == b+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn ones<S: Into<Shape>>(shape: S, dtype: DType, device: &Device) -> Result<Self> {+ Self::ones_impl(shape, dtype, device, false)+ }++ pub fn const_set(&self, value: crate::scalar::Scalar) -> Result<()> {+ self.storage_mut().const_set(value, self.layout())+ }++ pub fn zero_set(&self) -> Result<()> {+ self.const_set(crate::scalar::Scalar::zero(self.dtype()))+ }++ pub fn one_set(&self) -> Result<()> {+ self.const_set(crate::scalar::Scalar::one(self.dtype()))+ }++ /// Creates a new tensor filled with ones with same shape, dtype, and device as the other tensor.+ ///+ /// ```rust+ /// use candle_core::{Tensor, DType, Device};+ /// let a = Tensor::zeros((2, 3), DType::F32, &Device::Cpu)?;+ /// let b = a.ones_like()?;+ /// // b == a + 1+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn ones_like(&self) -> Result<Self> {+ Tensor::ones(self.shape(), self.dtype(), self.device())+ }++ // Do not expose outside of the crate, the `is_variable=true` case should only be accessed from+ // the variable module.+ pub(crate) fn zeros_impl<S: Into<Shape>>(+ shape: S,+ dtype: DType,+ device: &Device,+ is_variable: bool,+ ) -> Result<Self> {+ let none = BackpropOp::none();+ let shape = shape.into();+ let storage = device.zeros(&shape, dtype)?;+ Ok(from_storage(storage, shape, none, is_variable))+ }++ /// Creates a new tensor filled with zeros.+ ///+ /// ```rust+ /// use candle_core::{Tensor, DType, Device};+ /// let a = Tensor::zeros((2, 3), DType::F32, &Device::Cpu)?;+ /// let b = Tensor::from_slice(&[0.0f32, 0.0, 0.0, 0.0, 0.0, 0.0], (2, 3), &Device::Cpu)?;+ /// // a == b+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn zeros<S: Into<Shape>>(shape: S, dtype: DType, device: &Device) -> Result<Self> {+ Self::zeros_impl(shape, dtype, device, false)+ }++ /// Creates a new tensor filled with zeros with same shape, dtype, and device as the other+ /// tensor.+ ///+ /// ```rust+ /// use candle_core::{Tensor, DType, Device};+ /// let a = Tensor::zeros((2, 3), DType::F32, &Device::Cpu)?;+ /// let b = a.zeros_like()?;+ /// // b is on CPU f32.+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn zeros_like(&self) -> Result<Self> {+ Tensor::zeros(self.shape(), self.dtype(), self.device())+ }++ // Do not expose outside of the crate, the `is_variable=true` case should only be accessed from+ // the variable module.+ pub(crate) unsafe fn empty_impl<S: Into<Shape>>(+ shape: S,+ dtype: DType,+ device: &Device,+ is_variable: bool,+ ) -> Result<Self> {+ let none = BackpropOp::none();+ let shape = shape.into();+ let storage = device.alloc_uninit(&shape, dtype)?;+ Ok(from_storage(storage, shape, none, is_variable))+ }++ /// Creates a new tensor filled with uninitialized memory.+ ///+ /// # Safety+ /// This returns uninitialized memory.+ ///+ /// ```rust+ /// use candle_core::{Tensor, DType, Device};+ /// let a = unsafe { Tensor::empty((2, 3), DType::F32, &Device::Cpu)? };+ /// // a == b+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub unsafe fn empty<S: Into<Shape>>(shape: S, dtype: DType, device: &Device) -> Result<Self> {+ Self::empty_impl(shape, dtype, device, false)+ }++ /// Creates a new tensor filled with uninitialized memory of the same shape, dtype, and device as the other+ /// tensor.+ ///+ /// # Safety+ /// This returns uninitialized memory.+ ///+ /// ```rust+ /// use candle_core::{Tensor, DType, Device};+ /// let a = Tensor::zeros((2, 3), DType::F32, &Device::Cpu)?;+ /// let b = unsafe { a.empty_like()? };+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub unsafe fn empty_like(&self) -> Result<Self> {+ Tensor::empty(self.shape(), self.dtype(), self.device())+ }++ pub(crate) fn rand_impl<S: Into<Shape>, T: crate::FloatDType>(+ lo: T,+ up: T,+ s: S,+ device: &Device,+ is_variable: bool,+ ) -> Result<Self> {+ let s = s.into();+ let storage = device.rand_uniform(lo, up, &s)?;+ let none = BackpropOp::none();+ Ok(from_storage(storage, s, none, is_variable))+ }++ pub(crate) fn rand_f64_impl<S: Into<Shape>>(+ lo: f64,+ up: f64,+ s: S,+ dtype: DType,+ device: &Device,+ is_variable: bool,+ ) -> Result<Self> {+ let s = s.into();+ let storage = device.rand_uniform_f64(lo, up, &s, dtype)?;+ let none = BackpropOp::none();+ Ok(from_storage(storage, s, none, is_variable))+ }++ /// Creates a new tensor initialized with values sampled uniformly between `lo` and `up`.+ pub fn rand<S: Into<Shape>, T: crate::FloatDType>(+ lo: T,+ up: T,+ s: S,+ device: &Device,+ ) -> Result<Self> {+ Self::rand_impl(lo, up, s, device, false)+ }++ pub fn rand_like(&self, lo: f64, up: f64) -> Result<Self> {+ Tensor::rand_f64_impl(lo, up, self.shape(), self.dtype(), self.device(), false)+ }++ pub(crate) fn randn_impl<S: Into<Shape>, T: crate::FloatDType>(+ mean: T,+ std: T,+ s: S,+ device: &Device,+ is_variable: bool,+ ) -> Result<Self> {+ let s = s.into();+ let storage = device.rand_normal(mean, std, &s)?;+ let none = BackpropOp::none();+ Ok(from_storage(storage, s, none, is_variable))+ }++ pub(crate) fn randn_f64_impl<S: Into<Shape>>(+ mean: f64,+ std: f64,+ s: S,+ dtype: DType,+ device: &Device,+ is_variable: bool,+ ) -> Result<Self> {+ let s = s.into();+ let storage = device.rand_normal_f64(mean, std, &s, dtype)?;+ let none = BackpropOp::none();+ Ok(from_storage(storage, s, none, is_variable))+ }++ pub fn randn_like(&self, mean: f64, stdev: f64) -> Result<Self> {+ Tensor::randn_f64_impl(+ mean,+ stdev,+ self.shape(),+ self.dtype(),+ self.device(),+ false,+ )+ }++ /// Creates a new tensor initialized with values sampled from a normal distribution with the+ /// specified `mean` and standard deviation `std`.+ pub fn randn<S: Into<Shape>, T: crate::FloatDType>(+ mean: T,+ std: T,+ s: S,+ device: &Device,+ ) -> Result<Self> {+ Self::randn_impl(mean, std, s, device, false)+ }++ pub(crate) fn new_impl<A: crate::device::NdArray>(+ array: A,+ shape: Shape,+ device: &Device,+ is_variable: bool,+ ) -> Result<Self> {+ let n: usize = shape.elem_count();+ let buffer_size: usize = array.shape()?.elem_count();+ if buffer_size != n {+ return Err(Error::ShapeMismatch { buffer_size, shape }.bt());+ }+ let storage = device.storage(array)?;+ let none = BackpropOp::none();+ Ok(from_storage(storage, shape, none, is_variable))+ }++ /// Creates a new tensor on the specified device using the content and shape of the input.+ pub fn new<A: crate::device::NdArray>(array: A, device: &Device) -> Result<Self> {+ let shape = array.shape()?;+ Self::new_impl(array, shape, device, false)+ }++ /// Returns a new tensor with all the elements having the same specified value.+ ///```rust+ /// use candle_core::{Tensor, Device};+ /// let a = Tensor::full(3.5, (2, 4), &Device::Cpu)?;+ ///+ /// assert_eq!(a.to_vec2::<f64>()?, &[+ /// [3.5, 3.5, 3.5, 3.5],+ /// [3.5, 3.5, 3.5, 3.5],+ /// ]);+ /// # Ok::<(), candle_core::Error>(())+ pub fn full<D: crate::WithDType, S: Into<Shape>>(+ value: D,+ shape: S,+ device: &Device,+ ) -> Result<Self> {+ let none = BackpropOp::none();+ let shape = shape.into();+ let mut storage = unsafe { device.alloc_uninit(&shape, D::DTYPE)? };+ let layout = Layout::contiguous(shape.clone());+ storage.const_set(value.to_scalar(), &layout)?;+ Ok(from_storage(storage, shape, none, false))+ }++ /// Creates a new 1D tensor from an iterator.+ ///```rust+ /// use candle_core::{Tensor, Device};+ /// let a = Tensor::from_iter( [1.0, 2.0, 3.0, 4.0].into_iter(), &Device::Cpu)?;+ ///+ /// assert_eq!(a.to_vec1::<f64>()?, &[1.0, 2.0, 3.0, 4.0]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn from_iter<D: crate::WithDType>(+ iter: impl IntoIterator<Item = D>,+ device: &Device,+ ) -> Result<Self> {+ let data = iter.into_iter().collect::<Vec<_>>();+ let len = data.len();+ Self::from_vec_impl(data, len, device, false)+ }++ /// Creates a new 1D tensor with values from the interval `[start, end)` taken with a common+ /// difference `1` from `start`.+ ///```rust+ /// use candle_core::{Tensor, Device};+ /// let a = Tensor::arange(2., 5., &Device::Cpu)?;+ ///+ /// assert_eq!(a.to_vec1::<f64>()?, &[2., 3., 4.]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn arange<D: crate::WithDType>(start: D, end: D, device: &Device) -> Result<Self> {+ Self::arange_step(start, end, D::one(), device)+ }++ /// Creates a new 1D tensor with values from the interval `[start, end)` taken with a common+ /// difference `step` from `start`.+ ///```rust+ /// use candle_core::{Tensor, Device};+ /// let a = Tensor::arange_step(2.0, 4.0, 0.5, &Device::Cpu)?;+ ///+ /// assert_eq!(a.to_vec1::<f64>()?, &[2.0, 2.5, 3.0, 3.5]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn arange_step<D: crate::WithDType>(+ start: D,+ end: D,+ step: D,+ device: &Device,+ ) -> Result<Self> {+ if D::is_zero(&step) {+ bail!("step cannot be zero")+ }+ let mut data = vec![];+ let mut current = start;+ if step >= D::zero() {+ while current < end {+ data.push(current);+ current += step;+ }+ } else {+ while current > end {+ data.push(current);+ current += step;+ }+ }+ let len = data.len();+ Self::from_vec_impl(data, len, device, false)+ }++ pub(crate) fn from_vec_impl<S: ShapeWithOneHole, D: crate::WithDType>(+ data: Vec<D>,+ shape: S,+ device: &Device,+ is_variable: bool,+ ) -> Result<Self> {+ let shape = shape.into_shape(data.len())?;+ let storage = device.storage_owned(data)?;+ let none = BackpropOp::none();+ Ok(from_storage(storage, shape, none, is_variable))+ }++ /// Creates a new tensor initialized with values from the input vector. The number of elements+ /// in this vector must be the same as the number of elements defined by the shape.+ /// If the device is cpu, no data copy is made.+ ///```rust+ /// use candle_core::{Tensor, Device};+ /// let a = Tensor::from_vec(vec!{1., 2., 3., 4., 5., 6.}, (2, 3), &Device::Cpu)?;+ ///+ /// assert_eq!(a.to_vec2::<f64>()?, &[+ /// [1., 2., 3.],+ /// [4., 5., 6.]+ /// ]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn from_vec<S: ShapeWithOneHole, D: crate::WithDType>(+ data: Vec<D>,+ shape: S,+ device: &Device,+ ) -> Result<Self> {+ Self::from_vec_impl(data, shape, device, false)+ }++ /// Creates a new tensor initialized with values from the input slice. The number of elements+ /// in this vector must be the same as the number of elements defined by the shape.+ ///```rust+ /// use candle_core::{Tensor, Device};+ /// let values = vec![1., 2., 3., 4., 5., 6., 7., 8.];+ /// let a = Tensor::from_slice(&values[1..7], (2, 3), &Device::Cpu)?;+ ///+ /// assert_eq!(a.to_vec2::<f64>()?, &[+ /// [2., 3., 4.],+ /// [5., 6., 7.]+ /// ]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn from_slice<S: ShapeWithOneHole, D: crate::WithDType>(+ array: &[D],+ shape: S,+ device: &Device,+ ) -> Result<Self> {+ let shape = shape.into_shape(array.len())?;+ let storage = device.storage_from_slice(array)?;+ let none = BackpropOp::none();+ Ok(from_storage(storage, shape, none, false))+ }++ pub(crate) fn same_shape_binary_op(&self, rhs: &Self, op: &'static str) -> Result<&Shape> {+ let lhs = self.shape();+ let rhs = rhs.shape();+ if lhs != rhs {+ Err(Error::ShapeMismatchBinaryOp {+ lhs: lhs.clone(),+ rhs: rhs.clone(),+ op,+ }+ .bt())+ } else {+ Ok(lhs)+ }+ }++ /// Returns true if the computation graph should track this op, that is if it is+ /// a variable or if it has some variable as dependencies.+ pub fn track_op(&self) -> bool {+ self.is_variable || self.op.is_some()+ }++ /// Creates a fresh tensor structure based on a storage and a shape.+ ///+ /// # Note+ /// - This uses contiguous strides+ /// - Ensure the shape is compatible with the shape of the storage.+ pub fn from_storage<S: Into<Shape>>(+ storage: Storage,+ shape: S,+ op: BackpropOp,+ is_variable: bool,+ ) -> Tensor {+ from_storage(storage, shape, op, is_variable)+ }++ // TODO: Also make an inplace version or a pre-allocated? This could be tricky+ // if this can create cycles in the compute graph.+ binary_op!(add, Add);+ binary_op!(mul, Mul);+ binary_op!(sub, Sub);+ binary_op!(div, Div);+ binary_op_scalar!(maximum, Maximum);+ binary_op_scalar!(minimum, Minimum);+ broadcast_binary_op!(broadcast_add, add);+ broadcast_binary_op!(broadcast_mul, mul);+ broadcast_binary_op!(broadcast_sub, sub);+ broadcast_binary_op!(broadcast_div, div);+ broadcast_binary_op!(broadcast_maximum, maximum);+ broadcast_binary_op!(broadcast_minimum, minimum);+ broadcast_binary_op!(broadcast_eq, eq);+ broadcast_binary_op!(broadcast_ne, ne);+ broadcast_binary_op!(broadcast_lt, lt);+ broadcast_binary_op!(broadcast_le, le);+ broadcast_binary_op!(broadcast_gt, gt);+ broadcast_binary_op!(broadcast_ge, ge);++ unary_op!(recip, Recip);+ unary_op!(neg, Neg);+ unary_op!(exp, Exp);+ unary_op!(log, Log);+ unary_op!(sin, Sin);+ unary_op!(cos, Cos);+ unary_op!(tanh, Tanh);+ unary_op!(abs, Abs);+ unary_op!(sqr, Sqr);+ unary_op!(sqrt, Sqrt);+ unary_op!(gelu, Gelu);+ unary_op!(gelu_erf, GeluErf);+ unary_op!(erf, Erf);+ unary_op!(relu, Relu);+ unary_op!(silu, Silu);+ unary_op!(ceil, Ceil);+ unary_op!(floor, Floor);+ unary_op!(round, Round);+ unary_op!(sign, Sign);++ /// Round element of the input tensor to the nearest integer.+ ///+ /// If the number of decimals is negative, it specifies the number of positions to the left of+ /// the decimal point.+ pub fn round_to(&self, decimals: i32) -> Result<Self> {+ let mult = 10f64.powi(decimals);+ (self * mult)?.round()? * (1f64 / mult)+ }++ /// Retrieves the single scalar value hold in the tensor. If the tensor contains multiple+ /// dimensions, an error is returned instead.+ pub fn to_scalar<S: crate::WithDType>(&self) -> Result<S> {+ if self.rank() != 0 {+ Err(Error::UnexpectedNumberOfDims {+ expected: 0,+ got: self.rank(),+ shape: self.shape().clone(),+ }+ .bt())?+ }+ let from_cpu_storage = |cpu_storage: &crate::CpuStorage| {+ let data = S::cpu_storage_as_slice(cpu_storage)?;+ Ok::<_, Error>(data[self.layout().start_offset()])+ };+ match &*self.storage() {+ Storage::Cpu(cpu_storage) => from_cpu_storage(cpu_storage),+ Storage::Cuda(storage) => from_cpu_storage(&storage.to_cpu_storage()?),+ Storage::Metal(storage) => from_cpu_storage(&storage.to_cpu_storage()?),+ }+ }++ /// An alias for `to_scalar`.+ pub fn to_vec0<S: crate::WithDType>(&self) -> Result<S> {+ self.to_scalar::<S>()+ }++ /// Repeat this tensor along the specified dimensions.+ pub fn repeat<S: Into<Shape>>(&self, shape: S) -> Result<Tensor> {+ // Similar to PyTorch, we extend the number of dimensions of self if needed.+ let repeats = shape.into();+ let repeats = repeats.dims();+ let mut inp = if self.rank() < repeats.len() {+ let shape = [vec![1; repeats.len() - self.rank()], self.dims().to_vec()].concat();+ self.reshape(shape)?+ } else {+ self.clone()+ };+ for (idx, &repeat) in repeats.iter().enumerate() {+ inp = match repeat {+ 0 => inp.narrow(idx, 0, 0)?,+ 1 => inp,+ repeat => Tensor::cat(&vec![&inp; repeat], idx)?,+ };+ }+ Ok(inp)+ }++ /// Creates grids of coordinates specified by the 1D inputs.+ ///+ /// # Arguments+ ///+ /// * `args` - A slice of 1D tensors.+ /// * `xy_indexing` - Whether to use xy indexing or ij indexing. If xy is selected, the+ /// first dimension corresponds to the cardinality of the second input and the second+ /// dimension corresponds to the cardinality of the first input. If ij is selected, the+ /// dimensions are in the same order as the cardinality of the inputs.+ ///+ /// # Examples+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device, Shape};+ /// let x = Tensor::new(&[1f32, 2., 3.], &Device::Cpu)?;+ /// let y = Tensor::new(&[4f32, 5., 6.], &Device::Cpu)?;+ ///+ /// let grids_xy = Tensor::meshgrid(&[&x, &y], true)?;+ ///+ /// assert_eq!(grids_xy.len(), 2);+ /// assert_eq!(grids_xy[0].dims(), &[3, 3]);+ ///+ /// assert_eq!(grids_xy[0].to_vec2::<f32>()?, &[[1., 2., 3.], [1., 2., 3.], [1., 2., 3.]]);+ /// assert_eq!(grids_xy[1].to_vec2::<f32>()?, &[[4., 4., 4.], [5., 5., 5.], [6., 6., 6.]]);+ ///+ /// let grids_ij = Tensor::meshgrid(&[&x, &y], false)?;+ ///+ /// assert_eq!(grids_ij[0].to_vec2::<f32>()?, &[[1., 1., 1.], [2., 2., 2.], [3., 3., 3.]]);+ /// assert_eq!(grids_ij[1].to_vec2::<f32>()?, &[[4., 5., 6.], [4., 5., 6.], [4., 5., 6.]]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ ///+ /// # Errors+ ///+ /// * Will return `Err` if `args` contains less than 2 tensors.+ ///+ pub fn meshgrid<A: AsRef<Tensor>>(args: &[A], xy_indexing: bool) -> Result<Vec<Self>> {+ if args.len() <= 1 {+ Err(Error::OpRequiresAtLeastTwoTensors { op: "meshgrid" }.bt())?+ }+ let args: Vec<_> = if xy_indexing {+ args.iter().rev().collect()+ } else {+ args.iter().collect()+ };++ let mut shape = Vec::with_capacity(args.len());+ for arg in args.iter() {+ shape.push(arg.as_ref().dims1()?)+ }++ let mut grids = Vec::with_capacity(args.len());+ for idx in 0..args.len() {+ let mut ones = vec![1usize; args.len()];+ ones[idx] = shape[idx];+ let arg = args[idx].as_ref().reshape(ones)?;+ let mut repeats = shape.clone();+ repeats[idx] = 1;+ let repeated_tensor = arg.repeat(repeats)?;+ grids.push(repeated_tensor);+ }+ if xy_indexing {+ grids.reverse();+ }+ Ok(grids)+ }++ /// This operation multiplies the input tensor by `mul` then adds `add` and return the result.+ /// The input values `mul` and `add` are casted to the appropriate type so some rounding might+ /// be performed.+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let a = Tensor::new(&[[0f32, 1.], [2., 3.]], &Device::Cpu)?;+ /// let a = a.affine(4., -2.)?;+ /// assert_eq!(a.to_vec2::<f32>()?, &[[-2.0, 2.0], [6.0, 10.0]]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn affine(&self, mul: f64, add: f64) -> Result<Self> {+ if self.elem_count() == 0 {+ return Ok(self.clone());+ }+ let storage = self.storage().affine(self.layout(), mul, add)?;+ let op = BackpropOp::new1(self, |arg| Op::Affine { arg, mul, add });+ Ok(from_storage(storage, self.shape(), op, false))+ }++ /// Applies the Exponential Linear Unit (ELU) function on each element of the input tensor.+ pub fn elu(&self, alpha: f64) -> Result<Self> {+ if self.elem_count() == 0 {+ return Ok(self.clone());+ }+ let storage = self.storage().elu(self.layout(), alpha)?;+ let op = BackpropOp::new1(self, |t| Op::Elu(t, alpha));+ Ok(from_storage(storage, self.shape(), op, false))+ }++ /// Raise the tensor to some float exponent `e`.+ pub fn powf(&self, e: f64) -> Result<Self> {+ if self.elem_count() == 0 {+ return Ok(self.clone());+ }+ let storage = self.storage().powf(self.layout(), e)?;+ let op = BackpropOp::new1(self, |t| Op::Powf(t, e));+ Ok(from_storage(storage, self.shape(), op, false))+ }++ pub(crate) fn check_dim(&self, dim: usize, op: &'static str) -> Result<()> {+ if dim >= self.dims().len() {+ Err(Error::DimOutOfRange {+ shape: self.shape().clone(),+ dim: dim as i32,+ op,+ }+ .bt())?+ } else {+ Ok(())+ }+ }++ /// Split a tensor into the specified number of chunks, this may return less chunks than+ /// specified.+ pub fn chunk<D: Dim>(&self, chunks: usize, dim: D) -> Result<Vec<Self>> {+ let dim = dim.to_index(self.shape(), "chunk")?;+ let size = self.dim(dim)?;+ if size < chunks {+ (0..size).map(|i| self.narrow(dim, i, 1)).collect()+ } else {+ let chunk_size = size / chunks;+ let cnt_additional = size % chunks;+ let mut tensors = vec![];+ let mut sum_chunk_size = 0;+ for i in 0..chunks {+ let chunk_size = if i < cnt_additional {+ chunk_size + 1+ } else {+ chunk_size+ };+ let tensor = self.narrow(dim, sum_chunk_size, chunk_size)?;+ tensors.push(tensor);+ sum_chunk_size += chunk_size+ }+ Ok(tensors)+ }+ }++ /// Returns a new tensor that is a narrowed version of the input, the dimension `dim`+ /// ranges from `start` to `start + len`.+ /// ```+ /// use candle_core::{Tensor, Device};+ /// let a = Tensor::new(&[+ /// [0f32, 1., 2.],+ /// [3. , 4., 5.],+ /// [6. , 7., 8.]+ /// ], &Device::Cpu)?;+ ///+ /// let b = a.narrow(0, 1, 2)?;+ /// assert_eq!(b.shape().dims(), &[2, 3]);+ /// assert_eq!(b.to_vec2::<f32>()?, &[+ /// [3., 4., 5.],+ /// [6., 7., 8.]+ /// ]);+ ///+ /// let c = a.narrow(1, 1, 1)?;+ /// assert_eq!(c.shape().dims(), &[3, 1]);+ /// assert_eq!(c.to_vec2::<f32>()?, &[+ /// [1.],+ /// [4.],+ /// [7.]+ /// ]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn narrow<D: Dim>(&self, dim: D, start: usize, len: usize) -> Result<Self> {+ let dims = self.dims();+ let dim = dim.to_index(self.shape(), "narrow")?;+ let err = |msg| {+ Err::<(), _>(+ Error::NarrowInvalidArgs {+ shape: self.shape().clone(),+ dim,+ start,+ len,+ msg,+ }+ .bt(),+ )+ };+ if start > dims[dim] {+ err("start > dim_len")?+ }+ if start.saturating_add(len) > dims[dim] {+ err("start + len > dim_len")?+ }+ if start == 0 && dims[dim] == len {+ Ok(self.clone())+ } else {+ let op = BackpropOp::new1(self, |t| Op::Narrow(t, dim, start, len));+ let layout = self.layout().narrow(dim, start, len)?;+ let tensor_ = Tensor_ {+ id: TensorId::new(),+ storage: self.storage.clone(),+ layout,+ op,+ is_variable: false,+ dtype: self.dtype,+ device: self.device.clone(),+ };+ Ok(Tensor(Arc::new(tensor_)))+ }+ }++ fn squeeze_dims(self, dims: &[usize]) -> Result<Self> {+ match dims {+ [] => Ok(self),+ [i] => self.squeeze(*i),+ dims => {+ let dims = self+ .dims()+ .iter()+ .enumerate()+ .filter_map(|(dim_idx, &v)| {+ if dims.contains(&dim_idx) {+ None+ } else {+ Some(v)+ }+ })+ .collect::<Vec<_>>();+ self.reshape(dims)+ }+ }+ }++ fn reduce_impl<D: Dim>(&self, dim: D, keepdim: bool, op: ReduceOp) -> Result<Self> {+ let dim = dim.to_index(self.shape(), op.name())?;+ let storage = self.storage().reduce_op(op, self.layout(), &[dim])?;+ let mut dims = self.dims().to_vec();+ dims[dim] = 1;+ let op = match op {+ ReduceOp::Sum | ReduceOp::Min | ReduceOp::Max => {+ BackpropOp::new1(self, |arg| Op::Reduce(arg, op, dims.to_vec()))+ }+ ReduceOp::ArgMin | ReduceOp::ArgMax => BackpropOp::none(),+ };+ let res = from_storage(storage, dims, op, false);+ if keepdim {+ Ok(res)+ } else {+ res.squeeze_dims(&[dim])+ }+ }++ fn sum_impl<D: Dims>(&self, sum_dims: D, keepdim: bool) -> Result<Self> {+ let sum_dims = sum_dims.to_indexes(self.shape(), "sum")?;+ let storage = self+ .storage()+ .reduce_op(ReduceOp::Sum, self.layout(), &sum_dims)?;+ let mut dims = self.dims().to_vec();+ for &sum_dim in sum_dims.iter() {+ dims[sum_dim] = 1+ }+ let op = BackpropOp::new1(self, |a| Op::Reduce(a, ReduceOp::Sum, dims.to_vec()));+ let sum = from_storage(storage, dims, op, false);+ if keepdim {+ Ok(sum)+ } else {+ sum.squeeze_dims(&sum_dims)+ }+ }++ /// Roll the tensor input along the given dimension.+ /// Elements that are shifted beyond the last position are re-introduced at the first position.+ ///+ /// ```rust+ /// # use candle_core::{Tensor, Device};+ /// let tensor = Tensor::new(&[[0f32, 1.], [2., 3.], [4., 5.]], &Device::Cpu)?;+ /// let tensor = tensor.roll(1, 0)?;+ /// assert_eq!(tensor.to_vec2::<f32>()?, &[[4., 5.], [0., 1.], [2., 3.]]);+ /// let tensor = Tensor::new(&[[0f32, 1.], [2., 3.], [4., 5.]], &Device::Cpu)?;+ /// let tensor = tensor.roll(-1, 0)?;+ /// assert_eq!(tensor.to_vec2::<f32>()?, &[[2., 3.], [4., 5.], [0., 1.]]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn roll<D>(&self, shift: i32, dim: D) -> Result<Self>+ where+ D: Dim + Clone,+ {+ let dim = dim.to_index(self.shape(), "roll")?;+ let dim_size = self.dim(dim)?;+ let shift = shift.rem_euclid(dim_size as i32) as usize;+ if shift == 0 {+ Ok(self.clone())+ } else {+ let a = self.narrow(dim, 0, dim_size - shift)?;+ let b = self.narrow(dim, dim_size - shift, shift)?;+ Tensor::cat(&[&b, &a], dim)+ }+ }++ /// Returns the sum of all elements in the input tensor. The sum is performed over all the+ /// input dimensions.+ ///+ /// The resulting tensor has a shape that is similar to the shape of the input tensor, except+ /// that the number of elements for each dimension index in `sum_dims` is 1.+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let a = Tensor::new(&[[0f32, 1.], [2., 3.]], &Device::Cpu)?;+ /// let s = a.sum_keepdim(0)?;+ /// assert_eq!(s.to_vec2::<f32>()?, &[[2., 4.]]);+ /// let s = a.sum_keepdim(1)?;+ /// assert_eq!(s.to_vec2::<f32>()?, &[[1.], [5.]]);+ /// let s = a.sum_keepdim((0, 1))?;+ /// assert_eq!(s.to_vec2::<f32>()?, &[[6.]]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn sum_keepdim<D: Dims>(&self, sum_dims: D) -> Result<Self> {+ self.sum_impl(sum_dims, true)+ }++ /// Returns the sum of all elements in the input tensor. The sum is performed over all the+ /// input dimensions and compared to `sum_keepdim` these dimensions are squeezed rather than+ /// kept.+ pub fn sum<D: Dims>(&self, sum_dims: D) -> Result<Self> {+ self.sum_impl(sum_dims, false)+ }++ /// Returns the mean of all elements in the input tensor. The mean is performed over all the+ /// input dimensions.+ ///+ /// The resulting tensor has a shape that is similar to the shape of the input tensor, except+ /// that the number of elements for each dimension index in `mean_dims` is 1.+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let a = Tensor::new(&[[0f32, 1.], [2., 3.]], &Device::Cpu)?;+ /// let s = a.mean_keepdim(0)?;+ /// assert_eq!(s.to_vec2::<f32>()?, &[[1., 2.]]);+ /// let s = a.mean_keepdim(1)?;+ /// assert_eq!(s.to_vec2::<f32>()?, &[[0.5], [2.5]]);+ /// let s = a.mean_keepdim((0, 1))?;+ /// assert_eq!(s.to_vec2::<f32>()?, &[[1.5]]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn mean_keepdim<D: Dims>(&self, mean_dims: D) -> Result<Self> {+ let mean_dims = mean_dims.to_indexes(self.shape(), "mean-keepdim")?;+ let reduced_dim: usize = mean_dims.iter().map(|i| self.dims()[*i]).product();+ let scale = 1f64 / (reduced_dim as f64);+ self.sum_impl(mean_dims, true)? * scale+ }++ /// Returns the mean of all elements in the input tensor. The mean is performed over all the+ /// input dimensions and compared to `mean_keepdim` these dimensions are squeezed rather than+ /// kept.+ pub fn mean<D: Dims>(&self, mean_dims: D) -> Result<Self> {+ let mean_dims = mean_dims.to_indexes(self.shape(), "mean")?;+ let reduced_dim: usize = mean_dims.iter().map(|i| self.dims()[*i]).product();+ let scale = 1f64 / (reduced_dim as f64);+ self.sum_impl(mean_dims, false)? * scale+ }++ /// Returns the unbiased variance over the selected dimension.+ pub fn var_keepdim<D: Dim>(&self, dim: D) -> Result<Self> {+ let dim = dim.to_index(self.shape(), "var")?;+ let mean = self.mean_keepdim(dim)?;+ let squares = self.broadcast_sub(&mean)?.sqr()?;+ squares.sum_impl(dim, true)? / (self.dim(dim)? - 1) as f64+ }++ /// Returns the unbiased variance over the selected dimension.+ pub fn var<D: Dim>(&self, dim: D) -> Result<Self> {+ let dim = dim.to_index(self.shape(), "var")?;+ self.var_keepdim(dim)?.squeeze(dim)+ }++ /// Gathers the maximum value across the selected dimension. The resulting shape has the same+ /// number of dimensions as the original tensor and the select dimension has a single element.+ pub fn max_keepdim<D: Dim>(&self, dim: D) -> Result<Self> {+ self.reduce_impl(dim, true, ReduceOp::Max)+ }++ /// Similar to `max_keepdim` but the target dimension is squeezed.+ pub fn max<D: Dim>(&self, dim: D) -> Result<Self> {+ self.reduce_impl(dim, false, ReduceOp::Max)+ }++ /// Gathers the minimum value across the selected dimension. The resulting shape has the same+ /// number of dimensions as the original tensor and the select dimension has a single element.+ pub fn min_keepdim<D: Dim>(&self, dim: D) -> Result<Self> {+ self.reduce_impl(dim, true, ReduceOp::Min)+ }++ /// Similar to `min_keepdim` but the target dimension is squeezed.+ pub fn min<D: Dim>(&self, dim: D) -> Result<Self> {+ self.reduce_impl(dim, false, ReduceOp::Min)+ }++ pub fn argmax_keepdim<D: Dim>(&self, dim: D) -> Result<Self> {+ self.reduce_impl(dim, true, ReduceOp::ArgMax)+ }++ /// Similar to `argmax_keepdim` but the target dimension is squeezed.+ pub fn argmax<D: Dim>(&self, dim: D) -> Result<Self> {+ self.reduce_impl(dim, false, ReduceOp::ArgMax)+ }++ pub fn argmin_keepdim<D: Dim>(&self, dim: D) -> Result<Self> {+ self.reduce_impl(dim, true, ReduceOp::ArgMin)+ }++ /// Similar to `argmin_keepdim` but the target dimension is squeezed.+ pub fn argmin<D: Dim>(&self, dim: D) -> Result<Self> {+ self.reduce_impl(dim, false, ReduceOp::ArgMin)+ }++ /// Element-wise comparison between two tensors, e.g. equality, greater than, ... The actual+ /// comparison operation is specified by the `op` argument.+ ///+ /// The returned tensor has the same shape as the original tensors and uses `u8` elements.+ pub fn cmp<T: TensorOrScalar>(&self, rhs: T, op: CmpOp) -> Result<Self> {+ let rhs = match rhs.to_tensor_scalar()? {+ crate::scalar::TensorScalar::Tensor(rhs) => rhs,+ crate::scalar::TensorScalar::Scalar(rhs) => rhs+ .to_dtype(self.dtype())?+ .to_device(self.device())?+ .broadcast_as(self.shape())?,+ };+ let shape = self.same_shape_binary_op(&rhs, "cmp")?;+ let storage = self+ .storage()+ .cmp(op, &rhs.storage(), self.layout(), rhs.layout())?;+ let op = BackpropOp::new1(self, |a| Op::Cmp(a, op));+ Ok(from_storage(storage, shape.dims(), op, false))+ }++ /// Element-wise equality.+ pub fn eq<T: TensorOrScalar>(&self, rhs: T) -> Result<Self> {+ self.cmp(rhs, CmpOp::Eq)+ }++ /// Element-wise non-equality.+ pub fn ne<T: TensorOrScalar>(&self, rhs: T) -> Result<Self> {+ self.cmp(rhs, CmpOp::Ne)+ }++ /// Element-wise comparison with lower-than, the returned tensor uses value 1 where `self <+ /// rhs` and 0 otherwise.+ pub fn lt<T: TensorOrScalar>(&self, rhs: T) -> Result<Self> {+ self.cmp(rhs, CmpOp::Lt)+ }++ /// Element-wise comparison with greater-than, the returned tensor uses value 1 where `self >+ /// rhs` and 0 otherwise.+ pub fn gt<T: TensorOrScalar>(&self, rhs: T) -> Result<Self> {+ self.cmp(rhs, CmpOp::Gt)+ }++ /// Element-wise comparison with greater-equal, the returned tensor uses value 1 where `self >=+ /// rhs` and 0 otherwise.+ pub fn ge<T: TensorOrScalar>(&self, rhs: T) -> Result<Self> {+ self.cmp(rhs, CmpOp::Ge)+ }++ /// Element-wise comparison with lower-equal, the returned tensor uses value 1 where `self <=+ /// rhs` and 0 otherwise.+ pub fn le<T: TensorOrScalar>(&self, rhs: T) -> Result<Self> {+ self.cmp(rhs, CmpOp::Le)+ }++ /// Clamp the tensor values to be between `min` and `max`.+ pub fn clamp<T1: TensorOrScalar, T2: TensorOrScalar>(&self, min: T1, max: T2) -> Result<Self> {+ self.maximum(min)?.minimum(max)+ }++ /// Interpolate the input tensor to the `target_size` size, taking the value of the nearest element.+ ///+ /// The input tensor should have three dimensions, `(batch, channels, l)`, the returned+ /// tensor also has three dimensions, `(batch, channels, target_size)`.+ pub fn interpolate1d(&self, target_size: usize) -> Result<Self> {+ let (n, c, _l) = self.dims3()?;+ let op = BackpropOp::new1(self, |arg| Op::UpsampleNearest1D { arg, target_size });+ let storage = self+ .storage()+ .upsample_nearest1d(self.layout(), target_size)?;+ Ok(from_storage(storage, (n, c, target_size), op, false))+ }++ /// Alias for `interpolate1d`.+ pub fn upsample_nearest1d(&self, target_size: usize) -> Result<Self> {+ self.interpolate1d(target_size)+ }++ /// Interpolate the input tensor to the `(target_h, target_w)` size, taking the value of the+ /// nearest element.+ ///+ /// The input tensor should have four dimensions, `(batch, channels, h, w)`, the returned+ /// tensor also has four dimensions, `(batch, channels, target_h, target_w)`.+ pub fn interpolate2d(&self, target_h: usize, target_w: usize) -> Result<Self> {+ let (n, c, _h, _w) = self.dims4()?;+ let op = BackpropOp::new1(self, |arg| Op::UpsampleNearest2D {+ arg,+ target_h,+ target_w,+ });+ let storage = self+ .storage()+ .upsample_nearest2d(self.layout(), target_h, target_w)?;+ Ok(from_storage(storage, (n, c, target_h, target_w), op, false))+ }++ /// Alias for `interpolate2d`.+ pub fn upsample_nearest2d(&self, target_h: usize, target_w: usize) -> Result<Self> {+ self.interpolate2d(target_h, target_w)+ }++ /// Bilinear interpolation to resize the input tensor to the specified size.+ ///+ /// The input tensor should have four dimensions: `(batch, channels, h, w)`.+ /// The returned tensor also has four dimensions: `(batch, channels, target_h, target_w)`.+ ///+ /// # Arguments+ ///+ /// * `target_h` - Target height+ /// * `target_w` - Target width+ /// * `align_corners` - If true, corner pixels are aligned. If false (default),+ /// pixels are treated as areas (matches PyTorch default behavior).+ ///+ /// # Example+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// # fn main() -> candle_core::Result<()> {+ /// let t = Tensor::arange(0f32, 16f32, &Device::Cpu)?.reshape((1, 1, 4, 4))?;+ /// let upsampled = t.upsample_bilinear2d(8, 8, false)?;+ /// assert_eq!(upsampled.dims(), &[1, 1, 8, 8]);+ /// # Ok(())+ /// # }+ /// ```+ pub fn upsample_bilinear2d(+ &self,+ target_h: usize,+ target_w: usize,+ align_corners: bool,+ ) -> Result<Self> {+ let (n, c, _h, _w) = self.dims4()?;+ let op = BackpropOp::new1(self, |arg| Op::UpsampleBilinear2D {+ arg,+ target_h,+ target_w,+ align_corners,+ });+ // Pass None for scale factors (size mode)+ let storage = self.storage().upsample_bilinear2d(+ self.layout(),+ target_h,+ target_w,+ align_corners,+ None,+ None,+ )?;+ Ok(from_storage(storage, (n, c, target_h, target_w), op, false))+ }++ /// Bilinear interpolation using scale factors.+ ///+ /// Similar to `upsample_bilinear2d` but uses scale factors instead of absolute sizes.+ /// This matches PyTorch's `interpolate(scale_factor=...)` behavior.+ ///+ /// # Arguments+ ///+ /// * `scale_h` - Height scaling factor+ /// * `scale_w` - Width scaling factor+ /// * `align_corners` - If true, corner pixels are aligned+ ///+ /// # Example+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// # fn main() -> candle_core::Result<()> {+ /// let t = Tensor::arange(0f32, 16f32, &Device::Cpu)?.reshape((1, 1, 4, 4))?;+ /// // Scale by 2x in both dimensions+ /// let upsampled = t.upsample_bilinear2d_with_scale(2.0, 2.0, false)?;+ /// assert_eq!(upsampled.dims(), &[1, 1, 8, 8]);+ /// # Ok(())+ /// # }+ /// ```+ pub fn upsample_bilinear2d_with_scale(+ &self,+ scale_h: f64,+ scale_w: f64,+ align_corners: bool,+ ) -> Result<Self> {+ let (n, c, height_in, width_in) = self.dims4()?;++ // Calculate output size (floor, matching PyTorch)+ let height_out = (height_in as f64 * scale_h).floor() as usize;+ let width_out = (width_in as f64 * scale_w).floor() as usize;++ // Early return if size unchanged+ if height_in == height_out && width_in == width_out {+ return Ok(self.clone());+ }++ let op = BackpropOp::new1(self, |arg| Op::UpsampleBilinear2D {+ arg,+ target_h: height_out,+ target_w: width_out,+ align_corners,+ });++ // Pass original scale factors (scale_factor mode)+ // This ensures PyTorch-compatible scale calculation+ let storage = self.storage().upsample_bilinear2d(+ self.layout(),+ height_out,+ width_out,+ align_corners,+ Some(scale_h),+ Some(scale_w),+ )?;+ Ok(from_storage(+ storage,+ (n, c, height_out, width_out),+ op,+ false,+ ))+ }++ /// 2D average pooling over an input tensor with multiple channels.+ ///+ /// The input tensor should have four dimensions, `(batch, channels, h, w)`, the returned+ /// tensor also has four dimensions, `(batch, channels, h', w')`. The pooling is performed on+ /// the two last dimensions using a kernel of size `sz`. The returned element is the average+ /// value over the kernel window.+ pub fn avg_pool2d<T: crate::ToUsize2>(&self, sz: T) -> Result<Self> {+ let sz = sz.to_usize2();+ self.avg_pool2d_with_stride(sz, sz)+ }++ /// Same as `avg_pool2d` but with a `stride` that can be set to a value different from the+ /// kernel size.+ pub fn avg_pool2d_with_stride<T: crate::ToUsize2>(+ &self,+ kernel_size: T,+ stride: T,+ ) -> Result<Self> {+ let kernel_size = kernel_size.to_usize2();+ let stride = stride.to_usize2();+ let (n, c, h, w) = self.dims4()?;+ if h < kernel_size.0 || w < kernel_size.1 {+ bail!("kernel-size {kernel_size:?} is larger than the input size {h},{w}")+ }+ // https://pytorch.org/docs/stable/generated/torch.nn.AvgPool2d.html#torch.nn.AvgPool2d+ let h_out = (h - kernel_size.0) / stride.0 + 1;+ let w_out = (w - kernel_size.1) / stride.1 + 1;+ let op = BackpropOp::new1(self, |arg| Op::AvgPool2D {+ arg,+ kernel_size,+ stride,+ });+ let storage = self+ .storage()+ .avg_pool2d(self.layout(), kernel_size, stride)?;+ Ok(from_storage(storage, (n, c, h_out, w_out), op, false))+ }++ /// 2D max pooling over an input tensor with multiple channels.+ ///+ /// The input tensor should have four dimensions, `(batch, channels, h, w)`, the returned+ /// tensor also has four dimensions, `(batch, channels, h', w')`. The pooling is performed on+ /// the two last dimensions using a kernel of size `sz`, the returned element is the maximum+ /// value over the kernel window.+ pub fn max_pool2d<T: crate::ToUsize2>(&self, sz: T) -> Result<Self> {+ let sz = sz.to_usize2();+ self.max_pool2d_with_stride(sz, sz)+ }++ /// Same as `max_pool2d` but with a `stride` that can be set to a value different from the+ /// kernel size.+ pub fn max_pool2d_with_stride<T: crate::ToUsize2>(+ &self,+ kernel_size: T,+ stride: T,+ ) -> Result<Self> {+ let kernel_size = kernel_size.to_usize2();+ let stride = stride.to_usize2();+ let (n, c, h, w) = self.dims4()?;+ if h < kernel_size.0 || w < kernel_size.1 {+ bail!("kernel-size {kernel_size:?} is larger than the input size {h},{w}")+ }+ // https://pytorch.org/docs/stable/generated/torch.nn.MaxPool2d.html#torch.nn.MaxPool2d+ let h_out = (h - kernel_size.0) / stride.0 + 1;+ let w_out = (w - kernel_size.1) / stride.1 + 1;+ let op = BackpropOp::new1(self, |arg| Op::MaxPool2D {+ arg,+ kernel_size,+ stride,+ });+ let storage = self+ .storage()+ .max_pool2d(self.layout(), kernel_size, stride)?;+ Ok(from_storage(storage, (n, c, h_out, w_out), op, false))+ }++ /// Computes the dot product of two 1D tensors.+ ///+ /// - If inputs are 1D vectors (`[n]`), returns their scalar dot product.+ /// - Panics if shapes are not compatible+ /// - Not supported for integer dtypes+ ///+ /// # Example (vectors)+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let t1 = Tensor::new(&[1.0, 2.0, 3.0], &Device::Cpu)?;+ /// let t2 = Tensor::new(&[4.0, 5.0, 6.0], &Device::Cpu)?;+ /// let res = t1.dot(&t2)?;+ /// assert_eq!(res.to_scalar::<f64>()?, 32.);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn dot(&self, rhs: &Self) -> Result<Self> {+ if self.dims().len() != 1 || rhs.dims().len() != 1 {+ return Err(Error::ShapeMismatchBinaryOp {+ lhs: self.shape().clone(),+ rhs: rhs.shape().clone(),+ op: "dot",+ });+ }++ (self * rhs).and_then(|ret| ret.sum_all())+ }++ /// Computes the **Frobenius norm** (L2 norm of all elements) of the tensor.+ /// - Output is `sqrt(sum(x^2))`.+ /// - Always returns a scalar (`[]` shape).+ ///+ /// # Example+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let t = Tensor::new(&[[3., 4.], [0., 0.]], &Device::Cpu)?;+ /// let norm = t.norm()?;+ /// assert_eq!(norm.to_scalar::<f64>()?, 5.);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn norm(&self) -> Result<Self> {+ if self.dtype().is_int() {+ bail!("norm not supported for integer dtypes");+ }++ self.sqr().and_then(|x| x.sum_all()).and_then(|x| x.sqrt())+ }++ /// Performs strict matrix-vector multiplication (`[m, n] * [n] = [m]`).+ ///+ /// - If `self` is a matrix (`[m, n]`) and `rhs` is a vector (`[n]`), returns a vector (`[m]`).+ /// - **No broadcasting**: Panics if `self` is not 2D or if `rhs` is not 1D with matching size.+ ///+ /// # Example+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let mat = Tensor::new(&[[1., 2., 3.], [4., 5., 6.]], &Device::Cpu)?;+ /// let vec = Tensor::new(&[1., 1., 1.], &Device::Cpu)?;+ /// let res = mat.mv(&vec)?;+ /// assert_eq!(res.to_vec1::<f64>()?, [6., 15.]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn mv(&self, rhs: &Self) -> Result<Self> {+ // Strict shape checks+ let lhs_dims = self.dims();+ let rhs_dims = rhs.dims();+ if lhs_dims.len() != 2 || rhs_dims.len() != 1 || lhs_dims[1] != rhs_dims[0] {+ return Err(Error::ShapeMismatchBinaryOp {+ lhs: self.shape().clone(),+ rhs: rhs.shape().clone(),+ op: "mv",+ });+ }++ // Direct matmul after ensuring rhs is column vector+ self.matmul(&rhs.unsqueeze(1)?)?.squeeze(1)+ }++ /// Returns the matrix-multiplication of the input tensor with the other provided tensor.+ ///+ /// # Arguments+ ///+ /// * `self` - A tensor with dimensions `b1, b2, ..., bi, m, k`.+ /// * `rhs` - A tensor with dimensions `b1, b2, ..., bi, k, n`.+ ///+ /// The resulting tensor has dimensions `b1, b2, ..., bi, m, n`.+ pub fn matmul(&self, rhs: &Self) -> Result<Self> {+ let a_dims = self.shape().dims();+ let b_dims = rhs.shape().dims();++ let dim = a_dims.len();++ if dim < 2 || b_dims.len() != dim {+ Err(Error::ShapeMismatchBinaryOp {+ lhs: self.shape().clone(),+ rhs: rhs.shape().clone(),+ op: "matmul",+ }+ .bt())?+ }++ let m = a_dims[dim - 2];+ let k = a_dims[dim - 1];+ let k2 = b_dims[dim - 2];+ let n = b_dims[dim - 1];++ let c_shape = Shape::from(&a_dims[..dim - 2]).extend(&[m, n]);+ let batching: usize = a_dims[..dim - 2].iter().product();+ let batching_b: usize = b_dims[..dim - 2].iter().product();+ if k != k2 || batching != batching_b {+ Err(Error::ShapeMismatchBinaryOp {+ lhs: self.shape().clone(),+ rhs: rhs.shape().clone(),+ op: "matmul",+ }+ .bt())?+ }+ if c_shape.elem_count() == 0 || k == 0 {+ {+ let lhs_storage = self.storage();+ let rhs_storage = rhs.storage();+ lhs_storage.same_device(&rhs_storage, "matmul")?;+ lhs_storage.same_dtype(&rhs_storage, "matmul")?;+ }++ let storage = self.device().zeros(&c_shape, self.dtype())?;+ let op = BackpropOp::new2(self, rhs, Op::Matmul);+ return Ok(from_storage(storage, c_shape, op, false));+ }++ let storage = self.storage().matmul(+ &rhs.storage(),+ (batching, m, n, k),+ self.layout(),+ rhs.layout(),+ )?;+ let op = BackpropOp::new2(self, rhs, Op::Matmul);+ Ok(from_storage(storage, c_shape, op, false))+ }++ /// Matrix-multiplication with broadcasting support.+ ///+ /// Compared to `matmul` the two matrixes are allowed to have different dimensions as long as+ /// they are compatible for broadcast. E.g. if `self` has shape `(j, 1, n, k)` and `rhs` has+ /// shape `(l, k, m)`, the output will have shape `(j, l, n, m)`.+ pub fn broadcast_matmul(&self, rhs: &Self) -> Result<Self> {+ let lhs = self;+ let (l_shape, r_shape) = lhs.shape().broadcast_shape_matmul(rhs.shape())?;+ let l_broadcast = l_shape != *lhs.shape();+ let r_broadcast = r_shape != *rhs.shape();+ // TODO: Avoid concretising the broadcasted matrixes via contiguous.+ match (l_broadcast, r_broadcast) {+ (true, true) => lhs+ .broadcast_as(&l_shape)?+ .contiguous()?+ .matmul(&rhs.broadcast_as(&r_shape)?.contiguous()?),+ // A rank-2 rhs is only broadcast over the batch dimensions, so the whole product is+ // a single 2D matmul once the leading dims of lhs are folded into its row dimension.+ // Broadcasting the rhs instead would copy it `batch` times -- for an lm_head that is+ // the entire vocabulary matrix, per call. Same trick, and same contiguity guard, as+ // `candle_nn::Linear::forward`.+ (false, true) if rhs.rank() == 2 && lhs.is_contiguous() => {+ let (lhs_dims, rhs_dims) = (lhs.dims(), rhs.dims());+ let (m, k) = (lhs_dims[lhs.rank() - 2], lhs_dims[lhs.rank() - 1]);+ let n = rhs_dims[1];+ let batch: usize = lhs_dims[..lhs.rank() - 2].iter().product();+ let mut out_dims = lhs_dims.to_vec();+ out_dims.pop();+ out_dims.push(n);+ lhs.reshape((batch * m, k))?.matmul(rhs)?.reshape(out_dims)+ }+ (false, true) => lhs.matmul(&rhs.broadcast_as(&r_shape)?.contiguous()?),+ (true, false) => lhs.broadcast_as(&l_shape)?.contiguous()?.matmul(rhs),+ (false, false) => lhs.matmul(rhs),+ }+ }++ /// Returns a tensor with the same shape as the input tensor, the values are taken from+ /// `on_true` if the input tensor value is not zero, and `on_false` at the positions where the+ /// input tensor is equal to zero.+ pub fn where_cond(&self, on_true: &Self, on_false: &Self) -> Result<Self> {+ let _shap = self.same_shape_binary_op(on_true, "where_cond")?;+ let shape = self.same_shape_binary_op(on_false, "where_cond")?;+ let storage = self.storage().where_cond(+ self.layout(),+ &on_true.storage(),+ on_true.layout(),+ &on_false.storage(),+ on_false.layout(),+ )?;+ let op = BackpropOp::new3(self, on_true, on_false, Op::WhereCond);+ Ok(from_storage(storage, shape, op, false))+ }++ /// Returns a tensor with the values from the `self` tensor at the index corresponding to the+ /// values hold in the `ids` tensor.+ ///+ /// # Arguments+ ///+ /// * `self` - A tensor with dimensions `v, h`.+ /// * `ids` - A tensor with dimensions `s` and with integer values between 0 and v (exclusive).+ ///+ /// The resulting tensor has dimensions `s, h`. `s` is called the sequence length, `v` the+ /// vocabulary size, and `h` the hidden size.+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let values = Tensor::new(&[[0f32, 1.], [2., 3.], [4., 5.]], &Device::Cpu)?;+ /// let ids = Tensor::new(&[2u32, 1u32, 2u32], &Device::Cpu)?;+ /// let emb = values.embedding(&ids)?;+ /// assert_eq!(emb.to_vec2::<f32>()?, &[[4., 5.], [2., 3.], [4., 5.]]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn embedding(&self, ids: &Self) -> Result<Self> {+ if self.rank() != 2 || ids.rank() != 1 {+ Err(Error::ShapeMismatchBinaryOp {+ lhs: self.shape().clone(),+ rhs: ids.shape().clone(),+ op: "embedding",+ }+ .bt())?+ }+ self.index_select(ids, 0)+ }++ fn scatter_checks(&self, indexes: &Self, source: &Self, dim: usize) -> Result<()> {+ let source_dims = source.dims();+ let self_dims = self.dims();+ let mismatch = if source_dims.len() != self_dims.len() {+ true+ } else {+ let mut mismatch = false;+ for (i, (&d1, &d2)) in self_dims.iter().zip(source_dims.iter()).enumerate() {+ if i != dim && d1 != d2 {+ mismatch = true;+ break;+ }+ }+ mismatch+ };+ if mismatch {+ Err(Error::ShapeMismatchBinaryOp {+ op: "scatter (self, src)",+ lhs: self.shape().clone(),+ rhs: source.shape().clone(),+ }+ .bt())?+ }+ if indexes.dims() != source.dims() {+ Err(Error::ShapeMismatchBinaryOp {+ op: "scatter (indexes, src)",+ lhs: indexes.shape().clone(),+ rhs: source.shape().clone(),+ }+ .bt())?+ }+ Ok(())+ }++ pub fn scatter<D: Dim>(&self, indexes: &Self, source: &Self, dim: D) -> Result<Self> {+ let dim = dim.to_index(self.shape(), "scatter")?;+ self.scatter_checks(indexes, source, dim)?;+ let shape = self.shape();+ let mut storage = unsafe { self.device().alloc_uninit(shape, self.dtype())? };+ self.storage()+ .copy_strided_src(&mut storage, 0, self.layout())?;+ let layout = Layout::contiguous(shape);+ storage.scatter_set(+ &layout,+ &indexes.storage(),+ indexes.layout(),+ &source.storage(),+ source.layout(),+ dim,+ )?;+ let op = BackpropOp::new3(self, indexes, source, |t1, t2, t3| {+ Op::Scatter(t1, t2, t3, dim)+ });+ Ok(from_storage(storage, self.shape(), op, false))+ }++ pub fn scatter_set<D: Dim>(&self, indexes: &Self, source: &Self, dim: D) -> Result<()> {+ if self.same_storage(source) {+ crate::bail!("cannot use slice_set when self and src share their storage")+ }+ let dim = dim.to_index(self.shape(), "scatter-set")?;+ self.scatter_checks(indexes, source, dim)?;+ self.storage_mut().scatter_set(+ self.layout(),+ &indexes.storage(),+ indexes.layout(),+ &source.storage(),+ source.layout(),+ dim,+ )?;+ Ok(())+ }++ pub fn scatter_add<D: Dim>(&self, indexes: &Self, source: &Self, dim: D) -> Result<Self> {+ let dim = dim.to_index(self.shape(), "scatter-add")?;+ self.scatter_checks(indexes, source, dim)?;+ let shape = self.shape();+ let mut storage = unsafe { self.device().alloc_uninit(shape, self.dtype())? };+ self.storage()+ .copy_strided_src(&mut storage, 0, self.layout())?;+ let layout = Layout::contiguous(shape);+ storage.scatter_add(+ &layout,+ &indexes.storage(),+ indexes.layout(),+ &source.storage(),+ source.layout(),+ dim,+ )?;+ let op = BackpropOp::new3(self, indexes, source, |t1, t2, t3| {+ Op::ScatterAdd(t1, t2, t3, dim)+ });+ Ok(from_storage(storage, self.shape(), op, false))+ }++ pub fn scatter_add_set<D: Dim>(&self, indexes: &Self, source: &Self, dim: D) -> Result<()> {+ if self.same_storage(source) {+ crate::bail!("cannot use slice_set when self and src share their storage")+ }+ let dim = dim.to_index(self.shape(), "scatter-add-set")?;+ self.scatter_checks(indexes, source, dim)?;+ self.storage_mut().scatter_add(+ self.layout(),+ &indexes.storage(),+ indexes.layout(),+ &source.storage(),+ source.layout(),+ dim,+ )?;+ Ok(())+ }++ /// Embeds the values of the `src` tensor into the `self` tensor on the specified dimension.+ pub fn slice_scatter<D: Dim>(&self, src: &Self, dim: D, start: usize) -> Result<Self> {+ let dim = dim.to_index(self.shape(), "slice-scatter")?;+ if dim == 0 {+ self.slice_scatter0(src, start)+ } else {+ // TODO: Maybe we want to add a more efficient implementation at some point.+ self.transpose(0, dim)?+ .slice_scatter0(&src.transpose(0, dim)?, start)?+ .transpose(0, dim)+ }+ }++ /// Embeds the values of the `src` tensor into the `self` tensor on the first dimension.+ pub fn slice_scatter0(&self, src: &Self, start: usize) -> Result<Self> {+ if self.dtype() != src.dtype() {+ Err(Error::DTypeMismatchBinaryOp {+ lhs: self.dtype(),+ rhs: src.dtype(),+ op: "slice-scatter",+ }+ .bt())?+ }+ if self.device().location() != src.device.location() {+ Err(Error::DeviceMismatchBinaryOp {+ lhs: self.device().location(),+ rhs: src.device().location(),+ op: "slice-scatter",+ }+ .bt())?+ }+ if self.rank() != src.rank() {+ Err(Error::UnexpectedNumberOfDims {+ expected: self.rank(),+ got: src.rank(),+ shape: src.shape().clone(),+ }+ .bt())?+ }+ let shape_ok =+ self.dims()+ .iter()+ .zip(src.dims().iter())+ .enumerate()+ .all(|(dim_idx, (&d1, &d2))| {+ if 0 == dim_idx {+ d2 + start <= d1+ } else {+ d1 == d2+ }+ });+ if !shape_ok {+ Err(Error::ShapeMismatchBinaryOp {+ op: "slice-scatter (self, src)",+ lhs: self.shape().clone(),+ rhs: src.shape().clone(),+ }+ .bt())?+ }+ let mut storage = unsafe { self.device().alloc_uninit(self.shape(), self.dtype())? };+ self.storage()+ .copy_strided_src(&mut storage, 0, self.layout())?;+ let offset = start * src.dims()[1..].iter().product::<usize>();+ src.storage()+ .copy_strided_src(&mut storage, offset, src.layout())?;+ let op = BackpropOp::new2(self, src, |t1, t2| Op::SliceScatter0(t1, t2, start));+ Ok(from_storage(storage, self.shape(), op, false))+ }++ /// Accumulate element from `source` at indexes `indexes` and add them to `self`.+ pub fn index_add<D: Dim>(&self, indexes: &Self, source: &Self, dim: D) -> Result<Self> {+ let dim = dim.to_index(self.shape(), "index-add")?;+ let source_dims = source.dims();+ let self_dims = self.dims();+ let mismatch = if source_dims.len() != self_dims.len() {+ true+ } else {+ let mut mismatch = false;+ for (i, (&d1, &d2)) in self_dims.iter().zip(source_dims.iter()).enumerate() {+ if i != dim && d1 != d2 {+ mismatch = true;+ break;+ }+ }+ mismatch+ };+ if mismatch {+ Err(Error::ShapeMismatchBinaryOp {+ op: "index-add (self, source)",+ lhs: self.shape().clone(),+ rhs: source.shape().clone(),+ }+ .bt())?+ }+ // The number of element in indexes must match the dimension on which the add is+ // performed on the source tensor (and the index values from `indexes` are taken from+ // the target tensor self)+ let indexes_len = indexes.dims1()?;+ if source_dims[dim] != indexes_len {+ Err(Error::ShapeMismatchBinaryOp {+ op: "index-add (ids, source))",+ lhs: indexes.shape().clone(),+ rhs: source.shape().clone(),+ }+ .bt())?+ }+ let storage = self.storage().index_add(+ self.layout(),+ &indexes.storage(),+ indexes.layout(),+ &source.storage(),+ source.layout(),+ dim,+ )?;+ let op = BackpropOp::new3(self, indexes, source, |t1, t2, t3| {+ Op::IndexAdd(t1, t2, t3, dim)+ });+ Ok(from_storage(storage, self.shape(), op, false))+ }++ /// Gather values across the target dimension.+ ///+ /// # Arguments+ ///+ /// * `self` - The input tensor.+ /// * `indexes` - The indices of elements to gather, this should have same number of dimensions as `self`+ /// and indexes.dims()[d] <= self.dims()[d] for all dimensions d != dim+ /// * `dim` - the target dimension.+ ///+ /// The resulting tensor has the same shape as `indexes` and use values from `self` indexed on+ /// dimension `dim` by the values in `indexes`.+ pub fn gather<D: Dim>(&self, indexes: &Self, dim: D) -> Result<Self> {+ let dim = dim.to_index(self.shape(), "gather")?;++ let self_dims = self.dims();+ let indexes_dims = indexes.dims();+ let mismatch = if indexes_dims.len() != self_dims.len() {+ true+ } else {+ let mut mismatch = false;+ for (i, (&d1, &d2)) in self_dims.iter().zip(indexes_dims.iter()).enumerate() {+ if i != dim && d1 < d2 {+ mismatch = true;+ break;+ }+ }+ mismatch+ };+ if mismatch {+ Err(Error::ShapeMismatchBinaryOp {+ op: "gather",+ lhs: self.shape().clone(),+ rhs: indexes.shape().clone(),+ }+ .bt())?+ }+ let storage =+ self.storage()+ .gather(self.layout(), &indexes.storage(), indexes.layout(), dim)?;+ let op = BackpropOp::new2(self, indexes, |t1, t2| Op::Gather(t1, t2, dim));+ Ok(from_storage(storage, indexes.shape(), op, false))+ }++ /// Select values for the input tensor at the target indexes across the specified dimension.+ ///+ /// The `indexes` is argument is an int tensor with a single dimension.+ /// The output has the same number of dimension as the `self` input. The target dimension of+ /// the output has length the length of `indexes` and the values are taken from `self` using+ /// the index from `indexes`. Other dimensions have the same number of elements as the input+ /// tensor.+ pub fn index_select<D: Dim>(&self, indexes: &Self, dim: D) -> Result<Self> {+ let dim = dim.to_index(self.shape(), "index-select")?;+ let indexes_len = match indexes.dims() {+ [l] => *l,+ _ => Err(Error::ShapeMismatchBinaryOp {+ lhs: self.shape().clone(),+ rhs: indexes.shape().clone(),+ op: "index-select",+ }+ .bt())?,+ };+ let storage = self.storage().index_select(+ &indexes.storage(),+ self.layout(),+ indexes.layout(),+ dim,+ )?;+ let mut dims = self.dims().to_vec();+ dims[dim] = indexes_len;+ let op = BackpropOp::new2(self, indexes, |t1, t2| Op::IndexSelect(t1, t2, dim));+ Ok(from_storage(storage, dims, op, false))+ }++ /// Returns an iterator over position of the elements in the storage when ranging over the+ /// index tuples in lexicographic order.+ pub fn strided_index(&self) -> crate::StridedIndex<'_> {+ self.layout.strided_index()+ }++ /// Similar to `strided_index` but returns the position of the start of each contiguous block+ /// as well as the length of the contiguous blocks. For a contiguous tensor, the index iterator+ /// will only return the start offset and the size would be the number of elements in the+ /// tensor.+ pub fn strided_blocks(&self) -> crate::StridedBlocks<'_> {+ self.layout.strided_blocks()+ }++ /// Returns the data contained in a 1D tensor as a vector of scalar values.+ pub fn to_vec1<S: crate::WithDType>(&self) -> Result<Vec<S>> {+ if self.rank() != 1 {+ Err(Error::UnexpectedNumberOfDims {+ expected: 1,+ got: self.rank(),+ shape: self.shape().clone(),+ }+ .bt())?+ }+ let from_cpu_storage = |cpu_storage: &crate::CpuStorage| {+ let data = S::cpu_storage_as_slice(cpu_storage)?;+ let data = match self.layout.contiguous_offsets() {+ Some((o1, o2)) => data[o1..o2].to_vec(),+ None => self.strided_index().map(|i| data[i]).collect(),+ };+ Ok::<Vec<_>, Error>(data)+ };+ match &*self.storage() {+ Storage::Cpu(storage) => from_cpu_storage(storage),+ Storage::Cuda(storage) => from_cpu_storage(&storage.to_cpu_storage()?),+ Storage::Metal(storage) => from_cpu_storage(&storage.to_cpu_storage()?),+ }+ }++ /// Returns the data contained in a 2D tensor as a vector of vector of scalar values.+ pub fn to_vec2<S: crate::WithDType>(&self) -> Result<Vec<Vec<S>>> {+ let (dim1, dim2) = self.dims2()?;+ let from_cpu_storage = |cpu_storage: &crate::CpuStorage| {+ let data = S::cpu_storage_as_slice(cpu_storage)?;+ let mut rows = vec![];+ match self.layout.contiguous_offsets() {+ Some((o1, o2)) => {+ let data = &data[o1..o2];+ for idx_row in 0..dim1 {+ rows.push(data[idx_row * dim2..(idx_row + 1) * dim2].to_vec())+ }+ }+ None => {+ let mut src_index = self.strided_index();+ for _idx_row in 0..dim1 {+ let row = (0..dim2).map(|_| data[src_index.next().unwrap()]).collect();+ rows.push(row)+ }+ assert!(src_index.next().is_none());+ }+ }+ Ok(rows)+ };+ match &*self.storage() {+ Storage::Cpu(storage) => from_cpu_storage(storage),+ Storage::Cuda(storage) => from_cpu_storage(&storage.to_cpu_storage()?),+ Storage::Metal(storage) => from_cpu_storage(&storage.to_cpu_storage()?),+ }+ }++ /// Returns the data contained in a 3D tensor.+ pub fn to_vec3<S: crate::WithDType>(&self) -> Result<Vec<Vec<Vec<S>>>> {+ let (dim1, dim2, dim3) = self.dims3()?;+ let from_cpu_storage = |cpu_storage: &crate::CpuStorage| {+ let data = S::cpu_storage_as_slice(cpu_storage)?;+ let mut top_rows = vec![];+ match self.layout.contiguous_offsets() {+ Some((o1, o2)) => {+ let data = &data[o1..o2];+ let dim23 = dim2 * dim3;+ for idx1 in 0..dim1 {+ let data = &data[idx1 * dim23..(idx1 + 1) * dim23];+ let mut rows = vec![];+ for idx2 in 0..dim2 {+ rows.push(data[idx2 * dim3..(idx2 + 1) * dim3].to_vec())+ }+ top_rows.push(rows);+ }+ }+ None => {+ let mut src_index = self.strided_index();+ for _idx in 0..dim1 {+ let mut rows = vec![];+ for _jdx in 0..dim2 {+ let row = (0..dim3).map(|_| data[src_index.next().unwrap()]).collect();+ rows.push(row)+ }+ top_rows.push(rows);+ }+ assert!(src_index.next().is_none());+ }+ }+ Ok(top_rows)+ };+ match &*self.storage() {+ Storage::Cpu(storage) => from_cpu_storage(storage),+ Storage::Cuda(storage) => from_cpu_storage(&storage.to_cpu_storage()?),+ Storage::Metal(storage) => from_cpu_storage(&storage.to_cpu_storage()?),+ }+ }++ /// The dtype for the elements stored in the input tensor.+ pub fn dtype(&self) -> DType {+ self.dtype+ }++ /// The device on which the input tensor is located.+ pub fn device(&self) -> &Device {+ &self.device+ }++ /// The tensor shape, i.e. dimension sizes on each axis.+ pub fn shape(&self) -> &Shape {+ self.layout().shape()+ }++ /// The dimension size for this tensor on each axis.+ pub fn dims(&self) -> &[usize] {+ self.shape().dims()+ }++ /// The dimension size for a specified dimension index.+ pub fn dim<D: Dim>(&self, dim: D) -> Result<usize> {+ let dim = dim.to_index(self.shape(), "dim")?;+ Ok(self.dims()[dim])+ }++ /// The layout of the input tensor, this stores both the shape of the tensor as well as the+ /// strides and the start offset to apply to the underlying storage.+ pub fn layout(&self) -> &Layout {+ &self.layout+ }++ pub fn stride(&self) -> &[usize] {+ self.layout.stride()+ }++ /// The number of dimensions for this tensor, 0 for a scalar tensor, 1 for a 1D tensor, etc.+ pub fn rank(&self) -> usize {+ self.shape().rank()+ }++ /// The number of elements stored in this tensor.+ pub fn elem_count(&self) -> usize {+ self.shape().elem_count()+ }++ /// The unique identifier for this tensor.+ pub fn id(&self) -> TensorId {+ self.id+ }++ /// Whether this tensor is a variable or not. A variable is a tensor for which gradient is+ /// tracked and on which backpropagation can be performed.+ pub fn is_variable(&self) -> bool {+ self.is_variable+ }++ pub(crate) fn op(&self) -> &Option<Op> {+ &self.op+ }++ /// Computes the max of all the elements in this tensor and returns a tensor holding this+ /// scalar with zero dimensions.+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let tensor = Tensor::new(&[[0f32, 1.], [2., 3.], [4., 5.]], &Device::Cpu)?;+ /// let tensor = tensor.max_all()?;+ /// assert_eq!(tensor.to_scalar::<f32>()?, 5.);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn max_all(&self) -> Result<Tensor> {+ if self.rank() == 0 {+ Ok(self.clone())+ } else {+ self.flatten_all()?.max(0)+ }+ }++ /// Computes the min of all the elements in this tensor and returns a tensor holding this+ /// scalar with zero dimensions.+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let tensor = Tensor::new(&[[0f32, 1.], [2., 3.], [4., 5.]], &Device::Cpu)?;+ /// let tensor = tensor.min_all()?;+ /// assert_eq!(tensor.to_scalar::<f32>()?, 0.);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn min_all(&self) -> Result<Tensor> {+ if self.rank() == 0 {+ Ok(self.clone())+ } else {+ self.flatten_all()?.min(0)+ }+ }++ /// Computes the sum of all the elements in this tensor and returns a tensor holding this+ /// scalar with zero dimensions.+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let tensor = Tensor::new(&[[0f32, 1.], [2., 3.], [4., 5.]], &Device::Cpu)?;+ /// let tensor = tensor.sum_all()?;+ /// assert_eq!(tensor.to_scalar::<f32>()?, 15.);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn sum_all(&self) -> Result<Tensor> {+ let dims: Vec<_> = (0..self.rank()).collect();+ self.sum(dims)+ }++ pub fn mean_all(&self) -> Result<Tensor> {+ self.sum_all()? / self.elem_count() as f64+ }++ fn flatten_<D1: Dim, D2: Dim>(+ &self,+ start_dim: Option<D1>,+ end_dim: Option<D2>,+ ) -> Result<Tensor> {+ if self.rank() == 0 {+ self.reshape(1)+ } else {+ let start_dim = match start_dim {+ None => 0,+ Some(dim) => dim.to_index(self.shape(), "flatten")?,+ };+ let end_dim = match end_dim {+ None => self.rank() - 1,+ Some(dim) => dim.to_index(self.shape(), "flatten")?,+ };+ if start_dim < end_dim {+ let dims = self.dims();+ let mut dst_dims = dims[..start_dim].to_vec();+ dst_dims.push(dims[start_dim..end_dim + 1].iter().product::<usize>());+ if end_dim + 1 < dims.len() {+ dst_dims.extend(&dims[end_dim + 1..]);+ }+ self.reshape(dst_dims)+ } else {+ Ok(self.clone())+ }+ }+ }++ /// Flattens the input tensor on the dimension indexes from `start_dim` to `end_dim` (both+ /// inclusive).+ pub fn flatten<D1: Dim, D2: Dim>(&self, start_dim: D1, end_dim: D2) -> Result<Tensor> {+ self.flatten_(Some(start_dim), Some(end_dim))+ }++ /// Flattens the input tensor on the dimension indexes from `0` to `end_dim` (inclusive).+ pub fn flatten_to<D: Dim>(&self, end_dim: D) -> Result<Tensor> {+ self.flatten_(None::<usize>, Some(end_dim))+ }++ /// Flattens the input tensor on the dimension indexes from `start_dim` (inclusive) to the last+ /// dimension.+ pub fn flatten_from<D: Dim>(&self, start_dim: D) -> Result<Tensor> {+ self.flatten_(Some(start_dim), None::<usize>)+ }++ /// Flattens the input tensor by reshaping it into a one dimension tensor.+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let tensor = Tensor::new(&[[0f32, 1.], [2., 3.], [4., 5.]], &Device::Cpu)?;+ /// let tensor = tensor.flatten_all()?;+ /// assert_eq!(tensor.to_vec1::<f32>()?, &[0., 1., 2., 3., 4., 5.]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn flatten_all(&self) -> Result<Tensor> {+ self.flatten_(None::<usize>, None::<usize>)+ }++ /// Returns the sub-tensor fixing the index at `i` on the first dimension.+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let tensor = Tensor::new(&[[0f32, 1.], [2., 3.], [4., 5.]], &Device::Cpu)?;+ /// let t = tensor.get(0)?;+ /// assert_eq!(t.to_vec1::<f32>()?, &[0., 1.]);+ /// let t = tensor.get(1)?;+ /// assert_eq!(t.to_vec1::<f32>()?, &[2., 3.]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn get(&self, i: usize) -> Result<Tensor> {+ let dims = self.dims();+ if dims.is_empty() {+ Ok(self.clone())+ } else {+ self.narrow(0, i, 1)?.reshape(&dims[1..])+ }+ }++ /// Returns the sub-tensor fixing the index at `index` on the dimension `dim`.+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let tensor = Tensor::new(&[[0f32, 1.], [2., 3.], [4., 5.]], &Device::Cpu)?;+ /// let t = tensor.get_on_dim(1, 0)?;+ /// assert_eq!(t.to_vec1::<f32>()?, &[0., 2., 4.]);+ /// let t = tensor.get_on_dim(1, 1)?;+ /// assert_eq!(t.to_vec1::<f32>()?, &[1., 3., 5.]);+ /// let t = tensor.get_on_dim(0, 1)?;+ /// assert_eq!(t.to_vec1::<f32>()?, &[2., 3.]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn get_on_dim<D: Dim>(&self, dim: D, index: usize) -> Result<Tensor> {+ let dim = dim.to_index(self.shape(), "get_on_dim")?;+ self.narrow(dim, index, 1)?.squeeze(dim)+ }++ /// Returns a tensor that is a transposed version of the input, the two last dimensions of the+ /// input are swapped.+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let tensor = Tensor::new(&[[0f32, 1.], [2., 3.], [4., 5.]], &Device::Cpu)?;+ /// let tensor = tensor.t()?;+ /// assert_eq!(tensor.to_vec2::<f32>()?, &[[0.0, 2.0, 4.0], [1.0, 3.0, 5.0]]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn t(&self) -> Result<Tensor> {+ let rank = self.rank();+ if rank < 2 {+ Err(Error::UnexpectedNumberOfDims {+ expected: 2,+ got: rank,+ shape: self.shape().clone(),+ }+ .bt())?+ }+ self.transpose(rank - 2, rank - 1)+ }++ /// Returns a tensor that is a transposed version of the input, the given dimensions are+ /// swapped.+ pub fn transpose<D1: Dim, D2: Dim>(&self, dim1: D1, dim2: D2) -> Result<Tensor> {+ let dim1 = dim1.to_index(self.shape(), "transpose")?;+ let dim2 = dim2.to_index(self.shape(), "transpose")?;+ if dim1 == dim2 {+ return Ok(self.clone());+ }+ let op = BackpropOp::new1(self, |t| Op::Transpose(t, dim1, dim2));+ let tensor_ = Tensor_ {+ id: TensorId::new(),+ storage: self.storage.clone(),+ layout: self.layout.transpose(dim1, dim2)?,+ op,+ is_variable: false,+ dtype: self.dtype,+ device: self.device.clone(),+ };+ Ok(Tensor(Arc::new(tensor_)))+ }++ /// Returns a tensor with the same data as the input where the dimensions have been permuted.+ /// dims must be a permutation, i.e. include each dimension index exactly once.+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let tensor = Tensor::arange(0u32, 120u32, &Device::Cpu)?.reshape((2, 3, 4, 5))?;+ /// assert_eq!(tensor.dims(), &[2, 3, 4, 5]);+ /// let tensor = tensor.permute((2, 3, 1, 0))?;+ /// assert_eq!(tensor.dims(), &[4, 5, 3, 2]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn permute<D: Dims>(&self, dims: D) -> Result<Tensor> {+ let dims = dims.to_indexes(self.shape(), "permute")?;+ // O(n^2) permutation check but these arrays are small.+ let is_permutation =+ dims.len() == self.rank() && (0..dims.len()).all(|i| dims.contains(&i));+ if !is_permutation {+ bail!(+ "dimension mismatch in permute, tensor {:?}, dims: {:?}",+ self.dims(),+ dims+ )+ }+ let op = BackpropOp::new1(self, |t| Op::Permute(t, dims.clone()));+ let tensor_ = Tensor_ {+ id: TensorId::new(),+ storage: self.storage.clone(),+ layout: self.layout.permute(&dims)?,+ op,+ is_variable: false,+ dtype: self.dtype,+ device: self.device.clone(),+ };+ Ok(Tensor(Arc::new(tensor_)))+ }++ /// Returns true if the data is stored in a C contiguous (aka row major) way.+ pub fn is_contiguous(&self) -> bool {+ self.layout.is_contiguous()+ }++ /// Returns true if the data is stored in a Fortran contiguous (aka column major) way.+ pub fn is_fortran_contiguous(&self) -> bool {+ self.layout.is_fortran_contiguous()+ }++ /// Compared to clone, this copies the actual storage but may fail because of running out of+ /// memory.+ pub fn copy(&self) -> Result<Tensor> {+ let op = BackpropOp::new1(self, Op::Copy);+ let tensor_ = Tensor_ {+ id: TensorId::new(),+ storage: Arc::new(RwLock::new(self.storage().try_clone(self.layout())?)),+ layout: self.layout.clone(),+ op,+ is_variable: false,+ dtype: self.dtype,+ device: self.device.clone(),+ };+ Ok(Tensor(Arc::new(tensor_)))+ }++ /// Returns a new tensor detached from the current graph, gradient are not propagated through+ /// this new node. The storage of this tensor is shared with the initial tensor.+ ///+ /// If the tensor is already detached from the computation graph, the same tensor is returned.+ pub fn detach(&self) -> Tensor {+ if self.op.is_none() && !self.is_variable {+ self.clone()+ } else {+ let tensor_ = Tensor_ {+ id: TensorId::new(),+ storage: self.storage.clone(),+ layout: self.layout.clone(),+ op: BackpropOp::none(),+ is_variable: false,+ dtype: self.dtype,+ device: self.device.clone(),+ };+ Tensor(Arc::new(tensor_))+ }+ }++ /// If the target device is the same as the tensor device, only a shallow copy is performed.+ pub fn to_device(&self, device: &Device) -> Result<Tensor> {+ if self.device().same_device(device) {+ Ok(self.clone())+ } else {+ let storage = match (&*self.storage(), device) {+ (Storage::Cpu(storage), Device::Cuda(cuda)) => {+ Storage::Cuda(cuda.storage_from_cpu_storage(storage)?)+ }+ (Storage::Cpu(storage), Device::Metal(metal)) => {+ Storage::Metal(metal.storage_from_cpu_storage(storage)?)+ }+ (Storage::Cuda(storage), Device::Cpu) => Storage::Cpu(storage.to_cpu_storage()?),+ (Storage::Metal(storage), Device::Cpu) => Storage::Cpu(storage.to_cpu_storage()?),+ (Storage::Cuda(storage), Device::Cuda(cuda)) => {+ // can't clone storage if it's the same device because of the underlying device ptr+ let dst_storage = storage.transfer_to_device(cuda)?;+ Storage::Cuda(dst_storage)+ }+ (Storage::Cpu(storage), Device::Cpu) => Storage::Cpu(storage.clone()),+ _ => {+ bail!(+ "not implemented yet, self.device: {:?}, device: {:?}",+ self.device(),+ device+ )+ }+ };+ let op = BackpropOp::new1(self, Op::ToDevice);+ let tensor_ = Tensor_ {+ id: TensorId::new(),+ storage: Arc::new(RwLock::new(storage)),+ layout: self.layout.clone(),+ op,+ is_variable: false,+ dtype: self.dtype,+ device: device.clone(),+ };+ Ok(Tensor(Arc::new(tensor_)))+ }+ }++ /// Returns a new tensor duplicating data from the original tensor. New dimensions are inserted+ /// on the left.+ pub fn broadcast_left<S: Into<Shape>>(&self, left_shape: S) -> Result<Self> {+ let left_shape = left_shape.into();+ let mut dims = left_shape.into_dims();+ dims.extend(self.dims());+ self.broadcast_as(dims)+ }++ /// Broadcast the input tensor to the target shape. This returns an error if the input shape is+ /// not compatible with the target shape.+ ///+ /// If the input shape is `i_1, i_2, ... i_k`, the target shape has to have `k` dimensions or+ /// more and shape `j_1, ..., j_l, t_1, t_2, ..., t_k`. The dimensions `j_1` to `j_l` can have+ /// any value, the dimension `t_a` must be equal to `i_a` if `i_a` is different from 1. If+ /// `i_a` is equal to 1, any value can be used.+ pub fn broadcast_as<S: Into<Shape>>(&self, shape: S) -> Result<Self> {+ let tensor_ = Tensor_ {+ id: TensorId::new(),+ storage: self.storage.clone(),+ layout: self.layout.broadcast_as(shape)?,+ op: BackpropOp::new1(self, Op::Broadcast),+ is_variable: false,+ dtype: self.dtype,+ device: self.device.clone(),+ };+ Ok(Tensor(Arc::new(tensor_)))+ }++ /// An alias for broadcast_as.+ pub fn expand<S: Into<Shape>>(&self, shape: S) -> Result<Self> {+ self.broadcast_as(shape)+ }++ /// Casts the input tensor to the target `dtype`.+ ///+ /// ```rust+ /// use candle_core::{Tensor, Device};+ /// let tensor = Tensor::new(3.14159265358979f64, &Device::Cpu)?;+ /// assert_eq!(tensor.to_scalar::<f64>()?, 3.14159265358979);+ /// let tensor = tensor.to_dtype(candle_core::DType::F32)?;+ /// assert_eq!(tensor.to_scalar::<f32>()?, 3.1415927);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn to_dtype(&self, dtype: DType) -> Result<Self> {+ if self.dtype() == dtype {+ Ok(self.clone())+ } else {+ let shape = self.shape();+ let storage = self.storage().to_dtype(self.layout(), dtype)?;+ let op = BackpropOp::new1(self, Op::ToDType);+ Ok(from_storage(storage, shape.clone(), op, false))+ }+ }++ /// Returns a tensor that is in row major order. This is the same as the original tensor if it+ /// was already contiguous, otherwise a copy is triggered.+ pub fn contiguous(&self) -> Result<Tensor> {+ if self.is_contiguous() {+ Ok(self.clone())+ } else {+ let shape = self.shape();+ let mut storage = unsafe { self.device().alloc_uninit(shape, self.dtype())? };+ self.storage()+ .copy_strided_src(&mut storage, 0, self.layout())?;+ let op = BackpropOp::new1(self, Op::Copy);+ Ok(from_storage(storage, shape.clone(), op, false))+ }+ }++ /// Returns a tensor that is in row major order. This always makes a copy.+ pub fn force_contiguous(&self) -> Result<Tensor> {+ let shape = self.shape();+ let mut storage = unsafe { self.device().alloc_uninit(shape, self.dtype())? };+ self.storage()+ .copy_strided_src(&mut storage, 0, self.layout())?;+ let op = BackpropOp::new1(self, Op::Copy);+ Ok(from_storage(storage, shape.clone(), op, false))+ }++ /// Create a variable based on the values currently stored in a tensor. The storage is always+ /// copied.+ pub(crate) fn make_var(&self) -> Result<Tensor> {+ let shape = self.shape().clone();+ let mut storage = unsafe { self.device().alloc_uninit(&shape, self.dtype())? };+ self.storage()+ .copy_strided_src(&mut storage, 0, self.layout())?;+ Ok(from_storage(storage, shape, BackpropOp::none(), true))+ }++ /// Reshape returns a tensor with the target shape provided that the number of elements of the+ /// original tensor is the same.+ /// If the input tensor is contiguous, this is a view on the original data. Otherwise this uses+ /// a new storage and copies the data over, the returned tensor is always contiguous.+ ///+ /// The shape can be specified using a tuple of `usize` and at most one `()` in which case+ /// the behavior is the same as when using `-1` in PyTorch: this dimension size is adjusted so+ /// as to match the number of elements in the tensor.+ ///+ /// ```rust+ /// # use candle_core::{Tensor, DType, Device, D};+ /// let a = Tensor::zeros((2, 3), DType::F32, &Device::Cpu)?;+ ///+ /// let c = a.reshape((1, 6))?;+ /// assert_eq!(c.shape().dims(), &[1, 6]);+ ///+ /// let c = a.reshape((3, 2))?;+ /// assert_eq!(c.shape().dims(), &[3, 2]);+ ///+ /// let c = a.reshape((2, (), 1))?;+ /// assert_eq!(c.shape().dims(), &[2, 3, 1]);+ ///+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn reshape<S: ShapeWithOneHole>(&self, s: S) -> Result<Tensor> {+ let shape = s.into_shape(self.elem_count())?;+ if shape.elem_count() != self.elem_count() {+ return Err(Error::ShapeMismatchBinaryOp {+ lhs: self.shape().clone(),+ rhs: shape,+ op: "reshape",+ }+ .bt());+ }+ let op = BackpropOp::new1(self, Op::Reshape);+ if self.is_contiguous() {+ let tensor_ = Tensor_ {+ id: TensorId::new(),+ storage: self.storage.clone(),+ layout: Layout::contiguous_with_offset(shape, self.layout.start_offset()),+ op,+ is_variable: false,+ dtype: self.dtype,+ device: self.device.clone(),+ };+ Ok(Tensor(Arc::new(tensor_)))+ } else {+ let mut storage = unsafe { self.device().alloc_uninit(&shape, self.dtype())? };+ self.storage()+ .copy_strided_src(&mut storage, 0, self.layout())?;+ Ok(from_storage(storage, shape, op, false))+ }+ }++ /// Creates a new tensor with the specified dimension removed if its size was one.+ ///+ /// ```rust+ /// # use candle_core::{Tensor, DType, Device, D};+ /// let a = Tensor::zeros((2, 3, 1), DType::F32, &Device::Cpu)?;+ ///+ /// let c = a.squeeze(2)?;+ /// assert_eq!(c.shape().dims(), &[2, 3]);+ ///+ /// let c = a.squeeze(D::Minus1)?;+ /// assert_eq!(c.shape().dims(), &[2, 3]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn squeeze<D: Dim>(&self, dim: D) -> Result<Self> {+ // The PyTorch semantics are to return the same tensor if the target dimension+ // does not have a size of 1.+ let dims = self.dims();+ let dim = dim.to_index(self.shape(), "squeeze")?;+ if dims[dim] == 1 {+ let mut dims = dims.to_vec();+ let mut strides = self.stride().to_vec();+ dims.remove(dim);+ strides.remove(dim);+ let tensor_ = Tensor_ {+ id: TensorId::new(),+ storage: self.storage.clone(),+ layout: Layout::new(dims.into(), strides, self.layout.start_offset()),+ op: BackpropOp::new1(self, Op::Reshape),+ is_variable: false,+ dtype: self.dtype,+ device: self.device.clone(),+ };+ Ok(Tensor(Arc::new(tensor_)))+ } else {+ Ok(self.clone())+ }+ }++ /// Creates a new tensor with a dimension of size one inserted at the specified position.+ ///+ /// ```rust+ /// # use candle_core::{Tensor, DType, Device, D};+ /// let a = Tensor::zeros((2, 3), DType::F32, &Device::Cpu)?;+ ///+ /// let c = a.unsqueeze(0)?;+ /// assert_eq!(c.shape().dims(), &[1, 2, 3]);+ ///+ /// let c = a.unsqueeze(D::Minus1)?;+ /// assert_eq!(c.shape().dims(), &[2, 3, 1]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn unsqueeze<D: Dim>(&self, dim: D) -> Result<Self> {+ let mut dims = self.dims().to_vec();+ let mut strides = self.stride().to_vec();+ let dim = dim.to_index_plus_one(self.shape(), "unsqueeze")?;+ // Cannot panic because to_index_plus_one already checks dimensions+ dims.insert(dim, 1);+ // Any stride would work here, but we pick one so as to maximize the probability to remain+ // C contiguous.+ let stride = if dim < strides.len() { strides[dim] } else { 1 };+ strides.insert(dim, stride);+ let tensor_ = Tensor_ {+ id: TensorId::new(),+ storage: self.storage.clone(),+ layout: Layout::new(dims.into(), strides, self.layout.start_offset()),+ op: BackpropOp::new1(self, Op::Reshape),+ is_variable: false,+ dtype: self.dtype,+ device: self.device.clone(),+ };+ Ok(Tensor(Arc::new(tensor_)))+ }++ /// Stacks two or more tensors along a particular dimension.+ ///+ /// All tensors must have the same rank, and the output has one additional rank+ ///+ /// ```rust+ /// # use candle_core::{Tensor, DType, Device};+ /// let a = Tensor::zeros((2, 3), DType::F32, &Device::Cpu)?;+ /// let b = Tensor::zeros((2, 3), DType::F32, &Device::Cpu)?;+ ///+ /// let c = Tensor::stack(&[&a, &b], 0)?;+ /// assert_eq!(c.shape().dims(), &[2, 2, 3]);+ ///+ /// let c = Tensor::stack(&[&a, &b], 2)?;+ /// assert_eq!(c.shape().dims(), &[2, 3, 2]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn stack<A: AsRef<Tensor>, D: Dim>(args: &[A], dim: D) -> Result<Self> {+ if args.is_empty() {+ Err(Error::OpRequiresAtLeastOneTensor { op: "stack" }.bt())?+ }+ let dim = dim.to_index_plus_one(args[0].as_ref().shape(), "stack")?;+ let args = args+ .iter()+ .map(|t| t.as_ref().unsqueeze(dim))+ .collect::<Result<Vec<_>>>()?;+ Self::cat(&args, dim)+ }++ /// Pad the input tensor using 0s along dimension `dim`. This adds `left` elements before the+ /// input tensor values and `right` elements after.+ pub fn pad_with_zeros<D: Dim>(&self, dim: D, left: usize, right: usize) -> Result<Self> {+ if left == 0 && right == 0 {+ Ok(self.clone())+ } else if left == 0 {+ let dim = dim.to_index(self.shape(), "pad_with_zeros")?;+ let mut dims = self.dims().to_vec();+ dims[dim] = right;+ let right = Tensor::zeros(dims.as_slice(), self.dtype, self.device())?;+ Tensor::cat(&[self, &right], dim)+ } else if right == 0 {+ let dim = dim.to_index(self.shape(), "pad_with_zeros")?;+ let mut dims = self.dims().to_vec();+ dims[dim] = left;+ let left = Tensor::zeros(dims.as_slice(), self.dtype, self.device())?;+ Tensor::cat(&[&left, self], dim)+ } else {+ let dim = dim.to_index(self.shape(), "pad_with_zeros")?;+ let mut dims = self.dims().to_vec();+ dims[dim] = left;+ let left = Tensor::zeros(dims.as_slice(), self.dtype, self.device())?;+ dims[dim] = right;+ let right = Tensor::zeros(dims.as_slice(), self.dtype, self.device())?;+ Tensor::cat(&[&left, self, &right], dim)+ }+ }++ /// Pad the input tensor using same values along dimension `dim`. This adds `left` elements before the+ /// input tensor values and `right` elements after.+ pub fn pad_with_same<D: Dim>(&self, dim: D, left: usize, right: usize) -> Result<Self> {+ if left == 0 && right == 0 {+ Ok(self.clone())+ } else if self.elem_count() == 0 {+ bail!("cannot use pad_with_same on an empty tensor")+ } else if left == 0 {+ let dim = dim.to_index(self.shape(), "pad_with_same")?;+ let r = self.narrow(dim, self.dim(dim)? - 1, 1)?;+ let mut v = vec![self];+ for _ in 0..right {+ v.push(&r)+ }+ Tensor::cat(&v, dim)+ } else if right == 0 {+ let dim = dim.to_index(self.shape(), "pad_with_same")?;+ let l = self.narrow(dim, 0, 1)?;+ let mut v = vec![];+ for _ in 0..left {+ v.push(&l)+ }+ v.push(self);+ Tensor::cat(&v, dim)+ } else {+ let dim = dim.to_index(self.shape(), "pad_with_same")?;+ let l = self.narrow(dim, 0, 1)?;+ let r = self.narrow(dim, self.dim(dim)? - 1, 1)?;+ let mut v = vec![];+ for _ in 0..left {+ v.push(&l)+ }+ v.push(self);+ for _ in 0..right {+ v.push(&r)+ }+ Tensor::cat(&v, dim)+ }+ }++ /// Run the `forward` method of `m` on `self`.+ pub fn apply<M: crate::Module>(&self, m: &M) -> Result<Self> {+ m.forward(self)+ }++ /// Run the `forward` method of `m` on `self`.+ pub fn apply_t<M: crate::ModuleT>(&self, m: &M, train: bool) -> Result<Self> {+ m.forward_t(self, train)+ }++ /// Acquire read lock on storage and returns guard.+ /// `read_recursive` allows for shared read access.+ pub(crate) fn storage(&self) -> StorageRef<'_> {+ self.storage.read_recursive()+ }++ /// Acquire write lock on storage and returns guard.+ pub(crate) fn storage_mut(&self) -> StorageMutRef<'_> {+ self.storage.write()+ }++ // If we extend the visibility of this function to be usable outside of this crate, we should+ // make it unsafe.+ pub(crate) fn storage_mut_and_layout(&self) -> (StorageMutRef<'_>, &Layout) {+ let storage = self.storage.write();+ (storage, &self.layout)+ }++ /// The storage used by this tensor, together with the layout to use to access it safely.+ pub fn storage_and_layout(&self) -> (StorageRef<'_>, &Layout) {+ let storage = self.storage.read();+ (storage, &self.layout)+ }++ /// Unique key for this tensor's storage. Equal keys mean the tensors share the same allocation.+ #[inline]+ pub(crate) fn storage_key(&self) -> usize {+ let lock: &RwLock<Storage> = self.storage.as_ref();+ std::ptr::from_ref(lock).addr()+ }++ /// Check if two tensors share the same underlying allocation.+ #[inline]+ pub(crate) fn same_storage(&self, rhs: &Self) -> bool {+ self.storage_key() == rhs.storage_key()+ }++ /// Normalize a 'relative' axis value: positive values are kept, negative+ /// values means counting the dimensions from the back.+ pub fn normalize_axis(&self, axis: i64) -> Result<usize> {+ let rank = self.rank() as i64;+ if rank <= axis {+ bail!("axis {axis} is too large, tensor rank {rank}")+ } else if 0 <= axis {+ Ok(axis as usize)+ } else {+ let naxis = rank + axis;+ if naxis < 0 {+ bail!("axis {axis} is too small, tensor rank {rank}")+ }+ Ok(naxis as usize)+ }+ }++ /// Returns a lower triangular matrix of ones of size n by n.+ pub fn tril2(n: usize, dtype: DType, device: &Device) -> Result<Self> {+ let t = Tensor::arange(0u32, n as u32, device)?;+ let t1 = t.reshape((1, n))?.broadcast_as((n, n))?;+ let t2 = t.reshape((n, 1))?.broadcast_as((n, n))?;+ t1.le(&t2)?.to_dtype(dtype)+ }++ /// Returns an upper triangular matrix of ones of size n by n.+ pub fn triu2(n: usize, dtype: DType, device: &Device) -> Result<Self> {+ let t = Tensor::arange(0u32, n as u32, device)?;+ let t1 = t.reshape((1, n))?.broadcast_as((n, n))?;+ let t2 = t.reshape((n, 1))?.broadcast_as((n, n))?;+ t1.ge(&t2)?.to_dtype(dtype)+ }++ /// Returns a matrix with a diagonal of ones of size n by n.+ pub fn eye(n: usize, dtype: DType, device: &Device) -> Result<Self> {+ let t = Tensor::arange(0u32, n as u32, device)?;+ let t1 = t.reshape((1, n))?.broadcast_as((n, n))?;+ let t2 = t.reshape((n, 1))?.broadcast_as((n, n))?;+ t1.eq(&t2)?.to_dtype(dtype)+ }++ /// Returns the cumulative sum of elements of the input tensor summed over the specified+ /// dimension.+ ///+ /// This operation is most efficient when dim is the last dimension of the tensor.+ pub fn cumsum<D: Dim>(&self, dim: D) -> Result<Self> {+ let dim = dim.to_index(self.shape(), "cumsum")?;+ let rank = self.rank();+ if rank == 0 {+ return Ok(self.clone());+ }+ let n_axis = self.dim(dim)?;+ let triu = Tensor::triu2(n_axis, self.dtype(), self.device())?;+ if rank == 1 {+ self.unsqueeze(0)?.matmul(&triu)?.squeeze(0)+ } else {+ let last = rank - 1;+ let t = self.transpose(dim, last)?;+ let t = t.broadcast_matmul(&triu)?;+ t.transpose(dim, last)+ }+ }++ /// Returns a copy of `self` where the values within `ranges` have been replaced with the+ /// content of `src`.+ pub fn slice_assign<D: std::ops::RangeBounds<usize>>(+ &self,+ ranges: &[D],+ src: &Tensor,+ ) -> Result<Self> {+ let src_dims = src.dims();+ let self_dims = self.dims();+ if self_dims.len() != src_dims.len() {+ bail!(+ "slice-assign requires input with the same rank {} <> {}",+ self_dims.len(),+ src_dims.len()+ )+ }+ if self_dims.len() != ranges.len() {+ bail!(+ "slice-assign requires input with the same rank as there are ranges {} <> {}",+ self_dims.len(),+ ranges.len()+ )+ }+ let mut src = src.clone();+ let mut mask = Self::ones(src.shape(), DType::U8, src.device())?;+ for (i, range) in ranges.iter().enumerate() {+ let start_included = match range.start_bound() {+ std::ops::Bound::Unbounded => 0,+ std::ops::Bound::Included(v) => *v,+ std::ops::Bound::Excluded(v) => *v + 1,+ };+ let end_excluded = match range.end_bound() {+ std::ops::Bound::Unbounded => self_dims[i],+ std::ops::Bound::Included(v) => *v + 1,+ std::ops::Bound::Excluded(v) => *v,+ };+ if end_excluded <= start_included {+ bail!("slice-assign: empty range for dim {i}, {start_included} {end_excluded}")+ }+ if self_dims[i] < end_excluded {+ bail!(+ "slice-assign: upper bound is out of range for dim {i}, {end_excluded} {}",+ self_dims[i]+ )+ }+ if end_excluded - start_included != src_dims[i] {+ bail!(+ "slice-assign: the range for dim {i} ({start_included}..{end_excluded}) does not match the size of src {}", src_dims[i]+ )+ }+ src = src.pad_with_zeros(i, start_included, self_dims[i] - end_excluded)?;+ mask = mask.pad_with_zeros(i, start_included, self_dims[i] - end_excluded)?+ }+ mask.where_cond(/* on_true= */ &src, /* on_false= */ self)+ }++ /// Returns log(sum(exp(tensor), dim)).+ pub fn log_sum_exp<D: Dims>(&self, sum_dims: D) -> Result<Self> {+ let sum_dims = sum_dims.to_indexes(self.shape(), "log-sum-exp")?;+ if sum_dims.is_empty() {+ return Ok(self.clone());+ }+ let max = sum_dims[1..]+ .iter()+ .try_fold(self.max_keepdim(sum_dims[0])?, |max, &dim| {+ max.max_keepdim(dim)+ })?;+ let exp = self.broadcast_sub(&max)?.exp()?;+ let sum = exp.sum(sum_dims.clone())?;++ sum.log()? + max.squeeze_dims(&sum_dims)+ }++ /// Pointwise pow operation.+ pub fn pow(&self, rhs: &Tensor) -> Result<Self> {+ rhs.mul(&self.log()?)?.exp()+ }++ /// Broadcasting version of `pow`.+ pub fn broadcast_pow(&self, rhs: &Tensor) -> Result<Self> {+ rhs.broadcast_mul(&self.log()?)?.exp()+ }++ /// Returns a new tensor with the order of elements reversed along the specified dimensions.+ /// This function makes a copy of the tensor’s data.+ ///+ /// ```rust+ /// # use candle_core::{Tensor, Device};+ /// let t = Tensor::arange(0., 6., &Device::Cpu)?.reshape((2, 3))?;+ /// assert_eq!(t.to_vec2::<f64>()?, &[[0.0, 1.0, 2.0], [3.0, 4.0, 5.0]]);+ /// let t_flipped = t.flip(&[0])?;+ /// assert_eq!(t_flipped.to_vec2::<f64>()?, &[[3.0, 4.0, 5.0], [0.0, 1.0, 2.0]]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn flip(&self, dims: &[usize]) -> Result<Tensor> {+ let mut result = self.clone();+ for &dim in dims.iter() {+ let size = result.dim(dim)?;+ let indices: Vec<i64> = (0..size).rev().map(|x| x as i64).collect();+ let indices_tensor = Tensor::from_vec(indices, (size,), result.device())?;+ result = result.index_select(&indices_tensor, dim)?;+ }+ Ok(result)+ }++ /// Returns a view of which contains all slices of size `size` from self tensor in the dimension+ /// `dim` and stepped by `step`.+ pub fn unfold<D: Dim>(&self, dim: D, size: usize, step: usize) -> Result<Self> {+ // https://github.com/pytorch/pytorch/blob/75b0720a97ac5d82e8a7a1a6ae7c5f7a87d7183d/aten/src/ATen/native/TensorShape.cpp#L3785-L3804+ let mut sizes = self.dims().to_vec();+ let mut strides = self.stride().to_vec();++ let dim = dim.to_index(self.shape(), "unfold")?;++ let max_len = if self.dims().is_empty() {+ 1+ } else {+ sizes[dim]+ };+ if size > max_len {+ bail!(+ "unsqueeze: maximum size for tensor at dimension {dim} is {max_len} but size is {size}"+ )+ }+ sizes.push(size);+ strides.push(if self.dims().is_empty() {+ 1+ } else {+ strides[dim]+ });++ if !self.dims().is_empty() {+ sizes[dim] = ((sizes[dim] as f32 - size as f32) / step as f32 + 1.) as usize;+ strides[dim] *= step;+ }++ let tensor_ = Tensor_ {+ id: TensorId::new(),+ storage: self.storage.clone(),+ layout: Layout::new(sizes.into(), strides, self.layout.start_offset()),+ op: BackpropOp::new1(self, Op::Reshape),+ is_variable: false,+ dtype: self.dtype,+ device: self.device.clone(),+ };+ Ok(Tensor(Arc::new(tensor_)))+ }+}++macro_rules! bin_trait {+ ($trait:ident, $fn1:ident, $mul:expr, $add:expr) => {+ impl<B: std::borrow::Borrow<Tensor>> std::ops::$trait<B> for Tensor {+ type Output = Result<Tensor>;++ fn $fn1(self, rhs: B) -> Self::Output {+ Tensor::$fn1(&self, rhs.borrow())+ }+ }++ impl<B: std::borrow::Borrow<Tensor>> std::ops::$trait<B> for &Tensor {+ type Output = Result<Tensor>;++ fn $fn1(self, rhs: B) -> Self::Output {+ Tensor::$fn1(&self, rhs.borrow())+ }+ }++ impl<B: std::borrow::Borrow<Tensor>> std::ops::$trait<Tensor> for Result<B> {+ type Output = Result<Tensor>;++ fn $fn1(self, rhs: Tensor) -> Self::Output {+ Tensor::$fn1(self?.borrow(), &rhs)+ }+ }++ impl<B: std::borrow::Borrow<Tensor>> std::ops::$trait<&Tensor> for Result<B> {+ type Output = Result<Tensor>;++ fn $fn1(self, rhs: &Tensor) -> Self::Output {+ Tensor::$fn1(self?.borrow(), rhs)+ }+ }++ impl<B: std::borrow::Borrow<Tensor>> std::ops::$trait<Result<B>> for Tensor {+ type Output = Result<Tensor>;++ fn $fn1(self, rhs: Result<B>) -> Self::Output {+ Tensor::$fn1(&self, rhs?.borrow())+ }+ }++ impl<B: std::borrow::Borrow<Tensor>> std::ops::$trait<Result<B>> for &Tensor {+ type Output = Result<Tensor>;++ fn $fn1(self, rhs: Result<B>) -> Self::Output {+ Tensor::$fn1(&self, rhs?.borrow())+ }+ }++ impl std::ops::$trait<f64> for Tensor {+ type Output = Result<Tensor>;++ fn $fn1(self, rhs: f64) -> Self::Output {+ self.affine($mul(rhs), $add(rhs))+ }+ }++ impl std::ops::$trait<f64> for &Tensor {+ type Output = Result<Tensor>;++ fn $fn1(self, rhs: f64) -> Self::Output {+ self.affine($mul(rhs), $add(rhs))+ }+ }+ };+}++bin_trait!(Add, add, |_| 1., |v| v);+bin_trait!(Sub, sub, |_| 1., |v: f64| -v);+bin_trait!(Mul, mul, |v| v, |_| 0.);+bin_trait!(Div, div, |v| 1. / v, |_| 0.);++impl std::ops::Add<Tensor> for f64 {+ type Output = Result<Tensor>;++ fn add(self, rhs: Tensor) -> Self::Output {+ rhs + self+ }+}++impl std::ops::Add<&Tensor> for f64 {+ type Output = Result<Tensor>;++ fn add(self, rhs: &Tensor) -> Self::Output {+ rhs + self+ }+}++impl std::ops::Mul<Tensor> for f64 {+ type Output = Result<Tensor>;++ fn mul(self, rhs: Tensor) -> Self::Output {+ rhs * self+ }+}++impl std::ops::Mul<&Tensor> for f64 {+ type Output = Result<Tensor>;++ fn mul(self, rhs: &Tensor) -> Self::Output {+ rhs * self+ }+}++impl std::ops::Sub<Tensor> for f64 {+ type Output = Result<Tensor>;++ fn sub(self, rhs: Tensor) -> Self::Output {+ rhs.affine(-1., self)+ }+}++impl std::ops::Sub<&Tensor> for f64 {+ type Output = Result<Tensor>;++ fn sub(self, rhs: &Tensor) -> Self::Output {+ rhs.affine(-1., self)+ }+}++impl std::ops::Div<Tensor> for f64 {+ type Output = Result<Tensor>;++ #[allow(clippy::suspicious_arithmetic_impl)]+ fn div(self, rhs: Tensor) -> Self::Output {+ rhs.recip()? * self+ }+}++impl std::ops::Div<&Tensor> for f64 {+ type Output = Result<Tensor>;++ #[allow(clippy::suspicious_arithmetic_impl)]+ fn div(self, rhs: &Tensor) -> Self::Output {+ rhs.recip()? * self+ }+}++impl<S: Into<Shape>> From<(Storage, S)> for Tensor {+ fn from((storage, shape): (Storage, S)) -> Self {+ from_storage(storage, shape, BackpropOp::none(), false)+ }+}diff --git a/vendor/candle-core/src/tensor_cat.rs b/vendor/candle-core/src/tensor_cat.rsnew file mode 100644index 0000000..165af81--- /dev/null+++ b/vendor/candle-core/src/tensor_cat.rs@@ -0,0 +1,304 @@+use crate::{shape::Dim, Context, Error, Result, Shape, Tensor};++impl Tensor {+ /// Concatenates two or more tensors along a particular dimension.+ ///+ /// All tensors must of the same rank, and the output will have+ /// the same rank+ ///+ /// ```rust+ /// # use candle_core::{Tensor, DType, Device};+ /// let a = Tensor::zeros((2, 3), DType::F32, &Device::Cpu)?;+ /// let b = Tensor::zeros((2, 3), DType::F32, &Device::Cpu)?;+ ///+ /// let c = Tensor::cat(&[&a, &b], 0)?;+ /// assert_eq!(c.shape().dims(), &[4, 3]);+ ///+ /// let c = Tensor::cat(&[&a, &b], 1)?;+ /// assert_eq!(c.shape().dims(), &[2, 6]);+ /// # Ok::<(), candle_core::Error>(())+ /// ```+ pub fn cat<A: AsRef<Tensor>, D: Dim>(args: &[A], dim: D) -> Result<Self> {+ if args.is_empty() {+ Err(Error::OpRequiresAtLeastOneTensor { op: "cat" }.bt())?+ }+ let arg0 = args[0].as_ref();+ if args.len() == 1 {+ return Ok(arg0.clone());+ }+ let dim = dim.to_index(arg0.shape(), "cat")?;+ for arg in args {+ arg.as_ref().check_dim(dim, "cat")?;+ }+ for (arg_idx, arg) in args.iter().enumerate() {+ let arg = arg.as_ref();+ if arg0.rank() != arg.rank() {+ Err(Error::UnexpectedNumberOfDims {+ expected: arg0.rank(),+ got: arg.rank(),+ shape: arg.shape().clone(),+ }+ .bt())?+ }+ for (dim_idx, (v1, v2)) in arg0+ .shape()+ .dims()+ .iter()+ .zip(arg.shape().dims().iter())+ .enumerate()+ {+ if dim_idx != dim && v1 != v2 {+ Err(Error::ShapeMismatchCat {+ dim: dim_idx,+ first_shape: arg0.shape().clone(),+ n: arg_idx + 1,+ nth_shape: arg.shape().clone(),+ }+ .bt())?+ }+ }+ }+ let all_copy2d_compat = args.iter().all(|v| {+ let t = v.as_ref();+ t.is_contiguous() || t.layout().outer_stride_for_dim(dim).is_some()+ });+ if all_copy2d_compat {+ Self::cat_contiguous(args, dim)+ } else if dim == 0 {+ Self::cat0(args)+ } else {+ let args: Vec<Tensor> = args+ .iter()+ .map(|a| a.as_ref().transpose(0, dim))+ .collect::<Result<Vec<_>>>()?;+ let cat = Self::cat0(&args)?;+ cat.transpose(0, dim)+ }+ }++ fn cat0<A: AsRef<Tensor>>(args: &[A]) -> Result<Self> {+ if args.is_empty() {+ Err(Error::OpRequiresAtLeastOneTensor { op: "cat" }.bt())?+ }+ let arg0 = args[0].as_ref();+ if args.len() == 1 {+ return Ok(arg0.clone());+ }+ let rank = arg0.rank();+ let device = arg0.device();+ let dtype = arg0.dtype();+ let first_dims = arg0.shape().dims();+ let mut cat_dims = first_dims.to_vec();+ cat_dims[0] = 0;+ let mut offsets = vec![0usize];+ for (arg_idx, arg) in args.iter().enumerate() {+ let arg = arg.as_ref();+ if arg.dtype() != dtype {+ Err(Error::DTypeMismatchBinaryOp {+ lhs: dtype,+ rhs: arg.dtype(),+ op: "cat",+ }+ .bt())?+ }+ if arg.device().location() != device.location() {+ Err(Error::DeviceMismatchBinaryOp {+ lhs: device.location(),+ rhs: arg.device().location(),+ op: "cat",+ }+ .bt())?+ }+ if rank != arg.rank() {+ Err(Error::UnexpectedNumberOfDims {+ expected: rank,+ got: arg.rank(),+ shape: arg.shape().clone(),+ }+ .bt())?+ }+ for (dim_idx, (v1, v2)) in arg0+ .shape()+ .dims()+ .iter()+ .zip(arg.shape().dims().iter())+ .enumerate()+ {+ if dim_idx == 0 {+ cat_dims[0] += v2;+ }+ if dim_idx != 0 && v1 != v2 {+ Err(Error::ShapeMismatchCat {+ dim: dim_idx,+ first_shape: arg0.shape().clone(),+ n: arg_idx + 1,+ nth_shape: arg.shape().clone(),+ }+ .bt())?+ }+ }+ let next_offset = offsets.last().context("empty offsets")? + arg.elem_count();+ offsets.push(next_offset);+ }+ let shape = Shape::from(cat_dims);+ let op = crate::op::BackpropOp::new(args, |args| crate::op::Op::Cat(args, 0));+ let mut storage = unsafe { device.alloc_uninit(&shape, dtype)? };+ for (arg, &offset) in args.iter().zip(offsets.iter()) {+ let arg = arg.as_ref();+ arg.storage()+ .copy_strided_src(&mut storage, offset, arg.layout())?;+ }+ Ok(crate::tensor::from_storage(storage, shape, op, false))+ }++ fn cat_contiguous<A: AsRef<Tensor>>(args: &[A], dim: usize) -> Result<Self> {+ if args.is_empty() {+ Err(Error::OpRequiresAtLeastOneTensor { op: "cat" }.bt())?+ }+ let arg0 = args[0].as_ref();+ if args.len() == 1 {+ return Ok(arg0.clone());+ }+ let rank = arg0.rank();+ let device = arg0.device();+ let dtype = arg0.dtype();+ let first_dims = arg0.shape().dims();+ let mut cat_dims = first_dims.to_vec();+ cat_dims[dim] = 0;+ for (arg_idx, arg) in args.iter().enumerate() {+ let arg = arg.as_ref();+ if arg.dtype() != dtype {+ Err(Error::DTypeMismatchBinaryOp {+ lhs: dtype,+ rhs: arg.dtype(),+ op: "cat",+ }+ .bt())?+ }+ if arg.device().location() != device.location() {+ Err(Error::DeviceMismatchBinaryOp {+ lhs: device.location(),+ rhs: arg.device().location(),+ op: "cat",+ }+ .bt())?+ }+ if rank != arg.rank() {+ Err(Error::UnexpectedNumberOfDims {+ expected: rank,+ got: arg.rank(),+ shape: arg.shape().clone(),+ }+ .bt())?+ }+ for (dim_idx, (v1, v2)) in arg0+ .shape()+ .dims()+ .iter()+ .zip(arg.shape().dims().iter())+ .enumerate()+ {+ if dim_idx == dim {+ cat_dims[dim] += v2;+ }+ if dim_idx != dim && v1 != v2 {+ Err(Error::ShapeMismatchCat {+ dim: dim_idx,+ first_shape: arg0.shape().clone(),+ n: arg_idx + 1,+ nth_shape: arg.shape().clone(),+ }+ .bt())?+ }+ }+ }+ let cat_target_dim_len = cat_dims[dim];+ let block_size: usize = cat_dims.iter().skip(1 + dim).product();+ let shape = Shape::from(cat_dims);+ let op = crate::op::BackpropOp::new(args, |args| crate::op::Op::Cat(args, dim));+ let mut storage = unsafe { device.alloc_uninit(&shape, dtype)? };+ let mut dst_o = 0;+ for arg in args.iter() {+ let arg = arg.as_ref();+ let arg_dims = arg.shape().dims();+ let d1: usize = arg_dims.iter().take(dim).product();+ let d2 = block_size * arg_dims[dim];+ let dst_s = block_size * cat_target_dim_len;+ let src_o = arg.layout().start_offset();+ let src_s = if arg.is_contiguous() {+ d2+ } else {+ arg.layout().outer_stride_for_dim(dim).unwrap_or(d2)+ };+ arg.storage()+ .copy2d(&mut storage, d1, d2, src_s, dst_s, src_o, dst_o)?;+ dst_o += d2;+ }+ Ok(crate::tensor::from_storage(storage, shape, op, false))+ }++ /// Set the values on `self` using values from `src`. The copy starts at the specified+ /// `offset` for the target dimension `dim` on `self`.+ /// `self` and `src` must have the same shape except on dimension `dim` where the `self` size+ /// has to be greater than or equal to `offset` plus the `src` size.+ ///+ /// Note that this modifies `self` in place and as such is not compatible with+ /// back-propagation.+ pub fn slice_set<D: Dim>(&self, src: &Self, dim: D, offset: usize) -> Result<()> {+ let dim = dim.to_index(self.shape(), "slice-set")?;+ if !self.is_contiguous() || !src.is_contiguous() {+ Err(Error::RequiresContiguous { op: "slice-set" }.bt())?+ }+ if self.same_storage(src) {+ crate::bail!("cannot use slice_set when self and src share their storage")+ }+ if self.dtype() != src.dtype() {+ Err(Error::DTypeMismatchBinaryOp {+ lhs: self.dtype(),+ rhs: src.dtype(),+ op: "slice-set",+ }+ .bt())?+ }+ if self.device().location() != src.device().location() {+ Err(Error::DeviceMismatchBinaryOp {+ lhs: self.device().location(),+ rhs: src.device().location(),+ op: "slice-set",+ }+ .bt())?+ }+ if self.rank() != src.rank() {+ Err(Error::UnexpectedNumberOfDims {+ expected: self.rank(),+ got: src.rank(),+ shape: self.shape().clone(),+ }+ .bt())?+ }+ for (dim_idx, (v1, v2)) in self.dims().iter().zip(src.dims().iter()).enumerate() {+ if dim_idx == dim && *v2 + offset > *v1 {+ crate::bail!("shape mismatch on target dim, dst: {v1}, src: {v2} + {offset}")+ }+ if dim_idx != dim && v1 != v2 {+ crate::bail!("shape mismatch on dim {dim_idx}, {v1} <> {v2}")+ }+ }+ let block_size: usize = src.dims().iter().skip(1 + dim).product();+ let d1: usize = src.dims().iter().take(dim).product();+ let d2 = block_size * src.dims()[dim];+ let dst_o = self.layout().start_offset() + offset * block_size;+ let src_o = src.layout().start_offset();+ src.storage().copy2d(+ &mut self.storage_mut(),+ d1,+ d2,+ /* src_s */ d2,+ /* dst_s */ block_size * self.dims()[dim],+ src_o,+ dst_o,+ )?;++ Ok(())+ }+}diff --git a/vendor/candle-core/src/test_utils.rs b/vendor/candle-core/src/test_utils.rsnew file mode 100644index 0000000..e331399--- /dev/null+++ b/vendor/candle-core/src/test_utils.rs@@ -0,0 +1,71 @@+use crate::{Result, Tensor};++#[macro_export]+macro_rules! test_device {+ // TODO: Switch to generating the two last arguments automatically once concat_idents is+ // stable. https://github.com/rust-lang/rust/issues/29599+ ($fn_name: ident, $test_cpu: ident, $test_cuda: ident, $test_metal: ident) => {+ #[test]+ fn $test_cpu() -> Result<()> {+ $fn_name(&Device::Cpu)+ }++ #[cfg(feature = "cuda")]+ #[test]+ fn $test_cuda() -> Result<()> {+ $fn_name(&Device::new_cuda(0)?)+ }++ #[cfg(feature = "metal")]+ #[test]+ fn $test_metal() -> Result<()> {+ $fn_name(&Device::new_metal(0)?)+ }+ };+}++pub fn assert_tensor_eq(t1: &Tensor, t2: &Tensor) -> Result<()> {+ assert_eq!(t1.shape(), t2.shape());+ // Default U8 may not be large enough to hold the sum (`t.sum_all` defaults to the dtype of `t`)+ let eq_tensor = t1.eq(t2)?.to_dtype(crate::DType::U32)?;+ let all_equal = eq_tensor.sum_all()?;+ assert_eq!(all_equal.to_scalar::<u32>()?, eq_tensor.elem_count() as u32);+ Ok(())+}++pub fn to_vec0_round(t: &Tensor, digits: i32) -> Result<f32> {+ let b = 10f32.powi(digits);+ let t = t.to_vec0::<f32>()?;+ Ok(f32::round(t * b) / b)+}++pub fn to_vec1_round(t: &Tensor, digits: i32) -> Result<Vec<f32>> {+ let b = 10f32.powi(digits);+ let t = t.to_vec1::<f32>()?;+ let t = t.iter().map(|t| f32::round(t * b) / b).collect();+ Ok(t)+}++pub fn to_vec2_round(t: &Tensor, digits: i32) -> Result<Vec<Vec<f32>>> {+ let b = 10f32.powi(digits);+ let t = t.to_vec2::<f32>()?;+ let t = t+ .iter()+ .map(|t| t.iter().map(|t| f32::round(t * b) / b).collect())+ .collect();+ Ok(t)+}++pub fn to_vec3_round(t: &Tensor, digits: i32) -> Result<Vec<Vec<Vec<f32>>>> {+ let b = 10f32.powi(digits);+ let t = t.to_vec3::<f32>()?;+ let t = t+ .iter()+ .map(|t| {+ t.iter()+ .map(|t| t.iter().map(|t| f32::round(t * b) / b).collect())+ .collect()+ })+ .collect();+ Ok(t)+}diff --git a/vendor/candle-core/src/utils.rs b/vendor/candle-core/src/utils.rsnew file mode 100644index 0000000..324e623--- /dev/null+++ b/vendor/candle-core/src/utils.rs@@ -0,0 +1,637 @@+//! Useful functions for checking features.+use std::any::Any;+use std::cell::UnsafeCell;+use std::hint::spin_loop;+use std::str::FromStr;+use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};+use std::sync::{Mutex, OnceLock};+use std::thread::Thread;++const DEFAULT_BARRIER_POOL_SPIN_LIMIT: u32 = 25_000;++/// Per-worker control on its own 64-byte cache line to prevent false sharing.+#[repr(C, align(64))]+struct Slot {+ /// `Release`-stored by parent to signal work. `Acquire-loaded by worker.+ go: AtomicUsize,+ /// `Release`-stored by worker (and its subtree) when done. `Acquire`-loaded by parent.+ done: AtomicUsize,+}++/// Closure pointer written by main before the `Release` on each root slot's `go`.+struct WorkDesc {+ trampoline: unsafe fn(*const (), usize),+ data: *const (),+}++struct BarrierPoolInner {+ slots: Box<[Slot]>,+ /// Thread handles for unpark.+ handles: OnceLock<Box<[Thread]>>,+ work: UnsafeCell<WorkDesc>,+ stop: AtomicBool,+ panic_payload: Mutex<Option<Box<dyn Any + Send>>>,+}++unsafe impl Sync for BarrierPoolInner {}+unsafe impl Send for BarrierPoolInner {}++/// Two-level tree barrier pool.+///+/// Main signals one leader per CPU cluster (O(n_clusters) stores, not O(n_workers)).+/// Leaders fan work out to intra-cluster followers while main executes its own+/// slice. On-completion reduction mirrors the fan-out: followers -> leader -> main.+/// Main drains only n_cluster slots instead of n_worker slots on completion.+pub struct BarrierPool {+ inner: Box<BarrierPoolInner>,+ threads: Vec<std::thread::JoinHandle<()>>,+ /// Workers that main signals directly+ root_workers: Vec<usize>,+ /// Serializes concurrent callers. Stores the current generation.+ call_lock: Mutex<usize>,+}++unsafe impl Sync for BarrierPool {}+unsafe impl Send for BarrierPool {}++thread_local! {+ static IN_BARRIER_POOL: std::cell::Cell<bool> = const { std::cell::Cell::new(false) };+}++unsafe fn call_trampoline<F: Fn(usize)>(data: *const (), tid: usize) {+ (*(data as *const F))(tid);+}++/// Build the two-level fan-out tree. First worker in each cluster is leader.+fn compute_tree(n_workers: usize, cluster_size: usize) -> (Vec<Vec<usize>>, Vec<usize>) {+ let mut children = vec![vec![]; n_workers];+ if n_workers == 0 || cluster_size <= 1 || cluster_size == usize::MAX {+ // Flat: every worker is a root.+ return (children, (0..n_workers).collect());+ }++ let mut roots = Vec::new();++ // First cluster: main + (cluster_size - 1) workers.+ let c0 = (cluster_size - 1).min(n_workers);+ roots.push(0);+ for f in 1..c0 {+ children[0].push(f);+ }++ // Remaining clusters: cluster_size workers each.+ let mut next = c0;+ while next < n_workers {+ let leader = next;+ roots.push(leader);+ let end = n_workers.min(next + cluster_size);+ for f in (leader + 1)..end {+ children[leader].push(f);+ }+ next = end;+ }++ (children, roots)+}++fn get_cluster_size() -> usize {+ // Later we should use `sysctlbyname("hw.perflevel0.cpusperl2")` and similar for other systems to get precise cluster size.+ // For now we use these hardcoded numbers.+ // Using `usize::MAX` simply means that the tree is flat. Every core is it's own leader.+ #[cfg(target_os = "macos")]+ return 6;+ #[cfg(not(target_os = "macos"))]+ usize::MAX+}++fn barrier_pool_spin_limit() -> u32 {+ std::env::var("CANDLE_BARRIER_POOL_SPIN_LIMIT")+ .ok()+ .and_then(|s| u32::from_str(&s).ok())+ .unwrap_or(DEFAULT_BARRIER_POOL_SPIN_LIMIT)+}++impl BarrierPool {+ fn new(n_workers: usize) -> Self {+ let inner = Box::new(BarrierPoolInner {+ slots: (0..n_workers)+ .map(|_| Slot {+ go: AtomicUsize::new(0),+ done: AtomicUsize::new(0),+ })+ .collect::<Vec<_>>()+ .into_boxed_slice(),+ handles: OnceLock::new(),+ work: UnsafeCell::new(WorkDesc {+ trampoline: |_, _| {},+ data: std::ptr::null(),+ }),+ stop: AtomicBool::new(false),+ panic_payload: Mutex::new(None),+ });++ let inner_ptr = &*inner as *const BarrierPoolInner as usize;+ let cluster_size = get_cluster_size();+ let (children, root_workers) = compute_tree(n_workers, cluster_size);+ let spin_limit = barrier_pool_spin_limit();++ let threads: Vec<_> = (0..n_workers)+ .map(|tid| {+ let children = children[tid].clone();+ std::thread::Builder::new()+ .name(format!("candle-bp-{tid}"))+ .spawn(move || {+ IN_BARRIER_POOL.with(|f| f.set(true));+ set_thread_affinity();+ let inner = unsafe { &*(inner_ptr as *const BarrierPoolInner) };+ let slot = &inner.slots[tid];+ let mut gen = 0usize;++ loop {+ // Spin briefly then park, waiting for next work item.+ let mut spins = 0u32;+ loop {+ let g = slot.go.load(Ordering::Acquire);+ if g != gen {+ gen = g;+ break;+ }+ if inner.stop.load(Ordering::Relaxed) {+ return;+ }+ spins += 1;+ if spins < spin_limit {+ spin_loop();+ } else {+ // Park deposits a token; unpark() before park() returns immediately — no race.+ std::thread::park();+ spins = 0;+ }+ }+ if inner.stop.load(Ordering::Relaxed) {+ break;+ }++ // Fan work out to children.+ let handles = inner.handles.get().unwrap();+ for &child in &children {+ inner.slots[child].go.store(gen, Ordering::Release);+ handles[child].unpark();+ }++ // Execute own work, catching panics so done is always signalled.+ if let Err(payload) =+ std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {+ let work = unsafe { &*inner.work.get() };+ unsafe { (work.trampoline)(work.data, tid) };+ }))+ {+ let mut p = inner.panic_payload.lock().unwrap();+ if p.is_none() {+ *p = Some(payload);+ }+ }++ // Reduce: wait for all children before signalling parent.+ for &child in &children {+ while inner.slots[child].done.load(Ordering::Acquire) != gen {+ spin_loop();+ }+ }++ // Signal done unconditionally so main/parent never deadlocks.+ slot.done.store(gen, Ordering::Release);+ }+ })+ .expect("failed to spawn candle barrier pool worker")+ })+ .collect();++ inner+ .handles+ .set(+ threads+ .iter()+ .map(|jh| jh.thread().clone())+ .collect::<Vec<_>>()+ .into_boxed_slice(),+ )+ .ok();+ BarrierPool {+ inner,+ threads,+ root_workers,+ call_lock: Mutex::new(0),+ }+ }++ #[inline]+ pub fn n_workers(&self) -> usize {+ self.threads.len()+ }+}++/// RAII guard that blocks until all root workers have finished.+struct BarrierGuard<'a> {+ inner: &'a BarrierPoolInner,+ roots: &'a [usize],+ gen: usize,+}++impl Drop for BarrierGuard<'_> {+ fn drop(&mut self) {+ for &root in self.roots {+ while self.inner.slots[root].done.load(Ordering::Acquire) != self.gen {+ spin_loop();+ }+ }+ // Always clear any stored panic. Propagate unless main is not already unwinding.+ let payload = self.inner.panic_payload.lock().unwrap().take();+ if !std::thread::panicking() {+ if let Some(p) = payload {+ std::panic::resume_unwind(p);+ }+ }+ }+}++impl BarrierPool {+ pub fn execute<F: Fn(usize) + Sync>(&self, f: F) {+ let n = self.threads.len();+ if n == 0 {+ f(0);+ return;+ }++ let mut guard = self.call_lock.lock().unwrap();+ let new_gen = guard.wrapping_add(1);+ *guard = new_gen;++ unsafe {+ let work = &mut *self.inner.work.get();+ work.trampoline = call_trampoline::<F>;+ work.data = &f as *const F as *const ();+ }++ let handles = self.inner.handles.get().unwrap();+ for &root in &self.root_workers {+ self.inner.slots[root].go.store(new_gen, Ordering::Release);+ handles[root].unpark();+ }++ // Guard waits for workers and propagates panics on drop.+ let _guard = BarrierGuard {+ inner: &self.inner,+ roots: &self.root_workers,+ gen: new_gen,+ };++ // Main thread participates as worker n (workers are 0..n-1).+ f(n);+ // _guard drops here, waiting for root workers.+ }++ // Uniform, short work units can avoid shared-cursor contention by assigning+ // one contiguous range per thread.+ pub(crate) fn execute_static<F: Fn(std::ops::Range<usize>) + Sync>(+ &self,+ n_items: usize,+ f: F,+ ) {+ if n_items == 0 {+ return;+ }+ if IN_BARRIER_POOL.with(|f| f.get()) {+ f(0..n_items);+ return;+ }+ let chunk = n_items.div_ceil(self.threads.len() + 1);+ self.execute(|tid| {+ let start = tid * chunk;+ if start < n_items {+ f(start..n_items.min(start + chunk));+ }+ });+ }++ // Static per-tid slices make every op wait for the slowest thread; a shared cursor+ // lets fast threads absorb a straggler's work instead of spinning at the barrier.+ pub fn execute_chunked<F: Fn(std::ops::Range<usize>) + Sync>(&self, n_items: usize, f: F) {+ if n_items == 0 {+ return;+ }+ // nested use from a worker would deadlock on the call lock; run inline instead+ if IN_BARRIER_POOL.with(|f| f.get()) {+ f(0..n_items);+ return;+ }+ let n_threads = self.threads.len() + 1;+ // ~8 grabs per thread amortizes cursor contention while keeping tail imbalance small+ let chunk = n_items.div_ceil(n_threads * 8).max(1);+ let cursor = AtomicUsize::new(0);+ self.execute(|_| loop {+ let start = cursor.fetch_add(chunk, Ordering::Relaxed);+ if start >= n_items {+ break;+ }+ f(start..n_items.min(start + chunk));+ });+ }+}++impl Drop for BarrierPool {+ fn drop(&mut self) {+ if self.threads.is_empty() {+ return;+ }+ self.inner.stop.store(true, Ordering::Relaxed);+ let new_gen = self.call_lock.lock().unwrap().wrapping_add(1);+ let handles = self.inner.handles.get();+ // Signal ALL slots and unpark them so parked workers wake immediately.+ for (i, slot) in self.inner.slots.iter().enumerate() {+ slot.go.store(new_gen, Ordering::Release);+ if let Some(hs) = handles {+ hs[i].unpark();+ }+ }+ for t in self.threads.drain(..) {+ let _ = t.join();+ }+ }+}++static BARRIER_POOL: OnceLock<BarrierPool> = OnceLock::new();++/// Persistent barrier pool+pub fn barrier_pool() -> &'static BarrierPool {+ BARRIER_POOL.get_or_init(|| BarrierPool::new(candle_num_threads().saturating_sub(1)))+}++pub fn with_threadpool<F: FnOnce() -> R + Send, R: Send>(f: F) -> R {+ candle_pool().install(f)+}++pub fn init_global_threadpool() {+ set_thread_affinity();+ let _ = rayon::ThreadPoolBuilder::new()+ .num_threads(get_num_threads())+ .start_handler(|_| set_thread_affinity())+ .build_global();+}++fn default_num_threads() -> usize {+ let physical = {+ #[cfg(target_os = "macos")]+ {+ perf_core_count().unwrap_or_else(num_cpus::get_physical)+ }+ #[cfg(target_os = "linux")]+ {+ linux_env_cpu_mask()+ .or_else(linux_preferred_cpus)+ .map(<[_]>::len)+ .unwrap_or_else(num_cpus::get_physical)+ }+ #[cfg(not(any(target_os = "linux", target_os = "macos")))]+ {+ num_cpus::get_physical()+ }+ };+ physical.max(1) // safeguard against bad number+}++fn rayon_num_threads() -> usize {+ std::env::var("RAYON_NUM_THREADS")+ .ok()+ .and_then(|s| usize::from_str(&s).ok())+ .filter(|nt| nt > &0)+ .unwrap_or_else(default_num_threads)+}++fn candle_num_threads() -> usize {+ std::env::var("CANDLE_NUM_THREADS")+ .ok()+ .and_then(|s| usize::from_str(&s).ok())+ .filter(|nt| nt > &0)+ .unwrap_or_else(rayon_num_threads)+}++pub fn get_num_threads() -> usize {+ // Respond to the same environment variable as rayon.+ rayon_num_threads()+}++/// On Apple Silicon: P-core count via `hw.perflevel0.logicalcpu`.+/// Returns None on Intel Macs (key absent) or on error.+#[cfg(target_os = "macos")]+fn perf_core_count() -> Option<usize> {+ use std::os::raw::c_void;+ let mut count: u32 = 0;+ let mut size = std::mem::size_of::<u32>();+ let ret = unsafe {+ libc::sysctlbyname(+ c"hw.perflevel0.logicalcpu".as_ptr().cast(),+ &mut count as *mut u32 as *mut c_void,+ &mut size,+ std::ptr::null_mut(),+ 0,+ )+ };+ (ret == 0 && count > 0).then_some(count as usize)+}++#[cfg(target_os = "linux")]+fn linux_preferred_cpus() -> Option<&'static [usize]> {+ static CPUS: OnceLock<Option<Vec<usize>>> = OnceLock::new();+ CPUS.get_or_init(linux_detect_preferred_cpus).as_deref()+}++#[cfg(target_os = "linux")]+fn linux_env_cpu_mask() -> Option<&'static [usize]> {+ static CPUS: OnceLock<Option<Vec<usize>>> = OnceLock::new();+ CPUS.get_or_init(|| {+ std::env::var("CANDLE_CPU_MASK")+ .ok()+ .and_then(|mask| linux_parse_cpu_list(&mask))+ })+ .as_deref()+}++#[cfg(target_os = "linux")]+fn linux_affinity_enabled() -> bool {+ std::env::var("CANDLE_CPU_AFFINITY")+ .map(|v| v != "0" && !v.eq_ignore_ascii_case("false"))+ .unwrap_or(false)+}++#[cfg(target_os = "linux")]+fn linux_affinity_cpus() -> Option<&'static [usize]> {+ linux_env_cpu_mask().or_else(|| {+ linux_affinity_enabled()+ .then(linux_preferred_cpus)+ .flatten()+ })+}++#[cfg(target_os = "linux")]+fn linux_parse_cpu_list(mask: &str) -> Option<Vec<usize>> {+ let mut cpus = Vec::new();+ for part in mask+ .split(',')+ .map(str::trim)+ .filter(|part| !part.is_empty())+ {+ let (start, end) = if let Some((start, end)) = part.split_once('-') {+ let start = start.trim().parse::<usize>().ok()?;+ let end = end.trim().parse::<usize>().ok()?;+ (start, end)+ } else {+ let cpu = part.parse::<usize>().ok()?;+ (cpu, cpu)+ };+ if start > end {+ return None;+ }+ cpus.extend(start..=end);+ }+ cpus.sort_unstable();+ cpus.dedup();+ (!cpus.is_empty()).then_some(cpus)+}++#[cfg(target_os = "linux")]+fn linux_detect_preferred_cpus() -> Option<Vec<usize>> {+ struct Cpu {+ idx: usize,+ capacity: usize,+ }++ let mut cpus = Vec::new();+ for entry in std::fs::read_dir("/sys/devices/system/cpu").ok()?.flatten() {+ let cpu_name = entry.file_name();+ let cpu_name = cpu_name.to_str()?;+ let Some(idx) = cpu_name+ .strip_prefix("cpu")+ .and_then(|idx| idx.parse::<usize>().ok())+ else {+ continue;+ };+ let path = entry.path();+ let capacity = std::fs::read_to_string(path.join("cpu_capacity"))+ .ok()?+ .trim()+ .parse::<usize>()+ .ok()?;+ cpus.push(Cpu { idx, capacity });+ }+ if cpus.len() < 2 {+ return None;+ }+ let min = cpus.iter().map(|cpu| cpu.capacity).min()?;+ let max = cpus.iter().map(|cpu| cpu.capacity).max()?;+ if max == 0 || min * 10 >= max * 9 {+ return None;+ }+ let total_cpus = cpus.len();+ let mut preferred = cpus+ .iter()+ .filter(|cpu| cpu.capacity * 10 >= max * 9)+ .map(|cpu| cpu.idx)+ .collect::<Vec<_>>();+ preferred.sort_unstable();+ (preferred.len() < total_cpus).then_some(preferred)+}++static POOL: OnceLock<rayon::ThreadPool> = OnceLock::new();++pub(crate) fn candle_pool() -> &'static rayon::ThreadPool {+ POOL.get_or_init(|| {+ rayon::ThreadPoolBuilder::new()+ .num_threads(get_num_threads())+ .start_handler(|_| set_thread_affinity())+ .build()+ .expect("failed to build candle rayon threadpool")+ })+}++#[cfg(target_os = "macos")]+/// Elevate the thread QoS so macOS prefers running on Performance (P) cores.+pub fn set_thread_affinity() {+ use libc::{pthread_set_qos_class_self_np, qos_class_t::QOS_CLASS_USER_INTERACTIVE};+ unsafe {+ pthread_set_qos_class_self_np(QOS_CLASS_USER_INTERACTIVE, 0);+ }+}++#[cfg(not(target_os = "macos"))]+#[inline(always)]+pub fn set_thread_affinity() {+ #[cfg(target_os = "linux")]+ if let Some(cpus) = linux_affinity_cpus() {+ unsafe {+ let mut set: libc::cpu_set_t = std::mem::zeroed();+ libc::CPU_ZERO(&mut set);+ for &cpu in cpus {+ libc::CPU_SET(cpu, &mut set);+ }+ let _ = libc::pthread_setaffinity_np(+ libc::pthread_self(),+ std::mem::size_of::<libc::cpu_set_t>(),+ &set,+ );+ }+ }+}++pub fn has_accelerate() -> bool {+ cfg!(feature = "accelerate")+}++pub fn has_mkl() -> bool {+ cfg!(feature = "mkl")+}++pub fn cuda_is_available() -> bool {+ cfg!(feature = "cuda")+}++pub fn metal_is_available() -> bool {+ cfg!(feature = "metal")+}++pub fn with_avx() -> bool {+ crate::cpu::features::get().avx2+}++pub fn with_neon() -> bool {+ crate::cpu::features::get().neon+}++pub fn with_simd128() -> bool {+ cfg!(target_feature = "simd128")+}++pub fn with_f16c() -> bool {+ crate::cpu::features::get().f16c+}++#[cfg(all(test, target_os = "linux"))]+mod tests {+ use super::linux_parse_cpu_list;++ #[test]+ fn parses_cpu_list() {+ assert_eq!(+ linux_parse_cpu_list("5-9,15,17-19"),+ Some(vec![5, 6, 7, 8, 9, 15, 17, 18, 19])+ );+ assert_eq!(linux_parse_cpu_list(" 3 , 1-2,2 "), Some(vec![1, 2, 3]));+ }++ #[test]+ fn rejects_invalid_cpu_list() {+ assert_eq!(linux_parse_cpu_list(""), None);+ assert_eq!(linux_parse_cpu_list("9-5"), None);+ assert_eq!(linux_parse_cpu_list("5,a"), None);+ }+}diff --git a/vendor/candle-core/src/variable.rs b/vendor/candle-core/src/variable.rsnew file mode 100644index 0000000..1e4880e--- /dev/null+++ b/vendor/candle-core/src/variable.rs@@ -0,0 +1,152 @@+// Variables are wrappers around tensors that can be modified, they are typically used for holding+// weights and being modified by gradient descent.+// We do not expose a public way to create variables as this would break the invariant that the+// tensor within a variable is actually with `is_variable` set to `true`.+use crate::{DType, Device, Error, Result, Shape, Tensor};++/// A variable is a wrapper around a tensor, however variables can have their content modified+/// whereas tensors are immutable.+#[derive(Clone, Debug)]+pub struct Var(Tensor);++impl std::fmt::Display for Var {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ std::fmt::Display::fmt(&self.0, f)+ }+}++impl std::ops::Deref for Var {+ type Target = Tensor;++ fn deref(&self) -> &Self::Target {+ self.0.as_ref()+ }+}++impl Var {+ pub fn zeros<S: Into<Shape>>(shape: S, dtype: DType, device: &Device) -> Result<Self> {+ let inner = Tensor::zeros_impl(shape, dtype, device, true)?;+ Ok(Self(inner))+ }++ pub fn ones<S: Into<Shape>>(shape: S, dtype: DType, device: &Device) -> Result<Self> {+ let inner = Tensor::ones_impl(shape, dtype, device, true)?;+ Ok(Self(inner))+ }++ // Convert a tensor to a variable, if the tensor is already a variable then it is returned as is.+ pub fn from_tensor(t: &Tensor) -> Result<Self> {+ if t.is_variable() {+ Ok(Self(t.clone()))+ } else {+ let inner = t.make_var()?;+ Ok(Self(inner))+ }+ }++ pub fn rand_f64<S: Into<Shape>>(+ lo: f64,+ up: f64,+ s: S,+ dtype: DType,+ device: &Device,+ ) -> Result<Self> {+ let inner = Tensor::rand_f64_impl(lo, up, s, dtype, device, true)?;+ Ok(Self(inner))+ }++ pub fn randn_f64<S: Into<Shape>>(+ mean: f64,+ std: f64,+ s: S,+ dtype: DType,+ device: &Device,+ ) -> Result<Self> {+ let inner = Tensor::randn_f64_impl(mean, std, s, dtype, device, true)?;+ Ok(Self(inner))+ }++ pub fn rand<S: Into<Shape>, T: crate::FloatDType>(+ lo: T,+ up: T,+ s: S,+ device: &Device,+ ) -> Result<Self> {+ let inner = Tensor::rand_impl(lo, up, s, device, true)?;+ Ok(Self(inner))+ }++ pub fn randn<S: Into<Shape>, T: crate::FloatDType>(+ mean: T,+ std: T,+ s: S,+ device: &Device,+ ) -> Result<Self> {+ let inner = Tensor::randn_impl(mean, std, s, device, true)?;+ Ok(Self(inner))+ }++ /// Creates a new tensor on the specified device using the content and shape of the input.+ /// This is similar to `new` but the resulting tensor is a variable.+ pub fn new<A: crate::device::NdArray>(array: A, device: &Device) -> Result<Self> {+ let shape = array.shape()?;+ let inner = Tensor::new_impl(array, shape, device, true)?;+ Ok(Self(inner))+ }++ pub fn from_vec<S: Into<Shape>, D: crate::WithDType>(+ data: Vec<D>,+ shape: S,+ device: &Device,+ ) -> Result<Self> {+ let inner = Tensor::from_vec_impl(data, shape, device, true)?;+ Ok(Self(inner))+ }++ pub fn from_slice<S: Into<Shape>, D: crate::WithDType>(+ array: &[D],+ shape: S,+ device: &Device,+ ) -> Result<Self> {+ let inner = Tensor::new_impl(array, shape.into(), device, true)?;+ Ok(Self(inner))+ }++ pub fn as_detached_tensor(&self) -> Tensor {+ self.0.detach()+ }++ pub fn as_tensor(&self) -> &Tensor {+ &self.0+ }++ /// Consumes this `Var` and return the underlying tensor.+ pub fn into_inner(self) -> Tensor {+ self.0+ }++ /// Sets the content of the inner tensor, this does not require a mutable reference as inner+ /// mutability is used.+ pub fn set(&self, src: &Tensor) -> Result<()> {+ if self.same_storage(src) {+ let msg = "cannot set a variable to a tensor that is derived from its value";+ Err(Error::CannotSetVar { msg }.bt())?+ }+ let (mut dst, layout) = self.storage_mut_and_layout();+ if !layout.is_contiguous() {+ let msg = "cannot set a non-contiguous variable";+ Err(Error::CannotSetVar { msg }.bt())?+ }+ let (src, src_l) = src.storage_and_layout();+ if layout.shape() != src_l.shape() {+ Err(Error::ShapeMismatchBinaryOp {+ lhs: layout.shape().clone(),+ rhs: src_l.shape().clone(),+ op: "set",+ }+ .bt())?+ }+ src.copy_strided_src(&mut dst, layout.start_offset(), src_l)?;+ Ok(())+ }+}diff --git a/vendor/candle-metal-kernels/src/kernels/quantized.rs b/vendor/candle-metal-kernels/src/kernels/quantized.rsindex ad73950..6d98684 100644--- a/vendor/candle-metal-kernels/src/kernels/quantized.rs+++ b/vendor/candle-metal-kernels/src/kernels/quantized.rs@@ -22,6 +22,7 @@ pub enum GgmlDType {F16,F32,BF16,+ Mxfp4,}#[allow(clippy::too_many_arguments)]@@ -67,6 +68,7 @@ pub fn call_quantized_matmul_mv_t(| GgmlDType::Q5_0| GgmlDType::Q5_1| GgmlDType::Q8_0+ | GgmlDType::Mxfp4| GgmlDType::Q8_1 => {let nth0 = 8;let nth1 = 8;@@ -129,6 +131,7 @@ pub fn call_quantized_matmul_mv_t(GgmlDType::Q5_0 => "kernel_mul_mv_q5_0_f32",GgmlDType::Q5_1 => "kernel_mul_mv_q5_1_f32",GgmlDType::Q8_0 => "kernel_mul_mv_q8_0_f32",+ GgmlDType::Mxfp4 => "kernel_mul_mv_mxfp4_f32",GgmlDType::Q8_1 => "kernel_mul_mv_q8_1_f32",GgmlDType::Q2K => "kernel_mul_mv_q2_K_f32",GgmlDType::Q3K => "kernel_mul_mv_q3_K_f32",@@ -245,6 +248,7 @@ pub fn call_quantized_matmul_mm_t(GgmlDType::F32 => "kernel_mul_mm_f32_f32",GgmlDType::Q8_1 => Err(MetalKernelError::UnsupportedDTypeForOp("Q8_1", "qmatmul"))?,GgmlDType::Q8K => Err(MetalKernelError::UnsupportedDTypeForOp("Q8K", "qmatmul"))?,+ GgmlDType::Mxfp4 => Err(MetalKernelError::UnsupportedDTypeForOp("Mxfp4", "qmatmul mm"))?,};let pipeline = kernels.load_pipeline(device, Source::Quantized, name)?;@@ -313,6 +317,7 @@ pub fn call_quantized_get_rows(GgmlDType::Q6K => "kernel_get_rows_q6_K",GgmlDType::Q8_1 => Err(MetalKernelError::UnsupportedDTypeForOp("Q8_1", "get_rows"))?,GgmlDType::Q8K => Err(MetalKernelError::UnsupportedDTypeForOp("Q8K", "get_rows"))?,+ GgmlDType::Mxfp4 => Err(MetalKernelError::UnsupportedDTypeForOp("Mxfp4", "get_rows"))?,};let pipeline = kernels.load_pipeline(device, Source::Quantized, name)?;diff --git a/vendor/candle-metal-kernels/src/metal_src/quantized.metal b/vendor/candle-metal-kernels/src/metal_src/quantized.metalindex fcf1037..39af545 100644--- a/vendor/candle-metal-kernels/src/metal_src/quantized.metal+++ b/vendor/candle-metal-kernels/src/metal_src/quantized.metal@@ -2554,6 +2554,129 @@ void kernel_mul_mv_q8_0_f32_impl(}}+#define QK_MXFP4 32+typedef struct {+ uint8_t e;+ uint8_t qs[QK_MXFP4/2];+} block_mxfp4;++constant float kvalues_mxfp4_f[16] = {+ 0, .5f, 1.f, 1.5f, 2.f, 3.f, 4.f, 6.f, -0.f, -.5f, -1.f, -1.5f, -2.f, -3.f, -4.f, -6.f+};++static inline float e8m0_to_fp32(uint8_t x) {+ uint32_t bits = x == 0 ? 0x00400000u : ((uint32_t) x << 23);+ return as_type<float>(bits);+}++// MXFP4 rows times an f32 vector: GGML's inner loop (kernel_mul_mv_mxfp4_f32_impl) in the+// argument shape of the Q8_0 kernel above, with the 16-value ladder read from constant memory.+void kernel_mul_mv_mxfp4_f32_impl(+ device const void * src0,+ device const float * src1,+ device float * dst,+ int64_t ne00,+ int64_t ne01,+ int64_t ne02,+ int64_t ne10,+ int64_t ne12,+ int64_t ne0,+ int64_t ne1,+ uint r2,+ uint r3,+ threadgroup int8_t * shared_values,+ uint3 tgpig,+ uint tiisg,+ uint sgitg) {+ const int nr = N_DST;+ const int nsg = N_SIMDGROUP;++ const int nb = ne00/QK_MXFP4;+ const int r0 = tgpig.x;+ const int r1 = tgpig.y;+ const int im = tgpig.z;++ const int first_row = (r0 * nsg + sgitg) * nr;++ const uint i12 = im%ne12;+ const uint i13 = im/ne12;++ const uint offset0 = first_row * nb + (i12/r2)*(nb*ne01) + (i13/r3)*(nb*ne01*ne02);++ device const block_mxfp4 * x = (device const block_mxfp4 *) src0 + offset0;+ device const float * y = (device const float *) src1 + r1*ne10 + im*ne00*ne1;++ const short ix = tiisg/2;+ const short it = tiisg%2;++ float4 yl[4];+ float sumf[nr]={0.f};++ device const float * yb = y + ix*QK_MXFP4 + it*8;++ for (int ib = ix; ib < nb; ib += 16) {+ device const float4 * y4 = (device const float4 *) yb;++ yl[0] = y4[0];+ yl[1] = y4[4];+ yl[2] = y4[1];+ yl[3] = y4[5];++ for (short row = 0; row < nr; row++) {+ if (first_row + row >= ne01) {+ break;+ }+ device const block_mxfp4 & xb = x[row*nb + ib];+ device const uint8_t * q2 = (device const uint8_t *)(xb.qs + 8*it);++ float4 acc1 = yl[0]*float4(kvalues_mxfp4_f[q2[0] & 0x0F], kvalues_mxfp4_f[q2[1] & 0x0F], kvalues_mxfp4_f[q2[2] & 0x0F], kvalues_mxfp4_f[q2[3] & 0x0F]);+ float4 acc2 = yl[1]*float4(kvalues_mxfp4_f[q2[0] >> 4 ], kvalues_mxfp4_f[q2[1] >> 4 ], kvalues_mxfp4_f[q2[2] >> 4 ], kvalues_mxfp4_f[q2[3] >> 4 ]);+ float4 acc3 = yl[2]*float4(kvalues_mxfp4_f[q2[4] & 0x0F], kvalues_mxfp4_f[q2[5] & 0x0F], kvalues_mxfp4_f[q2[6] & 0x0F], kvalues_mxfp4_f[q2[7] & 0x0F]);+ float4 acc4 = yl[3]*float4(kvalues_mxfp4_f[q2[4] >> 4 ], kvalues_mxfp4_f[q2[5] >> 4 ], kvalues_mxfp4_f[q2[6] >> 4 ], kvalues_mxfp4_f[q2[7] >> 4 ]);++ acc1 = (acc1 + acc3) + (acc2 + acc4);++ sumf[row] += e8m0_to_fp32(xb.e) * ((acc1[0] + acc1[1]) + (acc1[2] + acc1[3]));+ }++ yb += 16 * QK_MXFP4;+ }++ for (int row = 0; row < nr; ++row) {+ const float tot = simd_sum(sumf[row]);+ if (tiisg == 0 && first_row + row < ne01) {+ dst[r1*ne0 + im*ne0*ne1 + first_row + row] = tot;+ }+ }+}++[[host_name("kernel_mul_mv_mxfp4_f32")]]+kernel void kernel_mul_mv_mxfp4_f32(+ device const void * src0,+ device const float * src1,+ device float * dst,+ constant int64_t & ne00,+ constant int64_t & ne01,+ constant int64_t & ne02,+ constant uint64_t & nb00,+ constant uint64_t & nb01,+ constant uint64_t & nb02,+ constant int64_t & ne10,+ constant int64_t & ne11,+ constant int64_t & ne12,+ constant uint64_t & nb10,+ constant uint64_t & nb11,+ constant uint64_t & nb12,+ constant int64_t & ne0,+ constant int64_t & ne1,+ constant uint & r2,+ constant uint & r3,+ uint3 tgpig[[threadgroup_position_in_grid]],+ uint tiisg[[thread_index_in_simdgroup]],+ uint sgitg[[simdgroup_index_in_threadgroup]]) {+ kernel_mul_mv_mxfp4_f32_impl(src0,src1,dst,ne00,ne01,ne02,ne10,ne12,ne0,ne1,r2,r3,nullptr,tgpig,tiisg,sgitg);+}+[[host_name("kernel_mul_mv_q8_0_f32")]]kernel void kernel_mul_mv_q8_0_f32(device const void * src0,@@ -7622,6 +7745,7 @@ typedef decltype(kernel_mul_mv_id<mmv_fn<kernel_mul_mv_impl<float, float4, floattemplate [[host_name("kernel_mul_mv_id_f32_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id<mmv_fn<kernel_mul_mv_impl<float, float4, float, float4>>>;template [[host_name("kernel_mul_mv_id_f16_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id<mmv_fn<kernel_mul_mv_impl<half, half4, float, float4>>>;template [[host_name("kernel_mul_mv_id_q8_0_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id<mmv_fn<kernel_mul_mv_q8_0_f32_impl>>;+template [[host_name("kernel_mul_mv_id_mxfp4_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id<mmv_fn<kernel_mul_mv_mxfp4_f32_impl>>;template [[host_name("kernel_mul_mv_id_q4_0_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id<mmv_fn<mul_vec_q_n_f32_impl<block_q4_0, N_DST, N_SIMDGROUP, N_SIMDWIDTH>>>;template [[host_name("kernel_mul_mv_id_q4_1_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id<mmv_fn<mul_vec_q_n_f32_impl<block_q4_1, N_DST, N_SIMDGROUP, N_SIMDWIDTH>>>;template [[host_name("kernel_mul_mv_id_q5_0_f32")]] kernel kernel_mul_mv_id_t kernel_mul_mv_id<mmv_fn<mul_vec_q_n_f32_impl<block_q5_0, N_DST, N_SIMDGROUP, N_SIMDWIDTH>>>;