LLM inference in C/C++
C++ ggml
hexagon: support for multi-device model split (aka row-split) (#28589)
Max Krasnyanskyeafe15a5e3d8
diff --git a/docs/backend/snapdragon/README.md b/docs/backend/snapdragon/README.mdindex 391c8bf230..5d32a5877a 100644--- a/docs/backend/snapdragon/README.md+++ b/docs/backend/snapdragon/README.md@@ -188,7 +188,7 @@ llama_memory_breakdown_print: | - Host | 439 =Op test for MUL_MAT:```-~/src/llama.cpp$ ./scripts/snapdragon/run.py --target adb --hex-hostbuf 0 --devices HTP0:0 -- test-backend-ops -b HTP0:0 -o MUL_MAT+~/src/llama.cpp$ ./scripts/snapdragon/run.py --target adb --devices HTP0:0 -- test-backend-ops -b HTP0:0 -o MUL_MAT...Backend 2/3: HTP0:0Device description: Hexagon@@ -213,14 +213,109 @@ ggml-hex: new session: HTP0 : session-id 0 domain-id 3 uri file:///libggml-htp-v| llama 1B Q4_0 | 729.75 MiB | 1.24 B | HTP | 99 | 4 | 128 | 0 | tg64 | 51.54 ± 1.13 |```+## Multi-Device Execution Modes++The Hexagon backend supports multiple execution and partitioning modes to accommodate different model sizes, memory+constraints, and single- or multi-NPU hardware topologies:++### 1. Single-Device Mode with Dynamic Buffer Mapping++Runs the model on a single NPU session (e.g. `HTP0` or `HTP0:0`).++A single NPU session provides ~3.5GB of available virtual address space. For models larger than 3.5GB, the backend+automatically maps and unmaps weight buffers during graph execution. This allows large models to run on a single NPU+without manual configuration:++```bash+./scripts/snapdragon/run.py --target adb --devices HTP0:0 -- \+ llama-cli -m models/Llama-3.2-3B-Instruct-Q4_0.gguf -ngl 99 -p "Hello"+```++### 2. Layer-Split Mode across Virtual Sessions (`HTP0,HTP1,...` or `HTP0:0,HTP0:1,...`)++Partitions model layers at load time across multiple virtual sessions hosted on a single physical NPU.++Each virtual session acts as an independent backend device from llama.cpp's perspective (similar to multiple GPUs).+Because layers are permanently distributed across sessions, each session's allocated weights remain within its private 3.5GB+address space window, eliminating runtime buffer re-mapping overhead.++Here is an example of running the GPT-OSS-20B model on a Snapdragon device using 4 virtual sessions on a single NPU:++```bash+./scripts/snapdragon/run.py --target adb \+ --devices HTP0:0,HTP0:1,HTP0:2,HTP0:3 -- \+ llama-cli --load-mode none -m /data/local/tmp/gguf/gpt-oss-20b-Q4_0.gguf -t 4 \+ --ctx-size 8192 --batch-size 128 -ctk q8_0 -ctv q8_0 -fa on -ngl 99 -no-cnv -f surfing.txt+```++Log output snippet:++```+...+llama_model_loader: - type f32: 289 tensors+llama_model_loader: - type q4_0: 96 tensors+llama_model_loader: - type q8_0: 2 tensors+llama_model_loader: - type mxfp4: 72 tensors+...+load_tensors: offloaded 25/25 layers to GPU+load_tensors: CPU model buffer size = 1182.09 MiB+load_tensors: HTP0:1 model buffer size = 2512.58 MiB+load_tensors: HTP0:3 model buffer size = 2093.83 MiB+load_tensors: HTP0:0 model buffer size = 2931.34 MiB+load_tensors: HTP0:2 model buffer size = 2512.58 MiB+...+llama_perf_context_print: prompt eval time = 3843.67 ms / 197 tokens ( 19.51 ms per token, 51.25 tokens per second)+llama_perf_context_print: eval time = 1686.13 ms / 31 runs ( 54.39 ms per token, 18.39 tokens per second)+llama_perf_context_print: total time = 6266.30 ms / 228 tokens+llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |+llama_memory_breakdown_print: | - HTP0:0 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |+llama_memory_breakdown_print: | - HTP0:1 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |+llama_memory_breakdown_print: | - HTP0:2 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |+llama_memory_breakdown_print: | - HTP0:3 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |+llama_memory_breakdown_print: | - Host | 1476 = 1208 + 105 + 162 |+```++### 3. Tensor-Split Mode across Physical Devices (`HTP0:0,HTP1:0,...`)++Distributes model tensors across distinct physical NPU hardware cores using llama.cpp's tensor parallelism+(`--split-mode tensor`).++Tensors are partitioned across physical NPUs for parallel execution (proportions are distributed equally by default without+needing an explicit `--tensor-split` option):++```bash+./scripts/snapdragon/run.py --target adb \+ --devices HTP0:0,HTP1:0 -- \+ llama-cli -m models/Llama-3.2-3B-Instruct-Q4_0.gguf --split-mode tensor -ngl 99 -p "Hello"+```++### 4. Row-Split Multi-Device Mode via Device Grouping (`HTP0[0-1]`)++Groups multiple physical NPU cores into a single logical device using bracket notation (`HTP0[0-1]` or `HTP0[0,1]`).++Unlike host-level tensor-splitting, row-splitting is executed entirely inside the Hexagon backend:++```bash+./scripts/snapdragon/run.py --target adb \+ --devices 'HTP0[0-1]' -- \+ llama-cli -m models/Llama-3.2-3B-Instruct-Q4_0.gguf -ngl 99 -p "Hello"+```++You can also combine row-splitting with layer-splitting across multiple grouped devices (e.g. `--devices 'HTP0[0-1],HTP1[2-3]'`+on 4 physical NPUs, or `--devices 'HTP0[0-1:0],HTP1[0-1:1]'` on 2 physical NPUs using virtual sessions 0 and 1).+## Environment variables- `GGML_HEXAGON_DEVICES` (default: not set, defaults to HTP0 session)- Controls which NPU devices and sessions to allocate. Can be configured as:- - A single integer `N`: Allocates `N` sessions named `HTP0`, `HTP1`, ..., `HTP<N-1>` (behaves identically to `GGML_HEXAGON_NDEV=N`).- - A comma-separated list of device names in `HTP<physical_idx>:<virtual_idx>` format (or legacy `HTP<idx>` format). For example, `HTP0:0,HTP0:1` creates two virtual- sessions on the first physical NPU (useful for memory limits). `HTP0:0,HTP1:0` allocates one session on each of the two physical NPUs- on a dual-NPU device.+ Controls which NPU devices and sessions to allocate. Configurable via `--devices` in `run.py`:+ - `N` (single integer): Allocates `N` virtual sessions named `HTP0`, `HTP1`, ..., `HTP<N-1>` on physical NPU 0.+ - `HTP<phys>:<virt>,...`: Comma-separated list of individual devices specifying physical and virtual index:+ - `HTP0:0,HTP0:1`: Two virtual sessions on physical NPU 0 (layer-split on single NPU).+ - `HTP0:0,HTP1:0`: One session on physical NPU 0 and one on physical NPU 1 (tensor-split across physical cores).+ - `HTP<name>[<phys_spec>]`: Device grouping syntax for row-split multi-device execution:+ - `HTP0[0-1]`: A single logical device `HTP0` that groups physical cores 0 and 1.+ - `HTP0[0-1],HTP1[2-3]`: Two layer-split devices across 4 physical NPUs (cores 0-1 and 2-3).+ - `HTP0[0-1:0],HTP1[0-1:1]`: Two layer-split devices across 2 physical NPUs using virtual sessions 0 and 1.- `GGML_HEXAGON_NDEV` (deprecated)Replaced by `GGML_HEXAGON_DEVICES`. Controls the number of virtual sessions to allocate on physical NPU `0`.@@ -229,9 +324,8 @@ ggml-hex: new session: HTP0 : session-id 0 domain-id 3 uri file:///libggml-htp-v- `GGML_HEXAGON_NHVX=0`Controls the number of HVX hardware threads to use. The default is all (actual number varies depending on the hardware version).-- `GGML_HEXAGON_HOSTBUF=1`- Controls whether the Hexagon backend allocates host buffers. By default, all buffers except for REPACK are host buffers.- This option is required for testing Ops that require REPACK buffers (MUL_MAT and MUL_MAT_ID).+- `GGML_HEXAGON_HOSTBUF=1` (default: 0, disabled)+ Enables allocating host buffers for debugging. By default, host buffers are disabled.- `GGML_HEXAGON_VERBOSE=1`Enables verbose logging of Ops from the backend. Example output:@@ -246,23 +340,26 @@ ggml-hex: new session: HTP0 : session-id 0 domain-id 3 uri file:///libggml-htp-v```- `GGML_HEXAGON_PROFILE=1`- Enables Op profiling:+ Enables Op profiling (configurable via `--hex-profile` in `run.py`):- - `1` Basic profile with per-op `usecs` and `cycles` counters- - `2` Extended profile with per-op `usecs`, `cycles` and default PMU counter data- - `0x1,...,0x8` Extended profile with per-op `usecs`, `cycles` and custom PMU counter data+ - `1`: Basic profile with per-op `usecs` and `cycles` counters+ - `2`: Extended profile with per-op `usecs`, `cycles` and default PMU counter data+ - `0x1,...,0x8`: Extended profile with per-op `usecs`, `cycles` and custom PMU counter data- The logging output can be either saved into a file for post-processing or it can be piped directly into the post-processing tool- to generate the report.- Examples:+ The logging output can be saved to a file or piped directly into the post-processing script:- `GGML_HEXAGON_PROFILE=1 ./scripts/snapdragon/run.py --target adb -- llama-cli ... |& ./scripts/snapdragon/ggml-hexagon-profile.py -`+ ```bash+ ./scripts/snapdragon/run.py --target adb --hex-profile 1 -- llama-cli ... |& \+ ./scripts/snapdragon/ggml-hexagon-profile.py -+ ```- `GGML_HEXAGON_OPFILTER=regex`- Allows filtering (disabling) Ops that match the regex pattern:+ Filters (disables) Ops matching the regex pattern (configurable via `--hex-opfilter` in `run.py`):- Examples:-- `GGML_HEXAGON_OPFILTER="FLASH_ATTN_EXT" ./scripts/snapdragon/run.py --target adb -- llama-cli ...` - Disable Flash Attention on Hexagon (falls back to CPU or GPU)- `GGML_HEXAGON_OPFILTER="ADD\|SUB" ./scripts/snapdragon/run.py --target adb -- llama-cli ...` - Disable ADD and SUB on Hexagon (fall back to CPU or GPU)+ ```bash+ # Disable Flash Attention on Hexagon (falls back to CPU or GPU)+ ./scripts/snapdragon/run.py --target adb --hex-opfilter "FLASH_ATTN_EXT" -- llama-cli ...+ # Disable ADD and SUB on Hexagon (fall back to CPU or GPU)+ ./scripts/snapdragon/run.py --target adb --hex-opfilter "ADD|SUB" -- llama-cli ...+ ```diff --git a/docs/backend/snapdragon/developer.md b/docs/backend/snapdragon/developer.mdindex d7d9f2a279..633643c16d 100644--- a/docs/backend/snapdragon/developer.md+++ b/docs/backend/snapdragon/developer.md@@ -2,16 +2,16 @@## Backend libraries-The Hexagon backend consist of two parts:+The Hexagon backend consists of two parts:- `libggml-hexagon`- This is the regular CPU-side GGML backend library, either shared or statically linked+ This is the regular CPU-side GGML backend library, either shared or statically linked.- `libggml-htp-vNN`This is the NPU-side (HTP stands for Hexagon Tensor Processor) shared library that contains the Op dispatcher and kernels.The correct library is selected automatically at runtime based on the HW version.-Here is an example of the build artifacts+Here is an example of the build artifacts:```~/src/llama.cpp$ ls -l pkg-adb/llama.cpp/lib/libggml*@@ -26,75 +26,307 @@ pkg-adb/llama.cpp/lib/libggml-htp-v81.so## Memory buffers-Hexagon NPU backend takes advantage of the Snapdragon's unified memory model where all buffers are fully accessible by the CPU and GPU.-The NPU does have a dedicated tightly-coupled memory called VTCM but that memory is used only for intermediate data (e.g. dynamically-quantized tensors) or temporary data (chunks of the weight tensors fetched via DMA).--Please note that currently the Hexagon backend does not implement SET/GET_ROWS Ops because there is no advantage in offloading those-to the NPU at this point.--The backend does allocates non-host buffers for the tensors with datatypes that require repacking: Q4_0, Q8_0, MXFP4.-From the MMU perspective these buffers are still regular buffers (normal access by the CPU) they are marked as non-host simply to force-the repacking.+The Hexagon NPU backend takes advantage of Snapdragon unified memory where all DDR buffers are accessible by CPU, GPU, and NPU.+The NPU has dedicated tightly-coupled memory called VTCM (Vector Tightly-Coupled Memory). VTCM is used for intermediate data (such as+dynamically quantized activations) and streaming buffers (chunks of weight and activation tensors fetched via DMA).## Large model handling-Hexagon NPU sessions (aka Process Domains (PD) in the Hexagon SDK) are limited to a maximum memory mapping window of around 3.5GB.+Hexagon NPU sessions have a 32-bit virtual address space window of around 3.5GB.In llama.cpp/GGML, each Hexagon session is mapped to a single GGML backend device (e.g., `HTP0:0`, `HTP0:1`, etc. when using`GGML_HEXAGON_DEVICES`, or `HTP0`, `HTP1` in legacy mode).-To support running models larger than 3.5GB on a single device, the Hexagon backend dynamically maps and unmaps execution buffers-during the graph execution cycle to stay within the Process Domain window. This enables large models to run successfully on a single-NPU device.+To support running models larger than 3.5GB on a single device, the Hexagon backend dynamically maps and unmaps buffers:+- Buffers are allocated in shared DDR (RPCMEM) via file descriptors (`fastrpc_mmap` using `FASTRPC_MAP_FD_DELAYED`).+- Pinned buffers (such as KV cache and active compute buffers) remain mapped throughout execution.+- Inactive weight buffers are dynamically mapped into the NPU session via `HAP_mmap()` during batch buffer preparation+ (`prep_op_bufs()` in `htp/main.c`) and unmapped via `htp_iface_munmap()` when no longer needed by the active batch.+- This dynamic sliding window allows a single NPU session to execute models that exceed the 3.5GB window.++Alternatively, users can partition and split the model across multiple virtual sessions or physical NPUs using layer-splitting,+tensor-splitting, or row-splitting modes. For user-facing execution modes and examples, see the+[Snapdragon user guide](README.md#multi-device-execution-modes).++## Op and Kernel Development Guidelines++Writing high-performance operators for Hexagon requires following specific guidelines.++### DDR -> DMA -> VTCM Execution Pipeline++- Strongly prefer the `DDR -> DMA -> VTCM -> compute (HVX/HMX) -> VTCM -> DMA -> DDR` data flow.+- Direct HVX reads/writes from/to DDR are less efficient and should only be used as a fallback.+- The DMA queue is a strict FIFO where operations must be pushed and popped in strict order.+- Follow the pipelined multi-buffering sequence properly (typically 2x to 16x buffering) so every push has a corresponding pop:++ 1. In the prologue, push initial DDR -> VTCM transfers to prime the pipeline.+ 2. In the loop body, wait for buffer N via DMA pop, launch HVX/HMX compute on buffer N, push VTCM -> DDR writeback of result N,+ and push DDR -> VTCM prefetch of buffer N+2.+ 3. In the epilogue, pop all remaining in-flight transfers to drain the pipeline.++- Because every push must be matched by a pop, `dma_queue_flush()` is not required when the pipeline sequence is followed+ properly. Flushing is only used in rare exceptions where a batch of operations is pushed without individual pops.+- Use the DMA queue interface from [`dma-queue.h`](../../../ggml/src/ggml-hexagon/htp/dma-queue.h)+ (`dma_queue_push_ddr_to_vtcm`, `dma_queue_pop`, `dma_queue_push_vtcm_to_ddr`).+ See [`cumsum-ops.c`](../../../ggml/src/ggml-hexagon/htp/cumsum-ops.c) and+ [`act-ops.c`](../../../ggml/src/ggml-hexagon/htp/act-ops.c) for reference implementations.++### Avoid Scalar Reads and Writes to VTCM++- Access VTCM data using DMA transfers or HVX/HMX vector instructions rather than scalar reads and writes.++### Avoid Scalar Division in Inner Loops++- Hexagon cores do not have hardware division instructions.+- For recurring divisions across iterations or threads, use `fastdiv` from+ [`hex-fastdiv.h`](../../../ggml/src/ggml-hexagon/htp/hex-fastdiv.h) with precomputed divisors (such as+ `octx->ctx->mdev.count_div` or `octx->n_threads_div`).+- Do not call `init_fastdiv_values()` for single-use divisions; use standard compiler division (`/`) instead.++### Host-Side Precomputation via `kernel_params`++- Precompute tensor shapes, strides, scale conversions, tiling layouts, and validation checks on the host CPU during graph+ preparation in [`ggml-hexagon.cpp`](../../../ggml/src/ggml-hexagon/ggml-hexagon.cpp).+- Pack precomputed parameters into the operator's fixed `kernel_params` structure in `htp_op_node` (such as+ `htp_mm_kernel_params`, `htp_unary_kernel_params`, `htp_fa_kernel_params`, `htp_get_rows_kernel_params`).+- The NPU executes directly using `octx->kernel_params` without redundant runtime metadata extraction or validation.+- **Strict Host-Kernel Alignment**:+ - Verify that parameters calculated by the host CPU are strictly honored by the NPU kernel.+ - Ensure the kernel does not ignore host-computed fields (for example, falling back to `octx->n_threads` instead of+ using `kparams->n_threads`, or ignoring precomputed `tasks_per_thread` and chunk counts).+ - Both human developers and coding agents must audit both sides of the interface: ensure fields populated in `kernel_params`+ in [`ggml-hexagon.cpp`](../../../ggml/src/ggml-hexagon/ggml-hexagon.cpp) are actively and consistently utilized by the+ corresponding operator entry point and worker threads in `htp/*-ops.c`.++### Tracing Instrumentation++- All kernels must include trace events for performance profiling and timeline visualization in Perfetto+ ([`hex-profile.h`](../../../ggml/src/ggml-hexagon/htp/hex-profile.h)).+- Surround compute sections with `htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) info)` and+ `htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) info)`.+- Use specific event types for major phases:+ - `HTP_TRACE_EVT_HVX_COMP`: Vector compute execution.+ - `HTP_TRACE_EVT_DMA`: DMA transfer wait or poll cycles.+ - `HTP_TRACE_EVT_FENCE`: Multi-device fence barrier synchronization.+ - `HTP_TRACE_EVT_L2FLUSH`: L2 cache cleaning operations.+- Pass meaningful progress metrics (such as row index, chunk index, or token index) in the 16-bit `info` parameter.++### Work Queue and Threading++- Distribute parallel work across NPU worker threads using the thread pool work queue:++ ```c+ work_queue_run(ctx->work_queue, worker_func, &op_ctx, n_threads);+ ```++- Keep worker functions independent and re-entrant. Worker threads should only operate on their designated chunk of rows or elements.++### Avoid Redundant Defensive NULL Checks++- Do not add defensive NULL checks or assertions for internal framework pointers or required graph operands and outputs.+ Internal pointers include `ctx`, `octx`, local context structs like `*ctx`, `kparams`, and worker callback `data`.+- These pointers are architectural invariants during kernel execution and host-side graph preparation.+ Graph compute receives allocated nodes with valid required `node->src[N]` and `node->data` pointers.+- Do not turn an invariant violation into an unsupported operation or missed fusion.+ Checks such as `if (!octx || !octx->ctx)` clutter the code, obscure intent, and hide upstream errors.+- **Distinction**: `octx->src[N]` pointers *can* be NULL by design and must be checked when optional.+ Examples include attention masks, optional bias or weights in fused kernels, and frequency factors.++### Multiline Macro Formatting++- Keep trailing backslashes in multiline `#define` macros cleanly aligned to a consistent column.+- Avoid trailing whitespace after macro backslashes.+- Use [`scripts/snapdragon/ggml-hexagon-align-macros.py`](../../../scripts/snapdragon/ggml-hexagon-align-macros.py) to inspect, diff,+ or automatically align macro definitions across Hexagon kernel sources:++ ```bash+ # Check for misaligned macros+ python3 scripts/snapdragon/ggml-hexagon-align-macros.py ggml/src/ggml-hexagon/htp/++ # Fix misaligned macros in-place+ python3 scripts/snapdragon/ggml-hexagon-align-macros.py --fix ggml/src/ggml-hexagon/htp/+ ```++## Multi-Device Partitioning (mdev)++Multi-device (mdev) mode enables row-level tensor parallel execution across multiple physical NPU cores or virtual NPU+sessions.++### 128-Byte Cache Line Alignment++- Shared tensor buffers reside in DDR (RPCMEM) with a 128-byte cache line granularity+ (`HEX_L2_LINE_SIZE` = 128 bytes, `HTP_TENSOR_MDEV_LINE_SIZE`).+- **Rule**: Multi-device work partitions must align destination write regions to 128-byte cache line boundaries so distinct+ devices never share or overwrite the same cache line.++### Partitioning Helpers in `htp-tensor.h`++Common partitioning logic is factored into reusable inline helpers in+[`htp-tensor.h`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h):++1. [`htp_tensor_mdev_rows_per_chunk`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L67):+ Determines the minimum number of rows per chunk so that the chunk byte size is a multiple of 128 bytes:++ ```+ rows_per_chunk = 128 / hex_gcd_u32(row_size, 128)+ ```++ If row stride `nb[1]` is already a multiple of 128 bytes, `rows_per_chunk = 1`.+ Returns `false` if the tensor cannot be safely row-partitioned (such as unaligned base pointer, permuted layout,+ or non-128-byte aligned outer strides).-Alternatively, users can choose to use standard llama.cpp/GGML layer-splitting mode to partition and split the model across-multiple Hexagon devices or virtual sessions (which behave like multiple GPUs from the offload and splitting perspective).+2. [`htp_tensor_mdev_partition`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L94):+ Calculates the per-device work range `struct htp_tensor_mdev_range { uint32_t start; uint32_t count; }` given+ `total_units`, `units_per_chunk`, `mdev_idx`, `mdev_count`, and the precomputed `mdev_count_div`.+ Handles chunk distribution across devices, assigns remainder units to the last device, and automatically triggers+ single-device fallback when partitioning is unsafe.-Here is an example of running GPT-OSS-20B model on a Snapdragon device using 4 virtual sessions on a single NPU (physical index 0).+### Row-Partitioned Operators+For row-wise operators+(such as activations in [`act-ops.c`](../../../ggml/src/ggml-hexagon/htp/act-ops.c),+binary ops in [`binary-ops.c`](../../../ggml/src/ggml-hexagon/htp/binary-ops.c),+unary ops in [`unary-ops.c`](../../../ggml/src/ggml-hexagon/htp/unary-ops.c), and+sameshape copies in [`cpy-ops.c`](../../../ggml/src/ggml-hexagon/htp/cpy-ops.c)):++```c+const uint32_t total_rows = ne01 * ne02 * ne03;+const size_t dst_row_size = dst->ne[0] * elem_size;++uint32_t row_start = 0;+uint32_t nrows = total_rows;++if (octx->ctx->mdev.count > 1) {+ uint32_t rows_per_chunk = 0;+ htp_tensor_mdev_rows_per_chunk(dst, elem_size, (uint32_t) dst_row_size, &rows_per_chunk);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(+ total_rows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ nrows = range.count;+}++if (nrows == 0) {+ return HTP_STATUS_OK;+}```-~/src/llama.cpp$ ./scripts/snapdragon/run.py --target adb --devices HTP0:0,HTP0:1,HTP0:2,HTP0:3 -- llama-cli --load-mode none -m /data/local/tmp/gguf/gpt-oss-20b-Q4_0.gguf -t 4 --ctx-size 8192 --batch-size 128 -ctk q8_0 -ctv q8_0 -fa on -ngl 99 -no-cnv -f surfing.txt-...-llama_model_loader: - type f32: 289 tensors-llama_model_loader: - type q4_0: 96 tensors-llama_model_loader: - type q8_0: 2 tensors-llama_model_loader: - type mxfp4: 72 tensors-...-load_tensors: offloaded 25/25 layers to GPU-load_tensors: CPU model buffer size = 1182.09 MiB-load_tensors: HTP0:1 model buffer size = 2512.58 MiB-load_tensors: HTP0:3 model buffer size = 2093.83 MiB-load_tensors: HTP0:0 model buffer size = 2931.34 MiB-load_tensors: HTP0:2 model buffer size = 2512.58 MiB-...-llama_context: n_ctx_per_seq (8192) < n_ctx_train (131072) -- the full capacity of the model will not be utilized-llama_context: CPU output buffer size = 0.77 MiB-llama_kv_cache_iswa: creating non-SWA KV cache, size = 8192 cells-llama_kv_cache: HTP0:1 KV buffer size = 25.50 MiB-llama_kv_cache: HTP0:3 KV buffer size = 25.50 MiB-llama_kv_cache: HTP0:0 KV buffer size = 25.50 MiB-llama_kv_cache: HTP0:2 KV buffer size = 25.50 MiB-llama_kv_cache: size = 102.00 MiB ( 8192 cells, 12 layers, 1/1 seqs), K (q8_0): 51.00 MiB, V (q8_0): 51.00 MiB-llama_kv_cache_iswa: creating SWA KV cache, size = 256 cells-llama_kv_cache: HTP0:1 KV buffer size = 0.80 MiB-llama_kv_cache: HTP0:3 KV buffer size = 0.53 MiB-llama_kv_cache: HTP0:0 KV buffer size = 1.06 MiB-llama_kv_cache: HTP0:2 KV buffer size = 0.80 MiB-llama_kv_cache: size = 3.19 MiB ( 256 cells, 12 layers, 1/1 seqs), K (q8_0): 1.59 MiB, V (q8_0): 1.59 MiB-llama_context: HTP0:0 compute buffer size = 16.06 MiB-llama_context: HTP0:1 compute buffer size = 16.06 MiB-llama_context: HTP0:2 compute buffer size = 16.06 MiB-llama_context: HTP0:3 compute buffer size = 16.06 MiB-llama_context: CPU compute buffer size = 98.19 MiB-...-llama_perf_context_print: prompt eval time = 3843.67 ms / 197 tokens ( 19.51 ms per token, 51.25 tokens per second)-llama_perf_context_print: eval time = 1686.13 ms / 31 runs ( 54.39 ms per token, 18.39 tokens per second)-llama_perf_context_print: total time = 6266.30 ms / 228 tokens-llama_perf_context_print: graphs reused = 30-llama_memory_breakdown_print: | memory breakdown [MiB] | total free self model context compute unaccounted |-llama_memory_breakdown_print: | - HTP0:0 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |-llama_memory_breakdown_print: | - HTP0:1 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |-llama_memory_breakdown_print: | - HTP0:2 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |-llama_memory_breakdown_print: | - HTP0:3 (Hexagon) | 2048 = 2048 + ( 0 = 0 + 0 + 0) + 0 |-llama_memory_breakdown_print: | - Host | 1476 = 1208 + 105 + 162 |++### Element-Partitioned Operators++For flat element-wise operations (such as reshape copies in+[`cpy-ops.c`](../../../ggml/src/ggml-hexagon/htp/cpy-ops.c)):+- Partition total linear elements N = ne0 * ne1 * ne2 * ne3 in 128-byte cache line chunks (`elems_per_line = (elem_size == 4) ? 32 : 64`).+- Requires strict 1D contiguity:+ [`htp_tensor_is_contiguous(dst, elem_size)`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L28)+ and 128-byte aligned destination pointer+ [`htp_tensor_mdev_data_aligned(dst)`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L47).+- If contiguous and aligned, pass `elems_per_line` to+ [`htp_tensor_mdev_partition`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L94);+ otherwise pass 0 to trigger Device 0 fallback.++### Single-Device Fallback (Device 0)++- Fallback to Device 0 (`mdev.idx == 0`) when partitioning would cause cache line tearing or when work cannot be evenly distributed.+- Triggers:+ 1. Destination tensor cannot be safely partitioned (`rows_per_chunk == 0` or non-contiguous/unaligned buffer).+ 2. Total aligned chunks < `mdev_count`.+- Device 0 processes the entire tensor `[0, total_units)`.+- Devices 1 ... N-1 receive `count = 0` and return `HTP_STATUS_OK` immediately.++### Flatten Outer Dimensions Globally++- **Never partition solely on `ne01` (dimension 1).**+- Partitioning only on `ne01` repeats the device boundary across every 2D slice (`ne02`, `ne03`). If each 2D slice is small,+ false sharing occurs repeatedly throughout the tensor.+- Always flatten outer dimensions globally: `total_rows = ne01 * ne02 * ne03` and partition once across the combined row space.++### Stateless Starting Coordinates++- Do not use incremental state variables across slices that assume the thread or device starts at index 0.+- Precompute starting multidimensional coordinates at `r = row_start` (or `e = elem_start`) once using `fastdiv`.+- In inner loops, step base pointers directly (`ptr += stride`) or reset/wrap coordinates explicitly (`if (++i01 == ne01) { ... }`).++### Clean Range Encapsulation++- Initialize single-device default ranges at declaration:++ ```c+ uint32_t row_start = 0;+ uint32_t nrows = total_rows;+ ```++- Encapsulate all multi-device logic inside `if (octx->ctx->mdev.count > 1)`. If the block is omitted or compiled out,+ the operator runs standard single-device execution untouched.+- Do not propagate `mdev_` prefixes to worker functions or context structs. Worker threads are device-agnostic and+ should only receive standard range parameters (`ctx.row_start`, `ctx.nrows`).+- In worker threads, calculate row intervals using standard arithmetic:++ ```c+ const uint32_t ir0 = ctx->row_start + dr * ith;+ const uint32_t ir1 = MIN(ir0 + dr, ctx->row_start + ctx->nrows);+ ```++ In single-device mode (`row_start == 0`), this naturally simplifies to `dr * ith` and `MIN(ir0 + dr, ctx->nrows)` with zero overhead.++## Multi-Device Synchronization++Multi-device execution synchronizes worker sessions across devices using explicit barriers and tensor cache flushing.++### Synchronization Fence Protocol++Multi-device execution synchronizes worker sessions through atomic fence slots and barriers defined in+[`htp-fence.h`](../../../ggml/src/ggml-hexagon/htp/htp-fence.h):+```+[NPU Session 0] [NPU Session 1]+ | |+ (Input Prep) (Input Prep)+ | |+ Pre-Op Barrier ----------------------------- Pre-Op Barrier+ (mdev_sync_fence) (mdev_sync_fence)+ | |+ Kernel Execution Kernel Execution+ (Output Slice 0) (Output Slice 1)+ | |+ Tensor Cache Flush Tensor Cache Flush+ (htp_tensor_flush_all) (htp_tensor_flush_all)+ | |+ Post-Op/Batch Barrier ---------------------- Post-Op/Batch Barrier+ (htp_mdev_group_barrier) (htp_mdev_group_barrier)+ | |+ Return Response to Host Return Response to Host+```++### Atomic Fence Slots and Cache Invalidation++- Fence synchronization operates on dedicated RPCMEM shared memory mapped across all participating sessions (`ctx->mdev.fence_base`).+- Each device owns a dedicated 128-byte cache-line aligned fence slot:++ ```c+ atomic_uint * my_fence = htp_mdev_fence_slot(fence_base, mdev_idx);+ ```++- **Writing to fence ([`htp_fence_write`](../../../ggml/src/ggml-hexagon/htp/htp-fence.h#L18))**:+ Stores `seq` and `status`, issues a `syncht` thread synchronization barrier, and flushes/invalidates the line+ using `Q6_dccleaninva_A(fence)`.+- **Reading from peer fence ([`htp_fence_read`](../../../ggml/src/ggml-hexagon/htp/htp-fence.h#L26))**:+ Executes `Q6_dccleaninva_A(fence)` and `syncht` before reading atomic values to ensure fresh data from DDR.++### Deterministic Monotonic Sequence Numbers++- Barrier fences use monotonically increasing sequence numbers:++ ```c+ const uint32_t seq = ++ctx->mdev.fence_seq;+ ```++- Comparing sequence numbers with signed arithmetic `(int32_t)(peer_seq - seq) >= 0` prevents race conditions or+ misaligned barrier arrivals across iterations.+- If any peer reports an error status (`peer_status > HTP_STATUS_OK`), the barrier propagates the error and unblocks immediately.++### Tensor Cache Flush and Pipeline Completion++- In the kernel, ensure all pushed DMA operations have been popped in strict FIFO order to drain the queue.+- Use [`htp_tensor_flush_all()`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h) to flush specific dirty tensors back to DDR:+ - [`htp_tensor_flush_all()`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h) flushes only modified tensor address ranges,+ ensuring peer devices and the host CPU observe consistent data in DDR.+- Never signal completion before all DMA transfers are drained and dirty tensor flushes have completed.+diff --git a/ggml/src/ggml-hexagon/ggml-hexagon.cpp b/ggml/src/ggml-hexagon/ggml-hexagon.cppindex 112e9bae60..ec78013886 100644--- a/ggml/src/ggml-hexagon/ggml-hexagon.cpp+++ b/ggml/src/ggml-hexagon/ggml-hexagon.cpp@@ -66,7 +66,6 @@ using u32vec = std::vector<uint32_t>;#define GGML_HEXAGON_MAX_SESSIONS 16-#define GGML_HEXAGON_FENCE_BUFFER_SIZE 8192#define GGML_HEXAGON_FENCE_SLOT_SIZE 128struct ggml_hexagon_device_config {@@ -75,6 +74,8 @@ struct ggml_hexagon_device_config {int domain_id = 0;std::string domain_name;std::string name;++ std::vector<ggml_hexagon_device_config> mdev_group;};static ggml_hexagon_device_config opt_device_configs[GGML_HEXAGON_MAX_SESSIONS];@@ -350,27 +351,48 @@ struct ggml_hexagon_tensor_extra {};static inline bool ggml_hexagon_tensor_is_fuseable(const struct ggml_tensor * t) {- if (!t || !t->extra) return false;+ if (!t->extra) return false;auto extra = (const struct ggml_hexagon_tensor_extra *) t->extra;return (extra->flags & GGML_HEXAGON_TENSOR_FUSEABLE) != 0;}+static inline bool ggml_hexagon_tensors_overlap(const struct ggml_tensor * a, const struct ggml_tensor * b) {+ const uintptr_t a0 = (uintptr_t) a->data;+ const uintptr_t b0 = (uintptr_t) b->data;+ const uintptr_t a1 = a0 + ggml_nbytes(a);+ const uintptr_t b1 = b0 + ggml_nbytes(b);++ return a0 < b1 && b0 < a1;+}+struct htp_opnode;struct ggml_hexagon_opbatch;struct ggml_hexagon_opqueue;struct ggml_hexagon_shared_buffer;+struct ggml_hexagon_fence_buffer;struct ggml_hexagon_session;+struct ggml_backend_hexagon_device_context;++struct ggml_hexagon_mdev_group {+ uint32_t idx = 0;+ uint32_t count = 1;+ std::vector<std::unique_ptr<ggml_hexagon_session>> sessions;+};struct ggml_backend_hexagon_comm_context {std::vector<ggml_backend_t> backends;size_t n_backends = 0;- uint32_t fence_seq = 0;+ volatile uint32_t * fence_slots[GGML_HEXAGON_MAX_SESSIONS] = {};+ ggml_tensor fence_tensors[GGML_HEXAGON_MAX_SESSIONS] = {};};struct ggml_hexagon_event {- ggml_hexagon_session * sess = nullptr;- uint64_t seq = 0;+ ggml_hexagon_session * sess = nullptr;+ ggml_hexagon_session * fence_sess = nullptr;+ volatile uint32_t * fence_slot = nullptr;+ ggml_tensor fence_tensor = {};+ uint32_t seq = 0;};struct ggml_hexagon_session {@@ -387,12 +409,12 @@ struct ggml_hexagon_session {bool valid_queue;bool valid_iface;- std::atomic<int> op_pending;ggml_hexagon_opbatch* op_batch;ggml_hexagon_opqueue* op_queue;std::unordered_map<int, std::unique_ptr<ggml_hexagon_shared_buffer>> cloned_buffers;- std::unordered_set<ggml_hexagon_session *> sync_peers;+ std::unordered_set<ggml_hexagon_session *> virt_peers;+ std::unordered_set<ggml_hexagon_session *> phys_peers;uint32_t n_threads = 0;uint32_t n_hvx = 0;@@ -400,14 +422,23 @@ struct ggml_hexagon_session {uint64_t vtcm_size = 0;size_t max_vmem = 0;size_t max_bufsize = 0;- uint32_t fence_seq;+ uint32_t fence_seq = 0;++ std::atomic<uint64_t> batch_req_seq{0};+ std::atomic<uint64_t> batch_rsp_seq{0};+ std::atomic<uint32_t> last_error{HTP_STATUS_OK};uint64_t cached_uid = 0;std::vector<htp_opnode> cached_nodes;mutable std::unordered_set<const ggml_tensor *> needs_repack;- ggml_hexagon_session(const ggml_hexagon_device_config & config, ggml_backend_dev_t dev = nullptr) noexcept(false);+ ggml_hexagon_mdev_group mdev;+ ggml_backend_dev_t dev = nullptr;+ ggml_backend_hexagon_device_context * dev_ctx = nullptr;+ ggml_hexagon_fence_buffer * fence_buf = nullptr;++ ggml_hexagon_session(const ggml_hexagon_device_config & config, ggml_backend_dev_t dev = nullptr, uint32_t mdev_idx = 0, uint32_t mdev_count = 0) noexcept(false);~ggml_hexagon_session() noexcept(true);const char* c_name() const { return name.c_str(); }@@ -415,31 +446,36 @@ struct ggml_hexagon_session {void allocate(const ggml_hexagon_device_config & config) noexcept(false);void release() noexcept(true);+ uint8_t * alloc_fence(uint32_t n_slots = 1);+ void free_fence(void * ptr, uint32_t n_slots = 1);++ uint8_t * mdev_fence_slot = nullptr;+ std::unordered_map<uint64_t, volatile uint32_t *> cpy_fence_slots;++ void enqueue_mdev_group();void enqueue_op(const htp_opnode & node);void enqueue_cpy(const ggml_tensor * src, ggml_tensor * dst, const ggml_tensor * sync_tensor = nullptr, uint32_t fence_seq = 0);- void enqueue_fence(const ggml_tensor * sync_tensor, uint32_t fence_seq = 0);- void enqueue_allreduce(const ggml_tensor * dst, const std::vector<const ggml_tensor *> & src_tensors, const std::vector<const ggml_tensor *> & sync_tensors, uint32_t rank, uint32_t n_ranks, uint32_t fence_seq_entry = 0, uint32_t fence_seq_exit = 0);+ void enqueue_fence(const ggml_tensor * sync_tensor, uint32_t fence_seq = 0, bool wait = true);+ void enqueue_allreduce(const ggml_tensor * dst, const std::vector<const ggml_tensor *> & src_tensors,+ const std::vector<const ggml_tensor *> & sync_tensors, uint32_t rank, uint32_t n_ranks,+ uint32_t fence_seq_entry = 0, uint32_t fence_seq_exit = 0);- void flush(bool all = true);- void flush_pending(bool all = false);+ void flush_sync(bool all = true);+ void flush_async();void flush_batch(size_t min_ops = 1);-- uint64_t record_event();- void wait_event(uint64_t seq);+ void flush_peers();+ void flush_pending(bool all = true);bool clone_buffer(const ggml_hexagon_shared_buffer*);+ void release_buffer(const ggml_hexagon_shared_buffer*);+ void unclone_buffer(const ggml_hexagon_shared_buffer*);- void add_sync_peer(ggml_hexagon_session * peer) {- sync_peers.insert(peer);- }-- void flush_sync_peers() {- if (sync_peers.empty()) return;-- for (auto * peer : sync_peers) {- peer->flush_batch();+ void add_peer(ggml_hexagon_session * peer) {+ if (this->phys_idx == peer->phys_idx) {+ virt_peers.insert(peer);+ } else {+ phys_peers.insert(peer);}- sync_peers.clear();}};@@ -451,8 +487,9 @@ struct ggml_backend_hexagon_device_context {ggml_backend_dev_t dev = nullptr;size_t max_bufsize = 0;- ggml_backend_buffer_type buffer_type = {};- ggml_backend_buffer_type host_buffer_type = {};+ ggml_backend_buffer_type buffer_type = {};+ ggml_backend_buffer_type host_buffer_type = {};+ ggml_backend_buffer_type fence_buffer_type = {};std::unique_ptr<ggml_hexagon_session> sess;@@ -484,6 +521,8 @@ struct ggml_hexagon_rpcmem_block {int fd = -1;size_t size = 0;+ std::unordered_set<ggml_hexagon_session *> mapped_clones;+ggml_hexagon_rpcmem_block(size_t size) {base = (uint8_t *) rpcmem_alloc2(RPCMEM_HEAP_ID_SYSTEM, RPCMEM_DEFAULT_FLAGS, size);if (!base) {@@ -508,8 +547,6 @@ struct ggml_hexagon_shared_buffer {ggml_hexagon_session * sess;std::shared_ptr<ggml_hexagon_rpcmem_block> mem;std::vector<ggml_hexagon_tensor_extra *> tensor_extra;- uint32_t fence_head = 0;- size_t fences_size = 0;bool mapped;bool pinned;@@ -518,16 +555,6 @@ struct ggml_hexagon_shared_buffer {size_t size() const { return mem ? mem->size : 0; }int fd() const { return mem ? mem->fd : -1; }- uint8_t * alloc_fence() {- if (fences_size == 0) return nullptr;- int max_slots = fences_size / GGML_HEXAGON_FENCE_SLOT_SIZE;- uint32_t slot = (fence_head++) % max_slots;-- size_t guard_offset = size() - fences_size;- uint8_t * fence_ptr = base() + guard_offset + (size_t)slot * GGML_HEXAGON_FENCE_SLOT_SIZE;- return fence_ptr;- }-void mmap() {if (!this->mem) return;fastrpc_map_flags flags = this->pinned ? FASTRPC_MAP_FD : FASTRPC_MAP_FD_DELAYED;@@ -581,29 +608,24 @@ struct ggml_hexagon_shared_buffer {this->mem = nullptr;}- ggml_hexagon_shared_buffer(ggml_hexagon_session * sess, size_t size, bool pinned = false, size_t fence_size = 0) {- this->sess = sess;- this->mapped = false;- this->pinned = pinned;- this->fences_size = fence_size;+ ggml_hexagon_shared_buffer(ggml_hexagon_session * sess, size_t size, bool pinned = false) {+ this->sess = sess;+ this->mapped = false;+ this->pinned = pinned;- // Size adjustment inside the buffer class+ // Size adjustment inside the buffer class: 4K aligned data size + 4K guard pagesize_t guard_offset = (size + 4095) & ~4095;- size_t total_size = guard_offset;- if (fence_size > 0) {- total_size += 4096 + fence_size;- }+ size_t total_size = guard_offset + 4096;alloc(total_size);}// Clone constructor for cross-session mappingggml_hexagon_shared_buffer(ggml_hexagon_session * sess, const ggml_hexagon_shared_buffer & other) {- this->sess = sess;- this->mem = other.mem;- this->mapped = false;- this->pinned = other.pinned;- this->fences_size = other.fences_size;+ this->sess = sess;+ this->mem = other.mem;+ this->mapped = false;+ this->pinned = other.pinned;}~ggml_hexagon_shared_buffer() {@@ -614,6 +636,59 @@ struct ggml_hexagon_shared_buffer {}};+struct ggml_hexagon_fence_buffer : public ggml_hexagon_shared_buffer {+ uint32_t slot_count = 0;+ uint32_t slot_head = 0;+ std::vector<uint32_t> free_slots;+ ggml_backend_buffer backend_buffer{};++ ggml_hexagon_fence_buffer(ggml_hexagon_session * sess, ggml_backend_buffer_type_t buft, size_t size)+ : ggml_hexagon_shared_buffer(sess, size, false /* pinned */),+ slot_count(size / GGML_HEXAGON_FENCE_SLOT_SIZE),+ slot_head(0) {+ backend_buffer.buft = buft;+ backend_buffer.context = static_cast<ggml_hexagon_shared_buffer *>(this);+ backend_buffer.size = size;+ }++ uint8_t * alloc_slot(uint32_t n_slots = 1) {+ uint8_t * ptr = nullptr;+ if (n_slots == 1 && !free_slots.empty()) {+ uint32_t slot = free_slots.back();+ free_slots.pop_back();+ ptr = base() + (size_t) slot * GGML_HEXAGON_FENCE_SLOT_SIZE;+ } else if (slot_head + n_slots <= slot_count) {+ uint32_t slot = slot_head;+ slot_head += n_slots;+ ptr = base() + (size_t) slot * GGML_HEXAGON_FENCE_SLOT_SIZE;+ }+ if (ptr) {+ memset(ptr, 0, (size_t) n_slots * GGML_HEXAGON_FENCE_SLOT_SIZE);+ }+ return ptr;+ }++ void free_slot(void * ptr, uint32_t n_slots = 1) {+ if (!ptr) return;+ uint32_t slot = ((uint8_t *) ptr - base()) / GGML_HEXAGON_FENCE_SLOT_SIZE;+ for (uint32_t i = 0; i < n_slots; i++) {+ free_slots.push_back(slot + i);+ }+ }+};++inline uint8_t * ggml_hexagon_session::alloc_fence(uint32_t n_slots) {+ uint8_t * ptr = fence_buf->alloc_slot(n_slots);+ GGML_ASSERT(ptr);+ return ptr;+}++inline void ggml_hexagon_session::free_fence(void * ptr, uint32_t n_slots) {+ if (fence_buf) {+ fence_buf->free_slot(ptr, n_slots);+ }+}+static ggml_hexagon_session * ggml_backend_hexagon_buffer_get_sess(ggml_backend_buffer_t buffer) {auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(buffer->context);return sbuf->sess;@@ -621,6 +696,7 @@ static ggml_hexagon_session * ggml_backend_hexagon_buffer_get_sess(ggml_backend_static void ggml_backend_hexagon_buffer_free_buffer(ggml_backend_buffer_t buffer) {auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(buffer->context);+ sbuf->sess->unclone_buffer(sbuf);delete sbuf;}@@ -1537,7 +1613,7 @@ static ggml_backend_buffer_t ggml_backend_hexagon_buffer_type_alloc_buffer(auto dev_ctx = static_cast<ggml_backend_hexagon_buffer_type_context *>(buffer_type->context)->dev_ctx;auto sess = dev_ctx->session();try {- ggml_hexagon_shared_buffer * sbuf = new ggml_hexagon_shared_buffer(sess, size, false, GGML_HEXAGON_FENCE_BUFFER_SIZE);+ ggml_hexagon_shared_buffer * sbuf = new ggml_hexagon_shared_buffer(sess, size, false);return ggml_backend_buffer_init(buffer_type, ggml_backend_hexagon_buffer_interface, sbuf, size);} catch (const std::exception & exc) {GGML_LOG_ERROR("ggml-hex: %s failed to allocate device buffer context: %s\n", dev_ctx->c_name(), exc.what());@@ -1550,7 +1626,7 @@ static ggml_backend_buffer_t ggml_backend_hexagon_host_buffer_type_alloc_buffer(auto dev_ctx = static_cast<ggml_backend_hexagon_buffer_type_context *>(buffer_type->context)->dev_ctx;auto sess = dev_ctx->session();try {- ggml_hexagon_shared_buffer * sbuf = new ggml_hexagon_shared_buffer(sess, size, false, GGML_HEXAGON_FENCE_BUFFER_SIZE);+ ggml_hexagon_shared_buffer * sbuf = new ggml_hexagon_shared_buffer(sess, size, false);return ggml_backend_buffer_init(buffer_type, ggml_backend_hexagon_host_buffer_interface, sbuf, size);} catch (const std::exception & exc) {GGML_LOG_ERROR("ggml-hex: %s failed to allocate host buffer context: %s\n", dev_ctx->c_name(), exc.what());@@ -1618,11 +1694,16 @@ ggml_backend_hexagon_device_context::ggml_backend_hexagon_device_context(int devhost_buffer_type.device = dev;host_buffer_type.iface = ggml_backend_hexagon_host_buffer_type_interface;host_buffer_type.context = new ggml_backend_hexagon_buffer_type_context(config.name + "-HOST", this);++ fence_buffer_type.device = dev;+ fence_buffer_type.iface = ggml_backend_hexagon_buffer_type_interface;+ fence_buffer_type.context = new ggml_backend_hexagon_buffer_type_context(config.name + "-FENCE", this);}ggml_backend_hexagon_device_context::~ggml_backend_hexagon_device_context() {delete static_cast<ggml_backend_hexagon_buffer_type_context *>(buffer_type.context);delete static_cast<ggml_backend_hexagon_buffer_type_context *>(host_buffer_type.context);+ delete static_cast<ggml_backend_hexagon_buffer_type_context *>(fence_buffer_type.context);}static bool ggml_backend_buffer_is_hexagon(const struct ggml_backend_buffer * b) {@@ -1698,8 +1779,8 @@ struct ggml_hexagon_opbatch {if (it != b_map.end()) { return it->second; }// Add new buffer to the batch- int bi = n_bufs++;GGML_ASSERT(n_bufs < HTP_OP_MAX_BUFS);+ int bi = n_bufs++;b_map.insert({sbuf->fd(), bi});@@ -1902,6 +1983,12 @@ struct ggml_hexagon_opbatch {}}+ void update_mdev_group(uint32_t mdev_idx) {+ if (n_ops > 0 && h_ops[0].opcode == HTP_OP_MDEV_GROUP) {+ h_ops[0].params[0] = (int32_t) mdev_idx;+ }+ }+bool try_fuse_allreduce_add(const htp_opnode & node) {if (n_ops == 0 || opt_ar_select != 2) return false;if (node.opcode != HTP_OP_ADD) return false;@@ -1910,15 +1997,16 @@ struct ggml_hexagon_opbatch {if (last_node.opcode != HTP_OP_ALLREDUCE) return false;auto * ar_kparams = (struct htp_allreduce_kernel_params *) last_node.kernel_params;- const uint32_t rank = (uint32_t) ar_kparams->rank;- const ggml_tensor * ar_local = (rank < last_node.inputs.size()) ? last_node.inputs[rank] : nullptr;+ const uint32_t rank = (uint32_t) ar_kparams->rank;+ const uint32_t n_ranks = (uint32_t) ar_kparams->n_ranks;+ const ggml_tensor * ar_local = last_node.inputs[rank];const ggml_tensor * add_src0 = node.src0();const ggml_tensor * add_src1 = node.src1();+ const ggml_tensor * add_dst = node.dst();- if (!add_src0 || !add_src1 || !ar_local) return false;if (!ggml_hexagon_tensor_is_fuseable(ar_local)) return false;- const ggml_tensor * res_tensor = nullptr;+ const ggml_tensor * res_tensor;if (add_src0 == ar_local || add_src0->data == ar_local->data) {res_tensor = add_src1;} else if (add_src1 == ar_local || add_src1->data == ar_local->data) {@@ -1927,14 +2015,12 @@ struct ggml_hexagon_opbatch {return false;}- if (!res_tensor || !res_tensor->data) return false;-if (ar_local->type != res_tensor->type) return false;const bool is_same_shape = (ar_local->ne[0] == res_tensor->ne[0] && ar_local->ne[1] == res_tensor->ne[1] &&ar_local->ne[2] == res_tensor->ne[2] && ar_local->ne[3] == res_tensor->ne[3]);- const bool is_row_bcast = (ar_local->ne[0] == res_tensor->ne[0] &&- res_tensor->ne[1] == 1 && res_tensor->ne[2] == 1 && res_tensor->ne[3] == 1);+ const bool is_row_bcast = !is_same_shape && (ar_local->ne[0] == res_tensor->ne[0] && res_tensor->ne[1] == 1 &&+ res_tensor->ne[2] == 1 && res_tensor->ne[3] == 1);if (!is_same_shape && !is_row_bcast) return false;@@ -1947,13 +2033,21 @@ struct ggml_hexagon_opbatch {return false;}}- if (ggml_is_contiguous(ar_local) != ggml_is_contiguous(node.dst())) {+ if (ggml_is_contiguous(ar_local) != ggml_is_contiguous(add_dst)) {return false;}+ for (uint32_t r = 0; r < n_ranks; r++) {+ const ggml_tensor * ar_src = last_node.inputs[r];+ if (ggml_hexagon_tensors_overlap(add_dst, ar_src)) {+ HEX_VERBOSE("ggml-hex: %s skip ALLREDUCE_ADD fusion: dst overlaps allreduce src %u\n", sess->c_name(), r);+ return false;+ }+ }+struct htp_allreduce_kernel_params new_kparams;if (!ggml_hexagon_precompute_allreduce_params(- sess, node.dst(), (uint32_t) ar_kparams->rank, (uint32_t) ar_kparams->n_ranks, true, is_row_bcast, &new_kparams+ sess, add_dst, (uint32_t) ar_kparams->rank, (uint32_t) ar_kparams->n_ranks, true, is_row_bcast, &new_kparams)) {HEX_VERBOSE("ggml-hex: %s skip ALLREDUCE_ADD fusion: solver failed\n", sess->c_name());return false;@@ -1961,7 +2055,6 @@ struct ggml_hexagon_opbatch {size_t extra_bufs = 0, extra_vmem = 0, extra_tens = 0;auto fit_t = [&](const ggml_tensor * t) {- if (!t) return;if (!t_map.count(t)) {extra_tens++;auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(t->buffer->context);@@ -1972,7 +2065,7 @@ struct ggml_hexagon_opbatch {}};fit_t(res_tensor);- fit_t(node.dst());+ fit_t(add_dst);if ((extra_bufs + n_bufs) > n_bufs_max || (extra_tens + n_tens) > n_tens_max || (extra_vmem + b_vmem) > b_vmem_max) {return false;}@@ -1981,7 +2074,7 @@ struct ggml_hexagon_opbatch {last_node.name = "ALLREDUCE+ADD";last_node.inputs.push_back(res_tensor);last_node.outputs.clear();- last_node.outputs.push_back(node.dst());+ last_node.outputs.push_back(add_dst);last_node.fused.push_back(node.node);memcpy(last_node.kernel_params, &new_kparams, sizeof(new_kparams));@@ -1989,9 +2082,8 @@ struct ggml_hexagon_opbatch {o.opcode = HTP_OP_ALLREDUCE_ADD;memcpy(o.kernel_params, &new_kparams, sizeof(new_kparams));- const uint32_t n_ranks = (uint32_t) ar_kparams->n_ranks;o.src[2 * n_ranks] = add_tensor(res_tensor);- o.dst[0] = add_tensor(node.dst());+ o.dst[0] = add_tensor(add_dst);for (uint32_t d = 1; d < HTP_OP_MAX_OUTPUTS; d++) {o.dst[d] = 0xffff;}@@ -2011,10 +2103,9 @@ struct ggml_hexagon_opbatch {const ggml_tensor * mul_src1 = node.src1();const ggml_tensor * rms_out = last_node.dst();- if (!mul_src0 || !mul_src1 || !rms_out) return false;if (!ggml_hexagon_tensor_is_fuseable(rms_out)) return false;- const ggml_tensor * weight = nullptr;+ const ggml_tensor * weight;if (mul_src0 == rms_out || mul_src0->data == rms_out->data) {weight = mul_src1;} else if (mul_src1 == rms_out || mul_src1->data == rms_out->data) {@@ -2023,10 +2114,7 @@ struct ggml_hexagon_opbatch {return false;}- if (!weight || !weight->data) return false;-const ggml_tensor * src0 = last_node.src0();- if (!src0 || !src0->data) return false;if (src0->ne[0] != weight->ne[0] || src0->ne[0] != node.dst()->ne[0]) {return false;@@ -2057,7 +2145,6 @@ struct ggml_hexagon_opbatch {size_t extra_bufs = 0, extra_vmem = 0, extra_tens = 0;auto fit_t = [&](const ggml_tensor * t) {- if (!t) return;if (!t_map.count(t)) {extra_tens++;auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(t->buffer->context);@@ -2112,10 +2199,9 @@ struct ggml_hexagon_opbatch {const ggml_tensor * add_src1 = node.src1();const ggml_tensor * mm_out = last_node.dst();- if (!add_src0 || !add_src1 || !mm_out) return false;if (!ggml_hexagon_tensor_is_fuseable(mm_out)) return false;- const ggml_tensor * src2 = nullptr;+ const ggml_tensor * src2;if (add_src0 == mm_out || add_src0->data == mm_out->data) {src2 = add_src1;} else if (add_src1 == mm_out || add_src1->data == mm_out->data) {@@ -2124,11 +2210,8 @@ struct ggml_hexagon_opbatch {return false;}- if (!src2 || !src2->data) return false;-const ggml_tensor * src0 = last_node.src0();const ggml_tensor * src1 = last_node.src1();- if (!src0 || !src1) return false;struct htp_mm_kernel_params kparams;ggml_hexagon_precompute_fused_matmul_add_params(sess, src0, src1, src2, node.dst(), &kparams);@@ -2144,7 +2227,6 @@ struct ggml_hexagon_opbatch {size_t extra_bufs = 0, extra_vmem = 0, extra_tens = 0;auto fit_t = [&](const ggml_tensor * t) {- if (!t) return;if (!t_map.count(t)) {extra_tens++;auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(t->buffer->context);@@ -2197,7 +2279,6 @@ struct ggml_hexagon_opbatch {const ggml_tensor * w_in = node.src0();const ggml_tensor * x_in = node.src1();const ggml_tensor * d_in = node.dst();- if (!w_in || !x_in || !d_in) return false;htp_opnode & last_node = ops[n_ops - 1];@@ -2231,7 +2312,6 @@ struct ggml_hexagon_opbatch {size_t extra_bufs = 0, extra_vmem = 0, extra_tens = 0;auto fit_t = [&](const ggml_tensor * t) {- if (!t) return;if (!t_map.count(t)) {extra_tens++;auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(t->buffer->context);@@ -2282,7 +2362,6 @@ struct ggml_hexagon_opbatch {const ggml_tensor * w0 = last_node.src0();const ggml_tensor * x = last_node.src1();const ggml_tensor * w1 = node.src0();- if (!w0 || !x || !w1) return false;struct htp_mm_kernel_params kparams;ggml_hexagon_precompute_fused_mmnx_params(sess, w0, x, 2, &kparams);@@ -2297,7 +2376,6 @@ struct ggml_hexagon_opbatch {size_t extra_bufs = 0, extra_vmem = 0, extra_tens = 0;auto fit_t = [&](const ggml_tensor * t) {- if (!t) return;if (!t_map.count(t)) {extra_tens++;auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(t->buffer->context);@@ -2359,7 +2437,6 @@ struct ggml_hexagon_opbatch {const ggml_tensor * x_in = node.src1();const ggml_tensor * ids_in = node.node->src[2];const ggml_tensor * d_in = node.dst();- if (!w_in || !x_in || !ids_in || !d_in) return false;htp_opnode & last_node = ops[n_ops - 1];@@ -2394,7 +2471,6 @@ struct ggml_hexagon_opbatch {size_t extra_bufs = 0, extra_vmem = 0, extra_tens = 0;auto fit_t = [&](const ggml_tensor * t) {- if (!t) return;if (!t_map.count(t)) {extra_tens++;auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(t->buffer->context);@@ -2447,7 +2523,6 @@ struct ggml_hexagon_opbatch {const ggml_tensor * x = last_node.src1();const ggml_tensor * ids = last_node.node->src[2];const ggml_tensor * w1 = node.src0();- if (!w0 || !x || !ids || !w1) return false;struct htp_mm_kernel_params kparams;ggml_hexagon_precompute_fused_mmidnx_params(sess, w0, x, node.dst(), 2, &kparams);@@ -2462,7 +2537,6 @@ struct ggml_hexagon_opbatch {size_t extra_bufs = 0, extra_vmem = 0, extra_tens = 0;auto fit_t = [&](const ggml_tensor * t) {- if (!t) return;if (!t_map.count(t)) {extra_tens++;auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(t->buffer->context);@@ -2540,17 +2614,14 @@ struct ggml_hexagon_opqueue {// Shared buffer for storing batchesggml_hexagon_shared_buffer *shm_buf;size_t shm_blk_size;-- uint64_t req_seq = 0;- uint64_t rsp_seq = 0;+ size_t depth;using opvec = std::vector<htp_opnode>;- std::queue<unsigned int> done; // completed batch idsstd::vector<opvec> op_cache; // per batch op cachestd::vector<uint64_t> start_usec; // per batch start time- ggml_hexagon_opqueue(ggml_hexagon_session *sess, size_t batch_size, size_t depth) {+ ggml_hexagon_opqueue(ggml_hexagon_session *sess, size_t batch_size, size_t depth) : depth(depth) {size_t n_bufs = HTP_OP_MAX_BUFS;size_t n_ops = batch_size;size_t n_tensors = n_ops * HTP_OP_MAX_OUTPUTS + n_ops * HTP_OP_MAX_INPUTS;@@ -2571,9 +2642,6 @@ struct ggml_hexagon_opqueue {op_cache.resize(depth);start_usec.resize(depth, 0);- // init done queue- for (unsigned int i = 0; i < depth; i++) { done.push(i); }-if (opt_verbose) {GGML_LOG_INFO("ggml-hex: %s allocated opqueue : batch-size %zu depth %zu shm-size %zu shm-block-size %zu\n",sess->c_name(), batch_size, depth, shm_buf->size(), shm_blk_size);@@ -2587,7 +2655,7 @@ struct ggml_hexagon_opqueue {size_t shm_size() const { return shm_buf ? shm_buf->size() : 0; }// push new batch- bool push(htp_opbatch_req& req, dspqueue_buffer& dbuf, ggml_hexagon_opbatch* op_batch) {+ bool push(htp_opbatch_req& req, dspqueue_buffer& dbuf, const ggml_hexagon_opbatch* op_batch, uint64_t seq) {static_assert(sizeof(htp_opbatch_req) % 8 == 0, "sizeof(htp_opbatch_req) must be multiple of 8");static_assert(sizeof(htp_opbatch_rsp) % 8 == 0, "sizeof(htp_opbatch_rsp) must be multiple of 8");static_assert(sizeof(htp_buf_desc) % 8 == 0, "sizeof(htp_buf_desc) must be multiple of 8");@@ -2595,16 +2663,17 @@ struct ggml_hexagon_opqueue {static_assert(sizeof(htp_op_desc) % 8 == 0, "sizeof(htp_op_desc) must be multiple of 8");static_assert(sizeof(htp_prof_desc) % 8 == 0, "sizeof(htp_prof_desc) must be multiple of 8");- if (done.empty()) { return false; }+ if (seq - shm_buf->sess->batch_rsp_seq > depth) { return false; }- req.id = done.front(); done.pop(); // batch id+ const uint32_t slot = (uint32_t) ((seq - 1) % depth);++ req.seq = seq;req.n_bufs = op_batch->n_bufs;req.n_tensors = op_batch->n_tens;req.n_ops = op_batch->n_ops;- req.seq = ++req_seq;- op_cache[req.id] = std::move(op_batch->ops);- start_usec[req.id] = ggml_time_us();+ op_cache[slot] = op_batch->ops;+ start_usec[slot] = ggml_time_us();const size_t b_size = sizeof(htp_buf_desc) * req.n_bufs;const size_t t_size = sizeof(htp_tensor) * req.n_tensors;@@ -2619,7 +2688,7 @@ struct ggml_hexagon_opqueue {req.n_traces = 0;}- dbuf.ptr = shm_buf->base() + (req.id * shm_blk_size);+ dbuf.ptr = shm_buf->base() + ((size_t) slot * shm_blk_size);dbuf.fd = shm_buf->fd();dbuf.flags = DSPQUEUE_BUFFER_FLAG_FLUSH_SENDER | DSPQUEUE_BUFFER_FLAG_INVALIDATE_RECIPIENT;dbuf.offset = (uint8_t*) dbuf.ptr - (uint8_t*) shm_buf->base();@@ -2632,18 +2701,14 @@ struct ggml_hexagon_opqueue {uint8_t * t_ptr = m_ptr; m_ptr += t_size;uint8_t * o_ptr = m_ptr;- op_batch->sort_buffers();-memcpy(b_ptr, (void *) op_batch->h_bufs.data(), b_size);memcpy(t_ptr, (void *) op_batch->h_tens.data(), t_size);memcpy(o_ptr, (void *) op_batch->h_ops.data(), o_size);- HEX_VERBOSE("ggml-hex: %s opqueue-push batch #%u : n-bufs %u n-tensors %u n-ops %u vmem %zu : b-size %zu t-size %zu o-size %zu m-size %zu\n",- shm_buf->sess->c_name(), req.id, req.n_bufs, req.n_tensors, req.n_ops, op_batch->b_vmem,+ HEX_VERBOSE("ggml-hex: %s opqueue-push batch #%llu : n-bufs %u n-tensors %u n-ops %u vmem %zu : b-size %zu t-size %zu o-size %zu m-size %zu\n",+ shm_buf->sess->c_name(), (unsigned long long) req.seq, req.n_bufs, req.n_tensors, req.n_ops, op_batch->b_vmem,b_size, t_size, o_size, (size_t) dbuf.size);- op_batch->reset();-if (opt_verbose > 1) {htp_buf_desc *b = (htp_buf_desc*) b_ptr;for (unsigned int i=0; i < req.n_bufs; i++) {@@ -2662,9 +2727,7 @@ struct ggml_hexagon_opqueue {}void pop(htp_opbatch_rsp rsp, dspqueue_buffer dbuf) {- GGML_ASSERT(rsp.id < op_cache.size());-- done.push(rsp.id);+ const uint32_t slot = (uint32_t) ((rsp.seq - 1) % depth);const size_t b_size = sizeof(htp_buf_desc) * rsp.n_bufs;const size_t t_size = sizeof(htp_tensor) * rsp.n_tensors;@@ -2681,15 +2744,15 @@ struct ggml_hexagon_opqueue {const size_t m_size = b_size + t_size + o_size + p_size + tr_size;GGML_ASSERT(m_size <= shm_blk_size);- HEX_VERBOSE("ggml-hex: %s opqueue-pop batch #%u : n-bufs %u n-tensors %u n-ops %u : m-size %zu b-size %zu t-size %zu o-size %zu\n",- shm_buf->sess->c_name(), rsp.id, rsp.n_bufs, rsp.n_tensors, rsp.n_ops,+ HEX_VERBOSE("ggml-hex: %s opqueue-pop batch #%llu : n-bufs %u n-tensors %u n-ops %u : m-size %zu b-size %zu t-size %zu o-size %zu\n",+ shm_buf->sess->c_name(), (unsigned long long) rsp.seq, rsp.n_bufs, rsp.n_tensors, rsp.n_ops,(size_t) dbuf.size, b_size, t_size, o_size);uint8_t * m_ptr = (uint8_t*) dbuf.ptr;uint8_t * p_ptr = m_ptr + (b_size + t_size + o_size);if (rsp.n_ops > 0) {- auto & ops = op_cache[rsp.id];+ auto & ops = op_cache[slot];GGML_ASSERT(rsp.n_ops <= ops.size());const htp_prof_desc * pd = (const htp_prof_desc *) p_ptr;@@ -2712,16 +2775,41 @@ struct ggml_hexagon_opqueue {ggml_hexagon_dump_trace_events(shm_buf->sess->name, rsp, trace_events, n_traces);}}-- if (rsp.seq > rsp_seq) {- rsp_seq = rsp.seq;- }}};-// Flush HTP response queue i.e wait for all outstanding requests to complete+void ggml_hexagon_session::flush_peers() {+ auto vpeers = std::move(virt_peers);+ virt_peers.clear();+ for (auto * peer : vpeers) {+ peer->flush_sync();+ }++ auto ppeers = std::move(phys_peers);+ phys_peers.clear();+ for (auto * peer : ppeers) {+ peer->flush_async();+ }++ for (auto & sub : this->mdev.sessions) {+ sub->flush_peers();+ }+}++void ggml_hexagon_session::flush_async() {+ flush_peers();+ flush_batch();+}+void ggml_hexagon_session::flush_pending(bool all) {- while (this->op_pending) {+ for (auto & sub : this->mdev.sessions) {+ sub->flush_pending(all);+ if (sub->last_error > HTP_STATUS_OK) {+ this->last_error = sub->last_error.load();+ }+ }++ while (this->batch_rsp_seq < this->batch_req_seq) {struct htp_opbatch_rsp rsp;uint32_t rsp_size;uint32_t flags;@@ -2746,32 +2834,64 @@ void ggml_hexagon_session::flush_pending(bool all) {GGML_ABORT("ggml-hex: %s dspcall : bad response : size %u dspbufs %u\n", this->c_name(), rsp_size, n_dbufs);}- if (rsp.status != HTP_STATUS_OK) {- GGML_LOG_ERROR("ggml-hex: %s dspcall : dsp-rsp: %s\n", this->c_name(), status_to_str(rsp.status));- // TODO: handle errors+ if (rsp.status > HTP_STATUS_OK) {+ GGML_LOG_ERROR("ggml-hex: %s dspcall : dsp-rsp %s\n", this->c_name(), status_to_str(rsp.status));+ this->last_error = rsp.status;+ for (auto & sub : this->mdev.sessions) {+ sub->last_error = rsp.status;+ }}op_queue->pop(rsp, dbuf);- this->op_pending--; // atomic dec+ GGML_ASSERT(rsp.seq == this->batch_rsp_seq + 1);+ this->batch_rsp_seq = rsp.seq;if (!all) break;}}+void ggml_hexagon_session::flush_sync(bool all) {+ flush_async();+ flush_pending(all);+}+void ggml_hexagon_session::flush_batch(size_t min_ops) {if (op_batch->n_ops < min_ops) { return; }+ op_batch->sort_buffers();+htp_opbatch_req req {};dspqueue_buffer dbuf{};- if (!op_queue->push(req, dbuf, op_batch)) {+ const uint64_t seq = ++this->batch_req_seq;++ op_batch->update_mdev_group(this->mdev.idx);++ if (!op_queue->push(req, dbuf, op_batch, seq)) {flush_pending(false);- op_queue->push(req, dbuf, op_batch);+ op_queue->push(req, dbuf, op_batch, seq);}- // Bump pending flag (cleared in the session::flush once we get the response)- this->op_pending++; // atomic inc+ for (auto & sub : this->mdev.sessions) {+ htp_opbatch_req sub_req {};+ dspqueue_buffer sub_dbuf{};++ sub->batch_req_seq = seq;+ op_batch->update_mdev_group(sub->mdev.idx);++ if (!sub->op_queue->push(sub_req, sub_dbuf, op_batch, seq)) {+ sub->flush_pending(false);+ sub->op_queue->push(sub_req, sub_dbuf, op_batch, seq);+ }++ HEX_VERBOSE("ggml-hex: %s queue-opbatch: %p size %u\n", sub->c_name(), sub_dbuf.ptr, sub_dbuf.size);++ int err = dspqueue_write(sub->queue, 0, 1, &sub_dbuf, sizeof(sub_req), (const uint8_t*) &sub_req, DSPQUEUE_TIMEOUT);+ if (err != 0) {+ GGML_ABORT("ggml-hex: %s dspqueue_write failed: 0x%08x\n", sub->c_name(), (unsigned) err);+ }+ }HEX_VERBOSE("ggml-hex: %s queue-opbatch: %p size %u\n", this->c_name(), dbuf.ptr, dbuf.size);@@ -2779,28 +2899,28 @@ void ggml_hexagon_session::flush_batch(size_t min_ops) {if (err != 0) {GGML_ABORT("ggml-hex: %s dspqueue_write failed: 0x%08x\n", this->c_name(), (unsigned) err);}-}-void ggml_hexagon_session::flush(bool all) {- flush_sync_peers();- flush_batch();- flush_pending(all);+ op_batch->reset();}void ggml_hexagon_session::enqueue_op(const htp_opnode & node) {- for (auto t : node.get_inputs()) {+ auto clone_tensor_buffer = [this](const ggml_tensor * t) {if (t && t->buffer && ggml_backend_buffer_is_hexagon(t->buffer)) {+ auto sbuf = static_cast<const ggml_hexagon_shared_buffer *>(t->buffer->context);if (ggml_backend_hexagon_buffer_get_sess(t->buffer) != this) {- this->clone_buffer(static_cast<const ggml_hexagon_shared_buffer *>(t->buffer->context));+ this->clone_buffer(sbuf);+ }+ for (auto & sub : this->mdev.sessions) {+ sub->clone_buffer(sbuf);}}+ };++ for (auto t : node.get_inputs()) {+ clone_tensor_buffer(t);}for (auto t : node.get_outputs()) {- if (t && t->buffer && ggml_backend_buffer_is_hexagon(t->buffer)) {- if (ggml_backend_hexagon_buffer_get_sess(t->buffer) != this) {- this->clone_buffer(static_cast<const ggml_hexagon_shared_buffer *>(t->buffer->context));- }- }+ clone_tensor_buffer(t);}if (opt_opfusion && op_batch->try_fuse(node)) {@@ -2808,39 +2928,84 @@ void ggml_hexagon_session::enqueue_op(const htp_opnode & node) {}if (!op_batch->fit_op(node)) {- flush_batch();+ flush_async();}++ if (this->mdev.count > 1 && op_batch->n_ops == 0) {+ enqueue_mdev_group();+ }+op_batch->add_op(node);}+void ggml_hexagon_session::enqueue_mdev_group() {+ htp_opnode group_node(HTP_OP_MDEV_GROUP);++ uint8_t * fence_slot = this->mdev_fence_slot;++ static ggml_hexagon_tensor_extra fence_extra { {}, 0, GGML_HEXAGON_TENSOR_FENCE };+ ggml_tensor dummy_t {};+ dummy_t.buffer = &this->fence_buf->backend_buffer;+ dummy_t.extra = &fence_extra;+ dummy_t.data = (void *) fence_slot;+ dummy_t.type = GGML_TYPE_I8;+ dummy_t.ne[0] = HTP_FENCE_SLOT_SIZE;+ dummy_t.ne[1] = (int64_t) this->mdev.count;+ dummy_t.ne[2] = 1;+ dummy_t.ne[3] = 1;+ dummy_t.nb[0] = 1;+ dummy_t.nb[1] = HTP_FENCE_SLOT_SIZE;+ dummy_t.nb[2] = dummy_t.nb[1] * dummy_t.ne[1];+ dummy_t.nb[3] = dummy_t.nb[2];+ dummy_t.op = GGML_OP_NONE;+ dummy_t.op_params[0] = (int32_t) this->mdev.idx;++ ggml_tensor * node = group_node.add_dummy(dummy_t);+ node->src[0] = node;+ group_node.init(node);+ group_node.outputs.clear();+ group_node.name = "MDEV_GROUP";++ if (this->fence_buf->sess != this) {+ this->clone_buffer(this->fence_buf);+ }+ for (auto & sub : this->mdev.sessions) {+ sub->clone_buffer(this->fence_buf);+ }++ op_batch->add_op(group_node);+}+void ggml_hexagon_session::enqueue_cpy(const ggml_tensor * src, ggml_tensor * dst, const ggml_tensor * sync_tensor, uint32_t fence_seq) {- htp_opnode cpy_node(HTP_OP_CPY);+ const bool with_fence = sync_tensor != nullptr;+ htp_opnode cpy_node(with_fence ? HTP_OP_CPY_FENCE : HTP_OP_CPY);ggml_tensor* node = cpy_node.add_dummy(*dst);node->op = GGML_OP_CPY;node->src[0] = const_cast<ggml_tensor *>(src);- node->src[1] = sync_tensor ? cpy_node.add_dummy(*sync_tensor) : nullptr;- if (sync_tensor) {+ node->src[1] = with_fence ? cpy_node.add_dummy(*sync_tensor) : nullptr;+ if (with_fence) {node->op_params[0] = (int32_t) fence_seq;}cpy_node.init(node);- if (sync_tensor) {+ if (with_fence) {cpy_node.name = "CPY+FENCE";}this->enqueue_op(cpy_node);}-void ggml_hexagon_session::enqueue_fence(const ggml_tensor * sync_tensor, uint32_t fence_seq) {+void ggml_hexagon_session::enqueue_fence(const ggml_tensor * sync_tensor, uint32_t fence_seq, bool wait) {htp_opnode sync_node(HTP_OP_FENCE);ggml_tensor* node = sync_node.add_dummy(*sync_tensor);node->op = GGML_OP_NONE;node->src[0] = node;node->op_params[0] = (int32_t) fence_seq;+ node->op_params[1] = wait ? 0 : 1;sync_node.init(node);- sync_node.name = "FENCE";+ sync_node.name = wait ? "FENCE_WAIT" : "FENCE_SIGNAL";this->enqueue_op(sync_node);}@@ -2858,7 +3023,6 @@ static bool ggml_hexagon_precompute_allreduce_params(kparams->n_ranks = (int32_t) n_ranks;kparams->is_row_bcast = (has_add && is_row_bcast) ? 1 : 0;- const uint32_t n_bufs = n_ranks + 1 + (has_add ? 1 : 0);const uint32_t nelem = (uint32_t) ggml_nelements(dst);const uint32_t elem_size = (dst->type == GGML_TYPE_F16) ? sizeof(ggml_fp16_t) : sizeof(float);const bool is_contiguous = ggml_is_contiguous(dst);@@ -2902,6 +3066,7 @@ static bool ggml_hexagon_precompute_allreduce_params(const uint32_t rank_nelem = (uint32_t) kparams->rank_nelem;const uint32_t n_threads = (std::min)((uint32_t) sess->n_threads, (std::max)(1u, rank_nelem / 128));kparams->n_threads = n_threads;+ const size_t n_vtcm_buffers = htp_allreduce_vtcm_buffer_count(n_ranks, n_threads, has_add, is_row_bcast);uint32_t block_elems = 65536;if (block_elems > rank_nelem / n_threads && rank_nelem / n_threads > 128) {@@ -2911,15 +3076,15 @@ static bool ggml_hexagon_precompute_allreduce_params(kparams->block_elems = block_elems;kparams->vtcm_size_per_thread = 2 * block_elems * elem_size;- kparams->vtcm_size = n_threads * n_bufs * kparams->vtcm_size_per_thread;+ kparams->vtcm_size = n_vtcm_buffers * kparams->vtcm_size_per_thread;while ((size_t) kparams->vtcm_size > sess->vtcm_size && block_elems > 128) {- const size_t max_bytes_per_buf = sess->vtcm_size / (n_threads * n_bufs * 2);+ const size_t max_bytes_per_buf = sess->vtcm_size / (n_vtcm_buffers * 2);block_elems = (uint32_t) hex_align_down((size_t) (max_bytes_per_buf / elem_size), 128);if (block_elems < 128) break;kparams->block_elems = block_elems;kparams->vtcm_size_per_thread = 2 * block_elems * elem_size;- kparams->vtcm_size = n_threads * n_bufs * kparams->vtcm_size_per_thread;+ kparams->vtcm_size = n_vtcm_buffers * kparams->vtcm_size_per_thread;}if (sess->vtcm_size < (size_t) kparams->vtcm_size || block_elems < 128) {@@ -2935,6 +3100,7 @@ static bool ggml_hexagon_precompute_allreduce_params(const uint32_t rank_nrows = (uint32_t) kparams->rank_nelem;const uint32_t n_threads = (std::min)((uint32_t) sess->n_threads, (std::max)(1u, rank_nrows));kparams->n_threads = n_threads;+ const size_t n_vtcm_buffers = htp_allreduce_vtcm_buffer_count(n_ranks, n_threads, has_add, is_row_bcast);const uint32_t row_bytes = ne0 * elem_size;const uint32_t row_size_aligned = (uint32_t) hex_align_up(row_bytes, 128);@@ -2946,14 +3112,14 @@ static bool ggml_hexagon_precompute_allreduce_params(kparams->block_elems = block_rows;kparams->vtcm_size_per_thread = 2 * (block_rows * row_size_aligned);- kparams->vtcm_size = n_threads * n_bufs * kparams->vtcm_size_per_thread;+ kparams->vtcm_size = n_vtcm_buffers * kparams->vtcm_size_per_thread;while ((size_t) kparams->vtcm_size > sess->vtcm_size && block_rows > 1) {- const size_t max_rows_per_buf = sess->vtcm_size / (n_threads * n_bufs * 2 * row_size_aligned);+ const size_t max_rows_per_buf = sess->vtcm_size / (n_vtcm_buffers * 2 * row_size_aligned);block_rows = (std::max)(1u, (uint32_t) max_rows_per_buf);kparams->block_elems = block_rows;kparams->vtcm_size_per_thread = 2 * (block_rows * row_size_aligned);- kparams->vtcm_size = n_threads * n_bufs * kparams->vtcm_size_per_thread;+ kparams->vtcm_size = n_vtcm_buffers * kparams->vtcm_size_per_thread;if (max_rows_per_buf == 0) break;}@@ -3009,28 +3175,20 @@ void ggml_hexagon_session::enqueue_allreduce(this->enqueue_op(ar_node);}-void ggml_hexagon_session::wait_event(uint64_t seq) {- flush_sync_peers();- HEX_VERBOSE("ggml-hex: %s opqueue-wait start: seq %llu, current rsp-seq %llu, pending %d\n",- this->name.c_str(), (unsigned long long)seq, (unsigned long long)op_queue->rsp_seq, (int)this->op_pending);- while (op_queue->rsp_seq < seq && this->op_pending > 0) {- this->flush_pending(false);- }- HEX_VERBOSE("ggml-hex: %s opqueue-wait end: seq %llu, current rsp-seq %llu, pending %d\n",- this->name.c_str(), (unsigned long long)seq, (unsigned long long)op_queue->rsp_seq, (int)this->op_pending);-}--uint64_t ggml_hexagon_session::record_event() {- flush_batch();- return op_queue->req_seq;-}-bool ggml_hexagon_session::clone_buffer(const ggml_hexagon_shared_buffer *sbuf){- if (this->cloned_buffers.find(sbuf->fd()) != this->cloned_buffers.end()) return true;+ GGML_ASSERT(sbuf && sbuf->mem);+ if (sbuf->sess == this) return true;++ auto mem = sbuf->mem;+ int fd = mem->fd;++ GGML_ASSERT(fd >= 0);++ if (this->cloned_buffers.find(fd) != this->cloned_buffers.end()) return true;HEX_VERBOSE("ggml-hex: %s clone-buffer: %s base %p size %zu fd %d\n", this->name.c_str(),- sbuf->c_name(), sbuf->base(), sbuf->size(), sbuf->fd());+ sbuf->c_name(), sbuf->base(), sbuf->size(), fd);auto clone = std::make_unique<ggml_hexagon_shared_buffer>(this, *sbuf);try {@@ -3040,10 +3198,38 @@ bool ggml_hexagon_session::clone_buffer(const ggml_hexagon_shared_buffer *sbuf)return false;}- this->cloned_buffers[sbuf->fd()] = std::move(clone);+ this->cloned_buffers[fd] = std::move(clone);+ mem->mapped_clones.insert(this);return true;}+void ggml_hexagon_session::release_buffer(const ggml_hexagon_shared_buffer * sbuf) {+ GGML_ASSERT(sbuf && sbuf->mem);++ auto mem = sbuf->mem;+ int fd = mem->fd;++ GGML_ASSERT(fd >= 0);++ auto it = this->cloned_buffers.find(fd);+ if (it != this->cloned_buffers.end()) {+ auto clone = std::move(it->second);+ this->cloned_buffers.erase(it);+ }+ mem->mapped_clones.erase(this);+}++void ggml_hexagon_session::unclone_buffer(const ggml_hexagon_shared_buffer * sbuf) {+ GGML_ASSERT(sbuf && sbuf->mem);++ auto mem = sbuf->mem;+ std::vector<ggml_hexagon_session *> sessions(mem->mapped_clones.begin(), mem->mapped_clones.end());++ for (auto * sess : sessions) {+ sess->release_buffer(sbuf);+ }+}+static size_t ggml_hexagon_measure_max_vmem(ggml_hexagon_session *sess) {// Allocate a bunch pinned buffers till failure.// This is kind of expensive but handy for figuring out exactly how much we can mmap on a specific device.@@ -3082,14 +3268,16 @@ void ggml_hexagon_session::allocate(const ggml_hexagon_device_config & config) nthis->valid_queue = false;this->valid_iface = false;- this->phys_idx = phys_idx;- this->virt_idx = virt_idx;- this->domain_id = config.domain_id;- this->session_id = 0;- this->name = config.name;- this->op_pending = 0;+ this->name = config.name;+ this->phys_idx = phys_idx;+ this->virt_idx = virt_idx;+ this->domain_id = config.domain_id;+ this->session_id = 0;+ this->batch_req_seq = 0;+ this->batch_rsp_seq = 0;+ this->last_error = HTP_STATUS_OK;- GGML_LOG_DEBUG("ggml-hex: %s allocating new session\n", this->name.c_str());+ GGML_LOG_DEBUG("ggml-hex: %s allocating new session : domain %u phys-idx %u virt-idx %u\n", this->name.c_str(), this->domain_id, phys_idx, virt_idx);if (config.domain_id < 0 || config.domain_name.empty()) {GGML_LOG_ERROR("ggml-hex: %s: invalid physical CDSP core %d\n", config.name.c_str(), config.physical_idx);@@ -3098,25 +3286,14 @@ void ggml_hexagon_session::allocate(const ggml_hexagon_device_config & config) nconst std::string & dom_name = config.domain_name;- // Enable Unsigned PD for all domains- {- struct remote_rpc_control_unsigned_module u;- u.domain = -1;- u.enable = 1;- int err = remote_session_control(DSPRPC_CONTROL_UNSIGNED_MODULE, (void *) &u, sizeof(u));- if (err != AEE_SUCCESS) {- GGML_LOG_ERROR("ggml-hex: %s failed to enable unsigned PD : error 0x%x\n", this->c_name(), err);- throw std::runtime_error("ggml-hex: remote_session_control(unsign) failed (see log for details)");- }- }-// Create new session if virtual_idx > 0if (virt_idx > 0) {- struct remote_rpc_reserve_new_session n;+ struct remote_rpc_reserve_new_session n {};n.domain_name_len = dom_name.size();n.domain_name = const_cast<char *>(dom_name.c_str());n.session_name = const_cast<char *>(this->name.c_str());n.session_name_len = this->name.size();+ n.session_id = virt_idx;int err = remote_session_control(FASTRPC_RESERVE_NEW_SESSION, (void *) &n, sizeof(n));if (err != AEE_SUCCESS) {@@ -3130,7 +3307,7 @@ void ggml_hexagon_session::allocate(const ggml_hexagon_device_config & config) nthis->domain_id = n.effective_domain_id;this->valid_session = true;} else {- struct remote_rpc_effective_domain_id eff = {};+ struct remote_rpc_effective_domain_id eff {};eff.domain_name = const_cast<char *>(dom_name.c_str());eff.domain_name_len = dom_name.size();eff.session_id = 0;@@ -3144,6 +3321,18 @@ void ggml_hexagon_session::allocate(const ggml_hexagon_device_config & config) n}}+ // Enable unsigned modules+ {+ struct remote_rpc_control_unsigned_module u;+ u.domain = this->domain_id;+ u.enable = 1;+ int err = remote_session_control(DSPRPC_CONTROL_UNSIGNED_MODULE, (void *) &u, sizeof(u));+ if (err != AEE_SUCCESS) {+ GGML_LOG_ERROR("ggml-hex: %s failed to enable unsigned PD : error 0x%x\n", this->c_name(), err);+ throw std::runtime_error("ggml-hex: remote_session_control(unsign) failed (see log for details)");+ }+ }+char session_uri[256];{char htp_uri[256];@@ -3171,7 +3360,7 @@ void ggml_hexagon_session::allocate(const ggml_hexagon_device_config & config) n// Open sessionint err = htp_iface_open(session_uri, &this->handle);if (err != AEE_SUCCESS) {- GGML_LOG_ERROR("ggml-hex: %s failed to open session : error 0x%x\n", this->c_name(), err);+ GGML_LOG_ERROR("ggml-hex: %s failed to open session : uri %s error 0x%x\n", this->c_name(), session_uri, err);throw std::runtime_error("ggml-hex: failed to open session (see log for details)");}@@ -3186,8 +3375,9 @@ void ggml_hexagon_session::allocate(const ggml_hexagon_device_config & config) nunsigned long long hw_vtcm_size = 0;int hw_err = htp_iface_hwinfo(this->handle, &hw_n_threads, &hw_n_hvx, &hw_n_hmx, &hw_vtcm_size);if (hw_err == 0) {- this->n_threads = opt_nhvx > 0 ? (uint32_t)opt_nhvx : (uint32_t)hw_n_threads;- this->n_hvx = opt_nhvx > 0 ? (uint32_t)opt_nhvx : (uint32_t)hw_n_hvx;+ const uint32_t max_n_threads = (std::min)((uint32_t) HTP_MAX_NTHREADS, (uint32_t) hw_n_threads);+ this->n_threads = opt_nhvx > 0 ? (uint32_t) (std::min)(opt_nhvx, (size_t) max_n_threads) : max_n_threads;+ this->n_hvx = this->n_threads;this->n_hmx = (opt_nhmx != 0) ? (uint32_t)hw_n_hmx : 0;this->vtcm_size = (uint64_t)hw_vtcm_size;GGML_LOG_INFO("ggml-hex: %s hwinfo: threads %u, hvx %u, hmx %u, vtcm %llu MB\n",@@ -3195,8 +3385,9 @@ void ggml_hexagon_session::allocate(const ggml_hexagon_device_config & config) n(unsigned long long)(this->vtcm_size / (1024 * 1024)));} else {GGML_LOG_WARN("ggml-hex: %s failed to query hwinfo (0x%x), using defaults\n", this->c_name(), hw_err);- this->n_threads = opt_nhvx > 0 ? (uint32_t)opt_nhvx : 8;- this->n_hvx = opt_nhvx > 0 ? (uint32_t)opt_nhvx : 8;+ const uint32_t default_n_threads = (std::min)(8u, (uint32_t) HTP_MAX_NTHREADS);+ this->n_threads = opt_nhvx > 0 ? (uint32_t) (std::min)(opt_nhvx, (size_t) HTP_MAX_NTHREADS) : default_n_threads;+ this->n_hvx = this->n_threads;this->n_hmx = (opt_nhmx != 0) ? 1 : 0;this->vtcm_size = 8 * 1024 * 1024;}@@ -3252,6 +3443,11 @@ void ggml_hexagon_session::allocate(const ggml_hexagon_device_config & config) n// Allocate buffers and state for op batchingthis->op_queue = new ggml_hexagon_opqueue(this, opt_opbatch, opt_opqueue);+ this->fence_buf = new ggml_hexagon_fence_buffer(this, &dev_ctx->fence_buffer_type, 64 * 1024);+ if (this->mdev.count > 1) {+ this->mdev_fence_slot = this->alloc_fence(this->mdev.count);+ }+if (!opt_vmem) {opt_vmem = ggml_hexagon_measure_max_vmem(this);GGML_LOG_INFO("ggml-hex: %s measured max vmem %zu\n", this->c_name(), opt_vmem);@@ -3262,7 +3458,7 @@ void ggml_hexagon_session::allocate(const ggml_hexagon_device_config & config) nthis->op_batch = new ggml_hexagon_opbatch(this, opt_opbatch, this->max_vmem);// Start dspqueue/opbatch processing- err = htp_iface_start(this->handle, this->session_id, this->queue_id, opt_nhvx, opt_nhmx, this->max_vmem);+ err = htp_iface_start(this->handle, this->session_id, this->queue_id, this->n_threads, opt_nhmx, this->max_vmem);if (err != 0) {GGML_LOG_ERROR("ggml-hex: %s failed to start session: 0x%08x\n", this->c_name(), (unsigned) err);throw std::runtime_error("ggml-hex: iface start failed (see log for details)");@@ -3283,6 +3479,8 @@ void ggml_hexagon_session::allocate(const ggml_hexagon_device_config & config) nvoid ggml_hexagon_session::release() noexcept(true) {GGML_LOG_INFO("ggml-hex: releasing session: %s\n", this->name.c_str());+ this->mdev.sessions.clear();+int err;if (this->valid_iface) {@@ -3295,6 +3493,19 @@ void ggml_hexagon_session::release() noexcept(true) {delete this->op_batch;delete this->op_queue;+ for (auto & it : this->cpy_fence_slots) {+ free_fence((void *) it.second, 1);+ }+ this->cpy_fence_slots.clear();++ if (this->fence_buf) {+ unclone_buffer(this->fence_buf);+ delete this->fence_buf;+ this->fence_buf = nullptr;+ }+ while (!this->cloned_buffers.empty()) {+ release_buffer(this->cloned_buffers.begin()->second.get());+ }if (opt_etm) {err = htp_iface_etm(this->handle, 0);@@ -3321,23 +3532,30 @@ void ggml_hexagon_session::release() noexcept(true) {if (this->valid_handle) {htp_iface_close(this->handle);}-- this->cloned_buffers.clear();}-ggml_hexagon_session::ggml_hexagon_session(const ggml_hexagon_device_config & config, ggml_backend_dev_t dev) noexcept(false) {- op_batch = nullptr;- op_queue = nullptr;- fence_seq = ((uintptr_t)this) & 0xFFFF;+ggml_hexagon_session::ggml_hexagon_session(const ggml_hexagon_device_config & config, ggml_backend_dev_t dev, uint32_t mdev_idx, uint32_t mdev_count) noexcept(false) {+ this->dev = dev;+ this->dev_ctx = static_cast<ggml_backend_hexagon_device_context *>(dev->context);+ this->mdev.idx = mdev_idx;+ this->mdev.count = mdev_count > 0 ? mdev_count : (uint32_t) (1 + config.mdev_group.size());+ op_batch = nullptr;+ op_queue = nullptr;+ fence_buf = nullptr;+ fence_seq = ((uintptr_t)this) & 0xFFFF;try {allocate(config);+ if (this->mdev.idx == 0 && !config.mdev_group.empty()) {+ for (size_t i = 0; i < config.mdev_group.size(); i++) {+ this->mdev.sessions.push_back(std::make_unique<ggml_hexagon_session>(+ config.mdev_group[i], this->dev, (uint32_t) (i + 1), this->mdev.count));+ }+ }} catch (const std::exception & exc) {release();throw;}-- GGML_UNUSED(dev);}ggml_hexagon_session::~ggml_hexagon_session() noexcept(true) {@@ -3563,10 +3781,6 @@ static bool ggml_hexagon_supported_gated_delta_net(const struct ggml_hexagon_sesconst struct ggml_tensor * state = op->src[5];const struct ggml_tensor * dst = op;- if (!q || !k || !v || !g || !beta || !state) {- return false;- }-if (q->type != GGML_TYPE_F32 || k->type != GGML_TYPE_F32 || v->type != GGML_TYPE_F32 ||g->type != GGML_TYPE_F32 || beta->type != GGML_TYPE_F32 || state->type != GGML_TYPE_F32 ||dst->type != GGML_TYPE_F32) {@@ -3754,6 +3968,7 @@ static void ggml_hexagon_precompute_hvx_mm_params(struct htp_mm_kernel_params * kparams) {kparams->n_hmx = 0;+ kparams->n_threads = sess->n_threads;const bool is_quant = (wtype != GGML_TYPE_F16 && wtype != GGML_TYPE_F32);const int src1_nrows = ne11 * ne12 * ne13;@@ -4193,6 +4408,7 @@ static void ggml_hexagon_precompute_fused_mmnx_params(struct htp_mm_kernel_params * kparams) {memset(kparams, 0, sizeof(*kparams));+ kparams->n_threads = sess->n_threads;const int ne00 = src0->ne[0];const int ne01 = src0->ne[1];@@ -4921,6 +5137,14 @@ static bool ggml_hexagon_supported_pad(const struct ggml_hexagon_session * sess,return false;}+ const int32_t lp0 = ((const int32_t *) op->op_params)[0];+ const int32_t rp0 = ((const int32_t *) op->op_params)[1];+ const int32_t circular = ((const int32_t *) op->op_params)[8];++ if (circular && (lp0 > src0->ne[0] || rp0 > src0->ne[0])) {+ return false;+ }+return true;GGML_UNUSED(sess);@@ -4972,10 +5196,6 @@ static bool ggml_hexagon_supported_solve_tri(const struct ggml_hexagon_session *const struct ggml_tensor * src1 = op->src[1]; // Bconst struct ggml_tensor * dst = op; // X- if (!src0 || !src1) {- return false;- }-if (src0->type != GGML_TYPE_F32 || src1->type != GGML_TYPE_F32 || dst->type != GGML_TYPE_F32) {return false;}@@ -5145,7 +5365,7 @@ static bool is_supported_mul_mat_id_nx_kernel(const ggml_tensor * src0, const st}static bool is_mergeable_mul_mat(const ggml_tensor * t) {- if (!t || t->op != GGML_OP_MUL_MAT) return false;+ if (t->op != GGML_OP_MUL_MAT) return false;const ggml_tensor * src0 = t->src[0];const ggml_tensor * src1 = t->src[1];@@ -5179,7 +5399,7 @@ static bool is_mergeable_mul_mat_pair(const ggml_tensor * n1, const ggml_tensor}static bool is_mergeable_mul_mat_id(const ggml_tensor * t) {- if (!t || t->op != GGML_OP_MUL_MAT_ID) return false;+ if (t->op != GGML_OP_MUL_MAT_ID) return false;const ggml_tensor * src0 = t->src[0];return ggml_hexagon_is_repack_type(src0->type);@@ -5213,6 +5433,10 @@ static bool is_mergeable_mul_mat_id_pair(const ggml_tensor * n1, const ggml_tensstatic ggml_status ggml_backend_hexagon_graph_compute(ggml_backend_t backend, ggml_cgraph * graph) {auto sess = static_cast<ggml_hexagon_session *>(backend->context);+ if (sess->last_error > HTP_STATUS_OK) {+ return GGML_STATUS_FAILED;+ }+HEX_VERBOSE("ggml-hex: %s graph-compute n_nodes %d\n", sess->c_name(), graph->n_nodes);const std::vector<htp_opnode> * nodes_ptr = nullptr;@@ -5228,6 +5452,8 @@ static ggml_status ggml_backend_hexagon_graph_compute(ggml_backend_t backend, ggauto * extra = (ggml_hexagon_tensor_extra *) graph->nodes[i]->extra;if (!extra) continue;+ extra->flags &= ~GGML_HEXAGON_TENSOR_FUSEABLE;+if (graph->nodes[i]->op == GGML_OP_RMS_NORM && ggml_can_fuse(graph, i, { GGML_OP_RMS_NORM, GGML_OP_MUL })) {extra->flags |= GGML_HEXAGON_TENSOR_FUSEABLE;} else if (graph->nodes[i]->op == GGML_OP_MUL_MAT || graph->nodes[i]->op == GGML_OP_MUL_MAT_ID) {@@ -5299,6 +5525,10 @@ static ggml_status ggml_backend_hexagon_graph_compute(ggml_backend_t backend, ggsess->enqueue_op(node);}+ if (sess->last_error > HTP_STATUS_OK) {+ return GGML_STATUS_FAILED;+ }+return GGML_STATUS_SUCCESS;}@@ -5308,7 +5538,10 @@ static void ggml_backend_hexagon_synchronize(ggml_backend_t backend) {HEX_VERBOSE("ggml-hex: %s synchronize\n", sess->c_name());// Wait until all pending ops complete- sess->flush();+ sess->flush_sync();+ if (sess->last_error > HTP_STATUS_OK) {+ GGML_ABORT("ggml-hex: %s synchronize failed : dsp-error %s\n", sess->c_name(), status_to_str(sess->last_error));+ }}enum ggml_hexagon_mem_range_type {@@ -5543,27 +5776,38 @@ static void ggml_backend_hexagon_graph_optimize(ggml_backend_t backend, ggml_cgrGGML_UNUSED(backend);}+static uint64_t ggml_hexagon_session_key(const ggml_hexagon_session * sess) {+ return ((uint64_t) (uint32_t) sess->phys_idx << 32) | (uint32_t) sess->virt_idx;+}+static bool ggml_hexagon_cpy_tensor_async_phys(ggml_backend_t backend_src, ggml_backend_t backend_dst, const ggml_tensor * src, ggml_tensor * dst) {auto sess_src = static_cast<ggml_hexagon_session *>(backend_src->context);auto sess_dst = static_cast<ggml_hexagon_session *>(backend_dst->context);auto sbuf_dst = (ggml_hexagon_shared_buffer *) dst->buffer->context;- if (sess_dst->fence_seq == 0) sess_dst->fence_seq = 1;- uint32_t fence_seq = sess_dst->fence_seq++;- if (sess_dst->fence_seq == 0) sess_dst->fence_seq = 1;+ if (!sess_src->clone_buffer(sbuf_dst)) { return false; }- volatile uint32_t * fence = (volatile uint32_t *) sbuf_dst->alloc_fence();+ const uint64_t src_key = ggml_hexagon_session_key(sess_src);+ auto & fence_slot = sess_dst->cpy_fence_slots[src_key];+ if (!fence_slot) {+ fence_slot = (volatile uint32_t *) sess_dst->alloc_fence(1);+ }++ if (!sess_src->clone_buffer(sess_dst->fence_buf)) { return false; }++ if (++sess_dst->fence_seq == 0) sess_dst->fence_seq = 1;+ uint32_t fence_seq = sess_dst->fence_seq;- HEX_VERBOSE("ggml-hex: %s cpy-tensor-async %s -> %s size %zu : seq %u\n",+ HEX_VERBOSE("ggml-hex: %s cpy-tensor-async %s -> %s size %zu : seq 0x%x\n",sess_dst->name.c_str(), src->name, dst->name, ggml_nbytes(src), fence_seq);- // dummy extra (must be static)+ // dummy fence extra (must be static)static ggml_hexagon_tensor_extra fence_extra { {}, 0, GGML_HEXAGON_TENSOR_FENCE };ggml_tensor fence_tensor {};- fence_tensor.buffer = dst->buffer;+ fence_tensor.buffer = &sess_dst->fence_buf->backend_buffer;fence_tensor.extra = &fence_extra;- fence_tensor.data = (void *) fence;+ fence_tensor.data = (void *) fence_slot;fence_tensor.type = GGML_TYPE_I32;fence_tensor.ne[0] = 1;fence_tensor.ne[1] = 1;@@ -5576,9 +5820,9 @@ static bool ggml_hexagon_cpy_tensor_async_phys(ggml_backend_t backend_src, ggml_fence_tensor.op = GGML_OP_NONE;sess_src->enqueue_cpy(src, dst, &fence_tensor, fence_seq);- sess_dst->enqueue_fence(&fence_tensor, fence_seq);+ sess_dst->enqueue_fence(&fence_tensor, fence_seq, /* wait = */ true);- sess_dst->add_sync_peer(sess_src);+ sess_dst->add_peer(sess_src);return true;}@@ -5586,15 +5830,15 @@ static bool ggml_hexagon_cpy_tensor_async_phys(ggml_backend_t backend_src, ggml_static bool ggml_hexagon_cpy_tensor_async_virt(ggml_backend_t backend_src, ggml_backend_t backend_dst, const ggml_tensor * src, ggml_tensor * dst) {auto sess_src = static_cast<ggml_hexagon_session *>(backend_src->context);auto sess_dst = static_cast<ggml_hexagon_session *>(backend_dst->context);- auto sbuf_dst = (ggml_hexagon_shared_buffer *) dst->buffer->context;+ auto sbuf_src = (ggml_hexagon_shared_buffer *) src->buffer->context;- if (!sess_src->clone_buffer(sbuf_dst)) { return false; }+ if (!sess_dst->clone_buffer(sbuf_src)) { return false; }HEX_VERBOSE("ggml-hex: %s cpy-tensor-async %s -> %s size %zu\n",sess_dst->name.c_str(), src->name, dst->name, ggml_nbytes(src));- sess_src->enqueue_cpy(src, dst);- sess_src->flush(true);+ sess_dst->enqueue_cpy(src, dst);+ sess_dst->add_peer(sess_src);return true;}@@ -5604,7 +5848,14 @@ static bool ggml_backend_hexagon_cpy_tensor_async(ggml_backend_t backend_src, ggreturn false;}- *(ggml_hexagon_tensor_extra *) dst->extra = *(const ggml_hexagon_tensor_extra *) src->extra;+ // FIXME: ggml-meta needs to call init_tensor on auxiliary tensors+ if (!dst->extra) {+ ggml_backend_buffer_init_tensor(dst->buffer, dst);+ }++ auto * dst_extra = static_cast<ggml_hexagon_tensor_extra *>(dst->extra);+ const auto * src_extra = static_cast<const ggml_hexagon_tensor_extra *>(src->extra);+ dst_extra->flags = src_extra->flags & ~GGML_HEXAGON_TENSOR_FUSEABLE;auto sess_src = static_cast<ggml_hexagon_session *>(backend_src->context);auto sess_dst = static_cast<ggml_hexagon_session *>(backend_dst->context);@@ -5612,7 +5863,6 @@ static bool ggml_backend_hexagon_cpy_tensor_async(ggml_backend_t backend_src, ggif (sess_src == sess_dst) {HEX_VERBOSE("ggml-hex: %s cpy-tensor-async %s -> %s size %zu\n", sess_dst->name.c_str(), src->name, dst->name, ggml_nbytes(src));sess_src->enqueue_cpy(src, dst);- sess_src->flush_batch();return true;}@@ -5623,8 +5873,30 @@ static bool ggml_backend_hexagon_cpy_tensor_async(ggml_backend_t backend_src, gg}static ggml_backend_event_t ggml_backend_hexagon_device_event_new(ggml_backend_dev_t dev) {+ auto dev_ctx = static_cast<ggml_backend_hexagon_device_context *>(dev->context);+ auto sess = dev_ctx->session();+ggml_hexagon_event * hex_event = new ggml_hexagon_event();- HEX_VERBOSE("ggml-hex: %s event-new : event %p\n", ggml_backend_dev_name(dev), (void *)hex_event);+ hex_event->fence_sess = sess;+ hex_event->sess = sess;+ hex_event->fence_slot = (volatile uint32_t *) sess->alloc_fence(1);++ static ggml_hexagon_tensor_extra fence_extra { {}, 0, GGML_HEXAGON_TENSOR_FENCE };+ hex_event->fence_tensor.buffer = &sess->fence_buf->backend_buffer;+ hex_event->fence_tensor.extra = &fence_extra;+ hex_event->fence_tensor.data = (void *) hex_event->fence_slot;+ hex_event->fence_tensor.type = GGML_TYPE_I32;+ hex_event->fence_tensor.ne[0] = 1;+ hex_event->fence_tensor.ne[1] = 1;+ hex_event->fence_tensor.ne[2] = 1;+ hex_event->fence_tensor.ne[3] = 1;+ hex_event->fence_tensor.nb[0] = sizeof(int32_t);+ hex_event->fence_tensor.nb[1] = sizeof(int32_t);+ hex_event->fence_tensor.nb[2] = sizeof(int32_t);+ hex_event->fence_tensor.nb[3] = sizeof(int32_t);+ hex_event->fence_tensor.op = GGML_OP_NONE;++ HEX_VERBOSE("ggml-hex: %s event-new : event %p fence %p\n", ggml_backend_dev_name(dev), (void *)hex_event, (void *)hex_event->fence_slot);return new ggml_backend_event {/* .device = */ dev,@@ -5632,49 +5904,83 @@ static ggml_backend_event_t ggml_backend_hexagon_device_event_new(ggml_backend_d};}-static void ggml_backend_hexagon_device_event_free(ggml_backend_dev_t dev, ggml_backend_event_t event) {- GGML_UNUSED(dev);-- if (event == nullptr) {+static void ggml_hexagon_event_synchronize(ggml_backend_dev_t dev, ggml_hexagon_event * hex_event) {+ if (hex_event->seq == 0) {return;}- ggml_hexagon_event * hex_event = (ggml_hexagon_event *)event->context;+ HEX_VERBOSE("ggml-hex: %s event-synchronize : event %p seq 0x%x fence %p\n",+ ggml_backend_dev_name(dev), (void *)hex_event, hex_event->seq, (void *)hex_event->fence_slot);++ auto * fence = reinterpret_cast<const volatile std::atomic<uint32_t> *>(hex_event->fence_slot);++ if ((int32_t)(fence[0].load(std::memory_order_relaxed) - hex_event->seq) < 0) {+ hex_event->sess->flush_async();+ }++ while (true) {+ if ((int32_t)(fence[0].load(std::memory_order_acquire) - hex_event->seq) >= 0) {+ uint32_t status = fence[1].load(std::memory_order_acquire);+ if (status > HTP_STATUS_OK) {+ GGML_ABORT("ggml-hex: %s event-synchronize failed : dsp-error %s\n",+ hex_event->sess->c_name(), status_to_str(status));+ }+ break;+ }+ std::this_thread::yield();+ }+}++static void ggml_backend_hexagon_device_event_free(ggml_backend_dev_t dev, ggml_backend_event_t event) {+ auto * hex_event = static_cast<ggml_hexagon_event *>(event->context);+ ggml_hexagon_event_synchronize(dev, hex_event);HEX_VERBOSE("ggml-hex: %s event-free : event %p\n", ggml_backend_dev_name(dev), (void *)hex_event);+ hex_event->fence_sess->free_fence((void *) hex_event->fence_slot, 1);delete hex_event;delete event;}static void ggml_backend_hexagon_device_event_synchronize(ggml_backend_dev_t dev, ggml_backend_event_t event) {- GGML_UNUSED(dev);-- ggml_hexagon_event * hex_event = (ggml_hexagon_event *)event->context;- HEX_VERBOSE("ggml-hex: %s event-synchronize : event %p seq %llu\n",- ggml_backend_dev_name(dev), (void *)hex_event, (unsigned long long)hex_event->seq);- if (hex_event->sess != nullptr) {- hex_event->sess->wait_event(hex_event->seq);- }+ auto * hex_event = static_cast<ggml_hexagon_event *>(event->context);+ ggml_hexagon_event_synchronize(dev, hex_event);}static void ggml_backend_hexagon_event_record(ggml_backend_t backend, ggml_backend_event_t event) {auto sess = static_cast<ggml_hexagon_session *>(backend->context);- ggml_hexagon_event * hex_event = (ggml_hexagon_event *)event->context;+ auto hex_event = static_cast<ggml_hexagon_event *>(event->context);+ if (++sess->fence_seq == 0) sess->fence_seq = 1;hex_event->sess = sess;- hex_event->seq = sess->record_event();- HEX_VERBOSE("ggml-hex: %s event-record : event %p seq %llu\n",- sess->c_name(), (void *)hex_event, (unsigned long long)hex_event->seq);+ hex_event->seq = sess->fence_seq;++ sess->enqueue_fence(&hex_event->fence_tensor, hex_event->seq, /* wait = */ false);++ HEX_VERBOSE("ggml-hex: %s event-record : event %p seq 0x%x fence %p\n",+ sess->c_name(), (void *)hex_event, hex_event->seq, (void *)hex_event->fence_slot);}static void ggml_backend_hexagon_event_wait(ggml_backend_t backend, ggml_backend_event_t event) {- GGML_UNUSED(backend);+ auto sess = static_cast<ggml_hexagon_session *>(backend->context);+ auto hex_event = static_cast<ggml_hexagon_event *>(event->context);++ if (hex_event->seq == 0) {+ return;+ }++ HEX_VERBOSE("ggml-hex: %s event-wait : event %p seq 0x%x fence %p\n",+ sess->c_name(), (void *)hex_event, hex_event->seq, (void *)hex_event->fence_slot);- ggml_hexagon_event * hex_event = (ggml_hexagon_event *)event->context;- if (hex_event->sess != nullptr) {- HEX_VERBOSE("ggml-hex: %s event-wait : event %p seq %llu\n",- hex_event->sess->c_name(), (void *)hex_event, (unsigned long long)hex_event->seq);- hex_event->sess->wait_event(hex_event->seq);+ // same physical NPU runs sequentially in FIFO order+ if (sess->phys_idx == hex_event->sess->phys_idx) {+ if (sess != hex_event->sess) {+ sess->add_peer(hex_event->sess);+ }+ return;}++ sess->clone_buffer(hex_event->fence_sess->fence_buf);+ sess->add_peer(hex_event->sess);+ sess->enqueue_fence(&hex_event->fence_tensor, hex_event->seq, /* wait = */ true);}static void ggml_backend_hexagon_set_tensor_async(ggml_backend_t backend, struct ggml_tensor * tensor, const void * data, size_t offset, size_t size) {@@ -5688,7 +5994,10 @@ static void ggml_backend_hexagon_get_tensor_async(ggml_backend_t backend, constauto sess = static_cast<ggml_hexagon_session *>(backend->context);HEX_VERBOSE("ggml-hex: %s get-tensor-async %s : data %p offset %zu size %zu usage %d\n",sess->c_name(), tensor->name, data, offset, size, tensor->buffer ? (int) tensor->buffer->usage : -1);- sess->flush(true);+ sess->flush_sync();+ if (sess->last_error > HTP_STATUS_OK) {+ GGML_ABORT("ggml-hex: %s get-tensor-async failed : dsp-error %s\n", sess->c_name(), status_to_str(sess->last_error));+ }ggml_backend_tensor_get(tensor, data, offset, size);}@@ -5717,7 +6026,10 @@ static void ggml_backend_hexagon_get_tensor_2d_async(ggml_backend_t backend,auto sess = static_cast<ggml_hexagon_session *>(backend->context);HEX_VERBOSE("ggml-hex: %s get-tensor-2d-async %s : data %p offset %zu size %zu n_copies %zu stride_tensor %zu stride_data %zu usage %d\n",sess->c_name(), tensor->name, data, offset, size, n_copies, stride_tensor, stride_data, tensor->buffer ? (int) tensor->buffer->usage : -1);- sess->flush(true);+ sess->flush_sync();+ if (sess->last_error > HTP_STATUS_OK) {+ GGML_ABORT("ggml-hex: %s get-tensor-2d-async failed : dsp-error %s\n", sess->c_name(), status_to_str(sess->last_error));+ }ggml_backend_tensor_get_2d(tensor, data, offset, size, n_copies, stride_tensor, stride_data);}@@ -6083,13 +6395,7 @@ static bool ggml_backend_hexagon_device_supports_op(ggml_backend_dev_t dev, consstatic bool ggml_backend_hexagon_device_supports_buft(ggml_backend_dev_t dev, ggml_backend_buffer_type_t buft) {auto dev_ctx = static_cast<ggml_backend_hexagon_device_context *>(dev->context);- // Technically we can clone hexagon buffers from any session but for some reason the output is garbled with layer-split,- // tensor-split works correctly, so it needs mode debugging and investigation. For now accept only our own buffers.-#if 0- bool supp = (buft->iface.get_alignment == ggml_backend_hexagon_buffer_type_get_alignment);-#elsebool supp = (buft == &dev_ctx->host_buffer_type) || (buft == &dev_ctx->buffer_type);-#endifHEX_VERBOSE("ggml-hex: %s device-supports-buft %s %s\n", dev_ctx->c_name(), ggml_backend_buft_name(buft), supp ? "yes" : "no");return supp;@@ -6122,6 +6428,19 @@ ggml_hexagon_registry::ggml_hexagon_registry(ggml_backend_reg_t reg) {// Create devicesfor (size_t i = 0; i < opt_ndev; i++) {+ const auto & cfg = opt_device_configs[i];+ if (cfg.mdev_group.empty()) {+ GGML_LOG_INFO("ggml-hex: device %zu: %s (phys=%d, virt=%d, domain=%s:%d)\n",+ i, cfg.name.c_str(), cfg.physical_idx, cfg.virtual_idx, cfg.domain_name.c_str(), cfg.domain_id);+ } else {+ std::string peers_str;+ for (const auto & p : cfg.mdev_group) {+ if (!peers_str.empty()) peers_str += ", ";+ peers_str += p.name + " (phys=" + std::to_string(p.physical_idx) + ")";+ }+ GGML_LOG_INFO("ggml-hex: device %zu: %s (phys=%d, virt=%d, domain=%s:%d) [mdev peers: %s]\n",+ i, cfg.name.c_str(), cfg.physical_idx, cfg.virtual_idx, cfg.domain_name.c_str(), cfg.domain_id, peers_str.c_str());+ }devices[i].iface = ggml_backend_hexagon_device_i;devices[i].reg = reg;devices[i].context = new ggml_backend_hexagon_device_context(i, opt_device_configs[i], &devices[i]);@@ -6172,17 +6491,51 @@ static void * ggml_backend_hexagon_comm_init(ggml_backend_t * backends, size_t n}}+ for (size_t i = 0; i < n_backends; i++) {+ auto sess_i = static_cast<ggml_hexagon_session *>(backends[i]->context);+ for (size_t j = i + 1; j < n_backends; j++) {+ auto sess_j = static_cast<ggml_hexagon_session *>(backends[j]->context);+ if (sess_i->phys_idx == sess_j->phys_idx) {+ return nullptr;+ }+ }+ }+auto * ctx = new ggml_backend_hexagon_comm_context();ctx->backends.assign(backends, backends + n_backends);ctx->n_backends = n_backends;- ctx->fence_seq = (((uintptr_t) ctx) & 0xFFFF) | 1;++ static ggml_hexagon_tensor_extra fence_extra { {}, 0, GGML_HEXAGON_TENSOR_FENCE };+ for (size_t i = 0; i < n_backends; i++) {+ auto sess_i = static_cast<ggml_hexagon_session *>(backends[i]->context);+ ctx->fence_slots[i] = (volatile uint32_t *) sess_i->alloc_fence(1);+ ctx->fence_tensors[i] = {};+ ctx->fence_tensors[i].buffer = &sess_i->fence_buf->backend_buffer;+ ctx->fence_tensors[i].extra = &fence_extra;+ ctx->fence_tensors[i].data = (void *) ctx->fence_slots[i];+ ctx->fence_tensors[i].type = GGML_TYPE_I32;+ ctx->fence_tensors[i].ne[0] = 4;+ ctx->fence_tensors[i].ne[1] = 1;+ ctx->fence_tensors[i].ne[2] = 1;+ ctx->fence_tensors[i].ne[3] = 1;+ ctx->fence_tensors[i].nb[0] = sizeof(int32_t);+ ctx->fence_tensors[i].nb[1] = sizeof(int32_t);+ ctx->fence_tensors[i].nb[2] = sizeof(int32_t);+ ctx->fence_tensors[i].nb[3] = sizeof(int32_t);+ ctx->fence_tensors[i].op = GGML_OP_NONE;+ }return ctx;}static void ggml_backend_hexagon_comm_free(void * comm_ctx_v) {if (!comm_ctx_v) return;- delete static_cast<ggml_backend_hexagon_comm_context *>(comm_ctx_v);+ auto * ctx = static_cast<ggml_backend_hexagon_comm_context *>(comm_ctx_v);+ for (size_t i = 0; i < ctx->n_backends; i++) {+ auto sess_i = static_cast<ggml_hexagon_session *>(ctx->backends[i]->context);+ sess_i->free_fence((void *) ctx->fence_slots[i], 1);+ }+ delete ctx;}static bool ggml_backend_hexagon_comm_allreduce_tensor(void * comm_ctx_v, struct ggml_tensor ** tensors) {@@ -6192,6 +6545,16 @@ static bool ggml_backend_hexagon_comm_allreduce_tensor(void * comm_ctx_v, structif (n_backends < 2 || n_backends > 4) return false;+ for (size_t i = 0; i < n_backends; i++) {+ auto sess_i = static_cast<ggml_hexagon_session *>(comm_ctx->backends[i]->context);+ for (size_t j = i + 1; j < n_backends; j++) {+ auto sess_j = static_cast<ggml_hexagon_session *>(comm_ctx->backends[j]->context);+ if (sess_i->phys_idx == sess_j->phys_idx) {+ return false;+ }+ }+ }+for (size_t i = 0; i < n_backends; i++) {if (!tensors[i] || !tensors[i]->buffer || !ggml_backend_buffer_is_hexagon(tensors[i]->buffer)) {return false;@@ -6219,42 +6582,28 @@ static bool ggml_backend_hexagon_comm_allreduce_tensor(void * comm_ctx_v, struct}}- if (comm_ctx->fence_seq == 0) comm_ctx->fence_seq = 1;- uint32_t fence_seq_entry = comm_ctx->fence_seq++;- if (comm_ctx->fence_seq == 0) comm_ctx->fence_seq = 1;- uint32_t fence_seq_exit = comm_ctx->fence_seq++;- if (comm_ctx->fence_seq == 0) comm_ctx->fence_seq = 1;-- volatile uint32_t * fences[GGML_HEXAGON_MAX_SESSIONS];- for (size_t i = 0; i < n_backends; i++) {- auto sbuf = (ggml_hexagon_shared_buffer *) tensors[i]->buffer->context;- fences[i] = (volatile uint32_t *) sbuf->alloc_fence();+ uint32_t max_seq = static_cast<ggml_hexagon_session *>(comm_ctx->backends[0]->context)->fence_seq;+ for (size_t i = 1; i < n_backends; i++) {+ auto sess_i = static_cast<ggml_hexagon_session *>(comm_ctx->backends[i]->context);+ if ((int32_t)(sess_i->fence_seq - max_seq) > 0) {+ max_seq = sess_i->fence_seq;+ }}+ if (++max_seq == 0) max_seq = 1;+ uint32_t fence_seq_entry = max_seq;+ if (++max_seq == 0) max_seq = 1;+ uint32_t fence_seq_exit = max_seq;- static ggml_hexagon_tensor_extra fence_extra { {}, 0, GGML_HEXAGON_TENSOR_FENCE };- ggml_tensor fence_tensors[GGML_HEXAGON_MAX_SESSIONS];for (size_t i = 0; i < n_backends; i++) {- fence_tensors[i] = {};- fence_tensors[i].buffer = tensors[i]->buffer;- fence_tensors[i].extra = &fence_extra;- fence_tensors[i].data = (void *) fences[i];- fence_tensors[i].type = GGML_TYPE_I32;- fence_tensors[i].ne[0] = 4;- fence_tensors[i].ne[1] = 1;- fence_tensors[i].ne[2] = 1;- fence_tensors[i].ne[3] = 1;- fence_tensors[i].nb[0] = sizeof(int32_t);- fence_tensors[i].nb[1] = sizeof(int32_t);- fence_tensors[i].nb[2] = sizeof(int32_t);- fence_tensors[i].nb[3] = sizeof(int32_t);- fence_tensors[i].op = GGML_OP_NONE;+ auto sess_i = static_cast<ggml_hexagon_session *>(comm_ctx->backends[i]->context);+ sess_i->fence_seq = max_seq;}std::vector<const ggml_tensor *> data_tensors(n_backends);std::vector<const ggml_tensor *> sync_tensors(n_backends);for (size_t i = 0; i < n_backends; i++) {data_tensors[i] = tensors[i];- sync_tensors[i] = &fence_tensors[i];+ sync_tensors[i] = &comm_ctx->fence_tensors[i];}for (size_t r = 0; r < n_backends; r++) {@@ -6262,7 +6611,7 @@ static bool ggml_backend_hexagon_comm_allreduce_tensor(void * comm_ctx_v, structsess->enqueue_allreduce(tensors[r], data_tensors, sync_tensors, (uint32_t) r, (uint32_t) n_backends, fence_seq_entry, fence_seq_exit);for (size_t j = 0; j < n_backends; j++) {if (r != j) {- sess->add_sync_peer(static_cast<ggml_hexagon_session *>(comm_ctx->backends[j]->context));+ sess->add_peer(static_cast<ggml_hexagon_session *>(comm_ctx->backends[j]->context));}}}@@ -6270,8 +6619,23 @@ static bool ggml_backend_hexagon_comm_allreduce_tensor(void * comm_ctx_v, structreturn true;}+static ggml_backend_buffer_type_t ggml_backend_hexagon_split_buffer_type(int main_device, const float * tensor_split) {+ GGML_UNUSED(tensor_split);+ auto reg = ggml_backend_hexagon_reg();+ auto dev = ggml_backend_reg_dev_get(reg, main_device);+ if (!dev) {+ dev = ggml_backend_reg_dev_get(reg, 0);+ }+ if (!dev) return nullptr;+ auto dev_ctx = static_cast<ggml_backend_hexagon_device_context *>(dev->context);+ return &dev_ctx->buffer_type;+}+static void * ggml_backend_hexagon_get_proc_address(ggml_backend_reg_t reg, const char * name) {GGML_UNUSED(reg);+ if (strcmp(name, "ggml_backend_split_buffer_type") == 0) {+ return (void *) ggml_backend_hexagon_split_buffer_type;+ }if (strcmp(name, "ggml_backend_comm_init") == 0) {return (void *) ggml_backend_hexagon_comm_init;}@@ -6304,6 +6668,41 @@ template<typename T, int BASE=10> std::string vec_to_str(std::vector<T> v) {return str;}+static void ggml_hexagon_resolve_device_domain(ggml_hexagon_device_config & cfg, bool discovery_supported, const std::unordered_map<int, fastrpc_domain> & cdsp_map) {+ if (discovery_supported) {+ auto it = cdsp_map.find(cfg.physical_idx);+ if (it != cdsp_map.end()) {+ cfg.domain_id = it->second.id;+ cfg.domain_name = it->second.name;+ } else {+ GGML_LOG_ERROR("ggml-hex: physical CDSP core %d not found on device (%zu CDSP core(s) available)\n",+ cfg.physical_idx, cdsp_map.size());+ cfg.domain_id = -1;+ cfg.domain_name = "";+ }+ } else {+ switch (cfg.physical_idx) {+ case 0:+ cfg.domain_id = 3;+ cfg.domain_name = CDSP_DOMAIN_NAME;+ break;+ case 1:+ cfg.domain_id = 4;+ cfg.domain_name = "cdsp1";+ break;+ default:+ GGML_LOG_ERROR("ggml-hex: physical CDSP core %d not supported without dynamic discovery\n",+ cfg.physical_idx);+ cfg.domain_id = -1;+ cfg.domain_name = "";+ break;+ }+ }+ for (auto & sub_cfg : cfg.mdev_group) {+ ggml_hexagon_resolve_device_domain(sub_cfg, discovery_supported, cdsp_map);+ }+}+// Enumerate NPU (aka CDSP) domains via FASTRPC_GET_DOMAINS if supported,// and populate domain_id and domain_name for all configured devices.static void ggml_hexagon_discover_devices() {@@ -6350,36 +6749,7 @@ static void ggml_hexagon_discover_devices() {// Populate domain IDs and names for all configured devicesfor (size_t i = 0; i < opt_ndev; i++) {- auto & cfg = opt_device_configs[i];- if (discovery_supported) {- auto it = cdsp_map.find(cfg.physical_idx);- if (it != cdsp_map.end()) {- cfg.domain_id = it->second.id;- cfg.domain_name = it->second.name;- } else {- GGML_LOG_ERROR("ggml-hex: physical CDSP core %d not found on device (%zu CDSP core(s) available)\n",- cfg.physical_idx, cdsp_map.size());- cfg.domain_id = -1;- cfg.domain_name = "";- }- } else {- switch (cfg.physical_idx) {- case 0:- cfg.domain_id = 3;- cfg.domain_name = CDSP_DOMAIN_NAME;- break;- case 1:- cfg.domain_id = 4;- cfg.domain_name = "cdsp1";- break;- default:- GGML_LOG_ERROR("ggml-hex: physical CDSP core %d not supported without dynamic discovery\n",- cfg.physical_idx);- cfg.domain_id = -1;- cfg.domain_name = "";- break;- }- }+ ggml_hexagon_resolve_device_domain(opt_device_configs[i], discovery_supported, cdsp_map);}}@@ -6487,21 +6857,126 @@ static void ggml_hexagon_init(ggml_backend_reg * reg) {opt_device_configs[i].physical_idx = 0;opt_device_configs[i].virtual_idx = (int)i;opt_device_configs[i].name = "HTP" + std::to_string(i);+ opt_device_configs[i].mdev_group.clear();}} else {std::string s_devices(str_devices);- std::stringstream ss(s_devices);- std::string item;- opt_ndev = 0;- while (std::getline(ss, item, ',')) {- size_t start = item.find_first_not_of(" \t\r\n");- size_t end = item.find_last_not_of(" \t\r\n");- if (start == std::string::npos) {- continue;+ std::vector<std::string> items;+ std::string curr_item;+ int bracket_depth = 0;+ for (char ch : s_devices) {+ if (ch == '[') {+ bracket_depth++;+ curr_item += ch;+ } else if (ch == ']') {+ if (bracket_depth > 0) bracket_depth--;+ curr_item += ch;+ } else if (ch == ',' && bracket_depth == 0) {+ size_t s = curr_item.find_first_not_of(" \t\r\n");+ size_t e = curr_item.find_last_not_of(" \t\r\n");+ if (s != std::string::npos) {+ items.push_back(curr_item.substr(s, e - s + 1));+ }+ curr_item.clear();+ } else {+ curr_item += ch;}- item = item.substr(start, end - start + 1);+ }+ size_t s = curr_item.find_first_not_of(" \t\r\n");+ size_t e = curr_item.find_last_not_of(" \t\r\n");+ if (s != std::string::npos) {+ items.push_back(curr_item.substr(s, e - s + 1));+ }++ opt_ndev = 0;+ for (const auto & item : items) {+ size_t b_open = item.find('[');+ size_t b_close = item.rfind(']');++ if (b_open != std::string::npos && b_close != std::string::npos && b_close > b_open) {+ // Grouped / composite syntax: Name[phys_spec:virt] or Name[phys_spec]+ std::string dev_name = item.substr(0, b_open);+ std::string content = item.substr(b_open + 1, b_close - b_open - 1);- if (item.rfind("HTP", 0) == 0) {+ int virt = 0;+ std::string phys_spec = content;+ size_t colon_pos = content.find(':');+ if (colon_pos != std::string::npos) {+ phys_spec = content.substr(0, colon_pos);+ try {+ virt = std::stoi(content.substr(colon_pos + 1));+ } catch (...) {+ virt = 0;+ }+ } else {+ size_t dev_colon = dev_name.find(':');+ if (dev_colon != std::string::npos) {+ try {+ virt = std::stoi(dev_name.substr(dev_colon + 1));+ } catch (...) {+ virt = 0;+ }+ }+ }++ // Parse physical indices from phys_spec (e.g. 0-1, 0,1, 0-3, etc.)+ std::vector<int> phys_list;+ std::stringstream pss(phys_spec);+ std::string p_part;+ while (std::getline(pss, p_part, ',')) {+ size_t ps = p_part.find_first_not_of(" \t\r\n");+ size_t pe = p_part.find_last_not_of(" \t\r\n");+ if (ps == std::string::npos) continue;+ p_part = p_part.substr(ps, pe - ps + 1);++ size_t dash_pos = p_part.find('-');+ if (dash_pos != std::string::npos) {+ try {+ int p_start = std::stoi(p_part.substr(0, dash_pos));+ int p_end = std::stoi(p_part.substr(dash_pos + 1));+ for (int p = p_start; p <= p_end; p++) {+ if (std::find(phys_list.begin(), phys_list.end(), p) == phys_list.end()) {+ phys_list.push_back(p);+ }+ }+ } catch (...) {+ GGML_LOG_WARN("ggml-hex: failed to parse physical range in '%s'\n", p_part.c_str());+ }+ } else {+ try {+ int p = std::stoi(p_part);+ if (std::find(phys_list.begin(), phys_list.end(), p) == phys_list.end()) {+ phys_list.push_back(p);+ }+ } catch (...) {+ GGML_LOG_WARN("ggml-hex: failed to parse physical index in '%s'\n", p_part.c_str());+ }+ }+ }++ if (phys_list.empty()) {+ phys_list.push_back(0);+ }++ if (opt_ndev < GGML_HEXAGON_MAX_SESSIONS) {+ auto & cfg = opt_device_configs[opt_ndev];+ cfg.name = dev_name;+ cfg.physical_idx = phys_list[0];+ cfg.virtual_idx = virt;+ cfg.mdev_group.clear();++ for (size_t k = 1; k < phys_list.size(); k++) {+ ggml_hexagon_device_config sub_cfg;+ sub_cfg.physical_idx = phys_list[k];+ sub_cfg.virtual_idx = virt;+ sub_cfg.name = "HTP" + std::to_string(phys_list[k]) + ":" + std::to_string(virt);+ cfg.mdev_group.push_back(sub_cfg);+ }+ opt_ndev++;+ } else {+ GGML_LOG_WARN("ggml-hex: max sessions limit reached (%d), ignoring device %s\n", GGML_HEXAGON_MAX_SESSIONS, item.c_str());+ }+ } else if (item.rfind("HTP", 0) == 0) {std::string rest = item.substr(3);size_t colon_pos = rest.find(':');int phys = 0;@@ -6525,6 +7000,7 @@ static void ggml_hexagon_init(ggml_backend_reg * reg) {opt_device_configs[opt_ndev].name = colon_pos == std::string::npos? "HTP" + std::to_string(phys): "HTP" + std::to_string(phys) + ":" + std::to_string(virt);+ opt_device_configs[opt_ndev].mdev_group.clear();opt_ndev++;} else {GGML_LOG_WARN("ggml-hex: max sessions limit reached (%d), ignoring device %s\n", GGML_HEXAGON_MAX_SESSIONS, item.c_str());@@ -6539,6 +7015,7 @@ static void ggml_hexagon_init(ggml_backend_reg * reg) {opt_device_configs[0].physical_idx = 0;opt_device_configs[0].virtual_idx = 0;opt_device_configs[0].name = "HTP0";+ opt_device_configs[0].mdev_group.clear();}#if defined(__ANDROID__)diff --git a/ggml/src/ggml-hexagon/htp-opnode.h b/ggml/src/ggml-hexagon/htp-opnode.hindex b083e26718..ef7b5184fc 100644--- a/ggml/src/ggml-hexagon/htp-opnode.h+++ b/ggml/src/ggml-hexagon/htp-opnode.h@@ -344,6 +344,12 @@ struct htp_opformat {} else if (htp_op_is_unary(node.opcode)) {const auto * kparams = (const struct htp_unary_kernel_params *) node.kernel_params;snprintf(str, max_size, "%s vtcm %d", kparams->col_tile ? "wide-row" : "row-block", (int) kparams->vtcm_size);+ } else if (node.opcode == HTP_OP_MDEV_GROUP && node.node) {+ snprintf(str, max_size, "idx %d count %d", (int) node.node->op_params[0], (int) node.dst()->ne[1]);+ } else if ((node.opcode == HTP_OP_FENCE || node.opcode == HTP_OP_CPY_FENCE) && node.node) {+ snprintf(str, max_size, "seq 0x%x", (uint32_t) node.node->op_params[0]);+ } else if (node.opcode == HTP_OP_ALLREDUCE && node.node) {+ snprintf(str, max_size, "seq 0x%x -> 0x%x", (uint32_t) node.node->op_params[0], (uint32_t) node.node->op_params[1]);} else {snprintf(str, max_size, "----");}diff --git a/ggml/src/ggml-hexagon/htp/act-ops.c b/ggml/src/ggml-hexagon/htp/act-ops.cindex ac00b447d9..5fff372f28 100644--- a/ggml/src/ggml-hexagon/htp/act-ops.c+++ b/ggml/src/ggml-hexagon/htp/act-ops.c@@ -3,7 +3,6 @@#pragma clang diagnostic ignored "-Wunused-but-set-variable"#include <HAP_farf.h>-#include <HAP_perf.h>#include <math.h>#include <string.h>@@ -15,7 +14,7 @@#include "ggml-common.h"#include "htp-ctx.h"#include "htp-ops.h"-#include "htp-ops.h"+#include "hex-common.h"#include "htp-tensor.h"#include "htp-vtcm.h"@@ -80,6 +79,7 @@ struct htp_act_context {uint32_t block;uint32_t src0_nrows;uint32_t src0_nrows_per_thread;+ uint32_t row_start;int nc;uint8_t * vtcm_src0;@@ -329,104 +329,104 @@ static void geglu_f32(const float * restrict src0,}}-#define DEFINE_GLU_PER_THREAD(NAME, OP_STR, CORE_EXPR) \- static void glu_##NAME##_f32_per_thread(unsigned int nth, unsigned int ith, void * data) { \- struct htp_act_context * actx = (struct htp_act_context *) data; \- htp_act_preamble; \- \- struct htp_thread_trace * tr = actx->octx->ctx ? &actx->octx->ctx->trace[ith] : NULL; \- \- size_t src0_row_size = actx->src0_row_size; \- size_t src1_row_size = actx->src1_row_size; \- size_t dst_row_size = actx->dst_row_size; \- \- size_t src0_row_stride = actx->src0_row_stride; \- size_t src1_row_stride = actx->src1_row_stride; \- \- const uint32_t src0_nrows = actx->src0_nrows; \- const uint32_t src0_nrows_per_thread = actx->src0_nrows_per_thread; \- \- const uint32_t src0_start_row = src0_nrows_per_thread * ith; \- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, src0_nrows); \- \- /* no work for this thread */ \- if (src0_start_row >= src0_end_row) { \- return; \- } \- \- const uint8_t * restrict data_src0 = actx->data_src0; \- const uint8_t * restrict data_src1 = actx->data_src1; \- uint8_t * restrict data_dst = actx->data_dst; \- \- const size_t src0_row_size_aligned = actx->src0_row_size_aligned; \- const size_t src1_row_size_aligned = actx->src1_row_size_aligned; \- const size_t dst_row_size_aligned = actx->dst_row_size_aligned; \- \- uint8_t * restrict src0_spad_data = actx->vtcm_src0 + (ith * actx->vtcm_src0_size_per_thread); \- uint8_t * restrict src1_spad_data = actx->vtcm_src1 + (ith * actx->vtcm_src1_size_per_thread); \- uint8_t * restrict dst_spad_data = actx->vtcm_dst + (ith * actx->vtcm_dst_size_per_thread); \- \- size_t src0_spad_half_size = actx->src0_spad_half_size; \- size_t src1_spad_half_size = actx->src1_spad_half_size; \- size_t dst_spad_half_size = actx->dst_spad_half_size; \- \- const int BLOCK = actx->block; \- if (BLOCK == 0) { \- FARF(ERROR, \- OP_STR \- " : current VTCM reservation %zu is too small for even 1 row per thread, needed at least %zu\n", \- actx->vtcm_src0_size_per_thread, src0_row_size_aligned); \- return; \- } \- \- dma_queue * dma_queue = actx->octx->ctx->dma[ith]; \- \- /* See discussion: https://github.com/ggml-org/llama.cpp/pull/18151#issuecomment-3678235379 */ \- for (uint32_t ir = src0_start_row, spad_idx = 0; ir < src0_end_row && spad_idx < 2; ir += BLOCK, spad_idx++) { \- const uint32_t block_size = MIN(BLOCK, src0_end_row - ir); \- \- /* Dummy DMA transation for sequencing (interleaving dst,src,dst,...) */ \- dma_queue_push_vtcm_to_ddr(dma_queue, \- dma_make_ptr(data_dst, dst_spad_data + (spad_idx * dst_spad_half_size)), \- dst_row_size, dst_row_size_aligned, 0); \- \- dma_queue_push( \- dma_queue, \- dma_make_ptr(src0_spad_data + (spad_idx * src0_spad_half_size), data_src0 + (ir * src0_row_stride)), \- src0_row_size_aligned, src0_row_stride, src0_row_size, block_size); \- dma_queue_push( \- dma_queue, \- dma_make_ptr(src1_spad_data + (spad_idx * src1_spad_half_size), data_src1 + (ir * src1_row_stride)), \- src1_row_size_aligned, src1_row_stride, src1_row_size, block_size); \- } \- \- for (uint32_t ir = src0_start_row; ir < src0_end_row; ir += BLOCK) { \- const uint32_t block_size = MIN(BLOCK, src0_end_row - ir); \- \- float * dst_spad = (float *) dma_queue_pop(dma_queue).src; \- float * src0_spad = (float *) dma_queue_pop(dma_queue).dst; \- float * src1_spad = (float *) dma_queue_pop(dma_queue).dst; \- \- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ir); \- CORE_EXPR; \- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ir); \- \- dma_queue_push_vtcm_to_ddr(dma_queue, dma_make_ptr(data_dst + (ir * dst_row_size), dst_spad), \- dst_row_size, dst_row_size_aligned, block_size); \- \- /* prefetch N+2 loop iteration if any */ \- const uint32_t pref_block = (ir + BLOCK * 2); \- if (pref_block < src0_end_row) { \- const uint32_t pref_block_size = MIN(BLOCK, src0_end_row - pref_block); \- dma_queue_push(dma_queue, dma_make_ptr(src0_spad, data_src0 + (pref_block * src0_row_stride)), \- src0_row_size_aligned, src0_row_stride, src0_row_size, pref_block_size); \- dma_queue_push(dma_queue, dma_make_ptr(src1_spad, data_src1 + (pref_block * src1_row_stride)), \- src1_row_size_aligned, src1_row_stride, src1_row_size, pref_block_size); \- } \- } \- \- dma_queue_flush(dma_queue); \- \+#define DEFINE_GLU_PER_THREAD(NAME, OP_STR, CORE_EXPR) \+ static void glu_##NAME##_f32_per_thread(unsigned int nth, unsigned int ith, void * data) { \+ struct htp_act_context * actx = (struct htp_act_context *) data; \+ htp_act_preamble; \+ \+ struct htp_thread_trace * tr = actx->octx->ctx ? &actx->octx->ctx->trace[ith] : NULL; \+ \+ size_t src0_row_size = actx->src0_row_size; \+ size_t src1_row_size = actx->src1_row_size; \+ size_t dst_row_size = actx->dst_row_size; \+ \+ size_t src0_row_stride = actx->src0_row_stride; \+ size_t src1_row_stride = actx->src1_row_stride; \+ \+ const uint32_t src0_nrows = actx->src0_nrows; \+ const uint32_t src0_nrows_per_thread = actx->src0_nrows_per_thread; \+ \+ const uint32_t src0_start_row = actx->row_start + src0_nrows_per_thread * ith; \+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, actx->row_start + src0_nrows); \+ \+ /* no work for this thread */ \+ if (src0_start_row >= src0_end_row) { \+ return; \+ } \+ \+ const uint8_t * restrict data_src0 = actx->data_src0; \+ const uint8_t * restrict data_src1 = actx->data_src1; \+ uint8_t * restrict data_dst = actx->data_dst; \+ \+ const size_t src0_row_size_aligned = actx->src0_row_size_aligned; \+ const size_t src1_row_size_aligned = actx->src1_row_size_aligned; \+ const size_t dst_row_size_aligned = actx->dst_row_size_aligned; \+ \+ uint8_t * restrict src0_spad_data = actx->vtcm_src0 + (ith * actx->vtcm_src0_size_per_thread); \+ uint8_t * restrict src1_spad_data = actx->vtcm_src1 + (ith * actx->vtcm_src1_size_per_thread); \+ uint8_t * restrict dst_spad_data = actx->vtcm_dst + (ith * actx->vtcm_dst_size_per_thread); \+ \+ size_t src0_spad_half_size = actx->src0_spad_half_size; \+ size_t src1_spad_half_size = actx->src1_spad_half_size; \+ size_t dst_spad_half_size = actx->dst_spad_half_size; \+ \+ const int BLOCK = actx->block; \+ if (BLOCK == 0) { \+ FARF(ERROR, \+ OP_STR \+ " : current VTCM reservation %zu is too small for even 1 row per thread, needed at least %zu\n", \+ actx->vtcm_src0_size_per_thread, src0_row_size_aligned); \+ return; \+ } \+ \+ dma_queue * dma_queue = actx->octx->ctx->dma[ith]; \+ \+ /* See discussion: https://github.com/ggml-org/llama.cpp/pull/18151#issuecomment-3678235379 */ \+ for (uint32_t ir = src0_start_row, spad_idx = 0; ir < src0_end_row && spad_idx < 2; ir += BLOCK, spad_idx++) { \+ const uint32_t block_size = MIN(BLOCK, src0_end_row - ir); \+ \+ /* Dummy DMA transation for sequencing (interleaving dst,src,dst,...) */ \+ dma_queue_push_vtcm_to_ddr(dma_queue, \+ dma_make_ptr(data_dst, dst_spad_data + (spad_idx * dst_spad_half_size)), \+ dst_row_size, dst_row_size_aligned, 0); \+ \+ dma_queue_push( \+ dma_queue, \+ dma_make_ptr(src0_spad_data + (spad_idx * src0_spad_half_size), data_src0 + (ir * src0_row_stride)), \+ src0_row_size_aligned, src0_row_stride, src0_row_size, block_size); \+ dma_queue_push( \+ dma_queue, \+ dma_make_ptr(src1_spad_data + (spad_idx * src1_spad_half_size), data_src1 + (ir * src1_row_stride)), \+ src1_row_size_aligned, src1_row_stride, src1_row_size, block_size); \+ } \+ \+ for (uint32_t ir = src0_start_row; ir < src0_end_row; ir += BLOCK) { \+ const uint32_t block_size = MIN(BLOCK, src0_end_row - ir); \+ \+ float * dst_spad = (float *) dma_queue_pop(dma_queue).src; \+ float * src0_spad = (float *) dma_queue_pop(dma_queue).dst; \+ float * src1_spad = (float *) dma_queue_pop(dma_queue).dst; \+ \+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ir); \+ CORE_EXPR; \+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ir); \+ \+ dma_queue_push_vtcm_to_ddr(dma_queue, dma_make_ptr(data_dst + (ir * dst_row_size), dst_spad), \+ dst_row_size, dst_row_size_aligned, block_size); \+ \+ /* prefetch N+2 loop iteration if any */ \+ const uint32_t pref_block = (ir + BLOCK * 2); \+ if (pref_block < src0_end_row) { \+ const uint32_t pref_block_size = MIN(BLOCK, src0_end_row - pref_block); \+ dma_queue_push(dma_queue, dma_make_ptr(src0_spad, data_src0 + (pref_block * src0_row_stride)), \+ src0_row_size_aligned, src0_row_stride, src0_row_size, pref_block_size); \+ dma_queue_push(dma_queue, dma_make_ptr(src1_spad, data_src1 + (pref_block * src1_row_stride)), \+ src1_row_size_aligned, src1_row_stride, src1_row_size, pref_block_size); \+ } \+ } \+ \+ dma_queue_flush(dma_queue); \+ \}DEFINE_GLU_PER_THREAD(swiglu, "swiglu-f32", swiglu_f32(src0_spad, src1_spad, dst_spad, block_size, actx))@@ -473,14 +473,30 @@ static int execute_op_activations_f32(struct htp_ops_context * octx) {}const uint32_t src0_nrows = src0->ne[1] * src0->ne[2] * src0->ne[3];- const uint32_t n_threads = MIN(octx->n_threads, src0_nrows);+ const size_t dst_row_size = dst->ne[0] * SIZEOF_FP32;++ uint32_t row_start = 0;+ uint32_t nrows = src0_nrows;++ if (octx->ctx->mdev.count > 1) {+ uint32_t rows_per_chunk = 0;+ htp_tensor_mdev_rows_per_chunk(dst, sizeof(float), (uint32_t) dst_row_size, &rows_per_chunk);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(src0_nrows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ nrows = range.count;+ }++ if (nrows == 0) {+ return HTP_STATUS_OK;+ }++ const uint32_t n_threads = octx->n_threads;// row_size = bytes of useful data per row (what the kernel touches / what DMA copies).// row_stride = bytes between successive rows in DDR (may exceed row_size for non-contig src).- const size_t nc_bytes = dst->ne[0] * SIZEOF_FP32;- const size_t src0_row_size = nc_bytes;- const size_t src1_row_size = nc_bytes;- const size_t dst_row_size = nc_bytes;+ const size_t nc_bytes = dst_row_size;+ const size_t src0_row_size = nc_bytes;+ const size_t src1_row_size = nc_bytes;const size_t src0_row_stride = src0->nb[1];const size_t src1_row_stride = src1 ? src1->nb[1] : src0->nb[1];@@ -518,7 +534,7 @@ static int execute_op_activations_f32(struct htp_ops_context * octx) {struct htp_act_context actx;actx.octx = octx;- actx.src0_nrows_per_thread = (src0_nrows + n_threads - 1) / n_threads;+ actx.src0_nrows_per_thread = fastdiv(nrows + n_threads - 1, &octx->n_threads_div);actx.src0_row_size = src0_row_size;actx.src1_row_size = src1_row_size;@@ -545,7 +561,8 @@ static int execute_op_activations_f32(struct htp_ops_context * octx) {actx.dst_spad_half_size = L.dst_bytes_per_thread / 2;actx.block = actx.src0_spad_half_size / actx.src0_row_size_aligned;- actx.src0_nrows = src0_nrows;+ actx.src0_nrows = nrows;+ actx.row_start = row_start;actx.nc = dst->ne[0];@@ -570,7 +587,7 @@ static int execute_op_activations_f32(struct htp_ops_context * octx) {actx.data_src1 = data_src1;actx.data_dst = (uint8_t *) dst->data;- worker_pool_run_func(octx->ctx->worker_pool, act_op_func, &actx, n_threads);+ work_queue_run(octx->ctx->work_queue, act_op_func, &actx, n_threads);return HTP_STATUS_OK;}diff --git a/ggml/src/ggml-hexagon/htp/allreduce-ops.c b/ggml/src/ggml-hexagon/htp/allreduce-ops.cindex d35f685a6d..d6e7f0d10c 100644--- a/ggml/src/ggml-hexagon/htp/allreduce-ops.c+++ b/ggml/src/ggml-hexagon/htp/allreduce-ops.c@@ -17,6 +17,7 @@#include "hex-dma.h"#include "hex-profile.h"#include "allreduce-ops.h"+#include "htp-fence.h"struct htp_allreduce_context {struct htp_ops_context * octx;@@ -242,7 +243,42 @@ DEFINE_ALLREDUCE_THREAD_DMA_2D(add_f32, float, hvx_add_f32_aaa, 1, 0)DEFINE_ALLREDUCE_THREAD_DMA_2D(add_bcast_f16, __fp16, hvx_add_f16_aaa, 1, 1)DEFINE_ALLREDUCE_THREAD_DMA_2D(add_bcast_f32, float, hvx_add_f32_aaa, 1, 1)+static int validate_allreduce(+ struct htp_ops_context * octx,+ const struct htp_allreduce_kernel_params * kparams,+ uint32_t n_ranks+) {+ if (!htp_ops_context_set_n_threads(octx, (uint32_t) kparams->n_threads)) {+ return HTP_STATUS_INVAL_PARAMS;+ }++ if (kparams->vtcm_size_per_thread <= 0 || kparams->vtcm_size <= 0) {+ return HTP_STATUS_INVAL_PARAMS;+ }++ const bool has_add = (octx->op == HTP_OP_ALLREDUCE_ADD);+ const size_t n_vtcm_buffers = htp_allreduce_vtcm_buffer_count(+ n_ranks, octx->n_threads, has_add, kparams->is_row_bcast != 0);+ const size_t vtcm_size = n_vtcm_buffers * (size_t) kparams->vtcm_size_per_thread;+ if (vtcm_size != (size_t) kparams->vtcm_size) {+ return HTP_STATUS_INVAL_PARAMS;+ }+ if (vtcm_size > octx->ctx->vtcm_size) {+ return HTP_STATUS_VTCM_TOO_SMALL;+ }++ if (octx->dst->type != HTP_TYPE_F16 && octx->dst->type != HTP_TYPE_F32) {+ return HTP_STATUS_NO_SUPPORT;+ }++ return HTP_STATUS_OK;+}+int op_allreduce(struct htp_ops_context * octx) {+ if (octx->ctx->mdev.count > 1 && octx->ctx->mdev.idx > 0) {+ return HTP_STATUS_OK;+ }+const struct htp_allreduce_kernel_params * kparams = (const struct htp_allreduce_kernel_params *) octx->kernel_params;const struct htp_tensor * dst = octx->dst;@@ -253,38 +289,53 @@ int op_allreduce(struct htp_ops_context * octx) {return HTP_STATUS_INVAL_PARAMS;}- if (dst->type != HTP_TYPE_F16 && dst->type != HTP_TYPE_F32) {- return HTP_STATUS_NO_SUPPORT;+ const uint32_t fence_seq_entry = (uint32_t) octx->op_params[0];+ const uint32_t fence_seq_exit = (uint32_t) octx->op_params[1];++ const struct htp_tensor * my_sync = octx->src[n_ranks + rank];+ atomic_uint * my_fence = (atomic_uint *) (uintptr_t) my_sync->data;++ const int status = validate_allreduce(octx, kparams, n_ranks);+ if (status != HTP_STATUS_OK) {+ if (status == HTP_STATUS_NO_SUPPORT) {+ FARF(ERROR, "ggml-hex: allreduce unsupported type %d : rank %u\n", dst->type, rank);+ }+ htp_fence_write(my_fence, fence_seq_exit, status);+ return status;}+ const bool has_add = (octx->op == HTP_OP_ALLREDUCE_ADD);const uint32_t nelem = dst->ne[0] * dst->ne[1] * dst->ne[2] * dst->ne[3];- const uint32_t fence_seq_entry = (uint32_t) octx->op_params[0];- const uint32_t fence_seq_exit = (uint32_t) octx->op_params[1];// 1. Entry Barrier: Synchronize all ranks before readingstruct htp_thread_trace * tr0 = &octx->ctx->trace[0];htp_trace_event_start(tr0, HTP_TRACE_EVT_FENCE, (uint16_t) fence_seq_entry);- const struct htp_tensor * my_sync = octx->src[n_ranks + rank];- atomic_uint * my_fence = (atomic_uint *) my_sync->data;-- atomic_store(&my_fence[0], fence_seq_entry);- asm volatile ("syncht" : : : "memory");- Q6_dccleaninva_A((void *) my_fence);+ htp_fence_write(my_fence, fence_seq_entry, octx->status);for (uint32_t j = 0; j < n_ranks; j++) {if (j == rank) continue;const struct htp_tensor * peer_sync = octx->src[n_ranks + j];- atomic_uint * peer_fence = (atomic_uint *) peer_sync->data;+ atomic_uint * peer_fence = (atomic_uint *) (uintptr_t) peer_sync->data;uint64_t spins = 0;while (1) {- Q6_dccleaninva_A((void *) peer_fence);- uint32_t val = atomic_load(&peer_fence[0]);- if (val == fence_seq_entry || val == fence_seq_exit) {+ uint32_t peer_seq;+ uint32_t peer_status;+ htp_fence_read(peer_fence, &peer_seq, &peer_status);+ if ((int32_t)(peer_seq - fence_seq_entry) >= 0) {+ if (peer_status > HTP_STATUS_OK) {+ FARF(ERROR, "ggml-hex: allreduce entry peer %u failed with status %u\n", j, peer_status);+ htp_fence_write(my_fence, fence_seq_exit, peer_status);+ htp_trace_event_stop(tr0, HTP_TRACE_EVT_FENCE, (uint16_t) fence_seq_entry);+ return peer_status;+ }break;}if (++spins > HTP_FENCE_TIMEOUT) {- FARF(ERROR, "ggml-hex: allreduce entry fence-wait TIMEOUT: rank %u waiting on %u (fence %p seq %u)\n", rank, j, peer_fence, fence_seq_entry);+ FARF(ERROR, "ggml-hex: allreduce entry fence-wait TIMEOUT : rank %u waiting on %u fence %p seq 0x%x peer-seq 0x%x\n",+ rank, j, peer_fence, fence_seq_entry, peer_seq);+ htp_fence_write(my_fence, fence_seq_exit, HTP_STATUS_INTERNAL_ERR);+ htp_trace_event_stop(tr0, HTP_TRACE_EVT_FENCE, (uint16_t) fence_seq_entry);return HTP_STATUS_INTERNAL_ERR;}hex_pause();@@ -301,8 +352,6 @@ int op_allreduce(struct htp_ops_context * octx) {const uint32_t elems_per_thread = (uint32_t) kparams->elems_per_thread;const uint32_t vtcm_size_per_thread = (uint32_t) kparams->vtcm_size_per_thread;- const bool has_add = (octx->op == HTP_OP_ALLREDUCE_ADD);-struct htp_allreduce_context actx;actx.octx = octx;actx.n_ranks = n_ranks;@@ -339,6 +388,8 @@ int op_allreduce(struct htp_ops_context * octx) {}break;default:+ FARF(ERROR, "ggml-hex: allreduce unsupported kernel %d : rank %u\n", kparams->kernel_type, rank);+ htp_fence_write(my_fence, fence_seq_exit, HTP_STATUS_NO_SUPPORT);return HTP_STATUS_NO_SUPPORT;}@@ -368,23 +419,31 @@ int op_allreduce(struct htp_ops_context * octx) {// 4. Exit Barrier: Synchronize all ranks after writinghtp_trace_event_start(tr0, HTP_TRACE_EVT_FENCE, (uint16_t) fence_seq_exit);- atomic_store(&my_fence[0], fence_seq_exit);- asm volatile ("syncht" : : : "memory");- Q6_dccleaninva_A((void *) my_fence);+ htp_fence_write(my_fence, fence_seq_exit, octx->status);for (uint32_t j = 0; j < n_ranks; j++) {if (j == rank) continue;const struct htp_tensor * peer_sync = octx->src[n_ranks + j];- atomic_uint * peer_fence = (atomic_uint *) peer_sync->data;+ atomic_uint * peer_fence = (atomic_uint *) (uintptr_t) peer_sync->data;uint64_t spins = 0;while (1) {- Q6_dccleaninva_A((void *) peer_fence);- uint32_t val = atomic_load(&peer_fence[0]);- if (val == fence_seq_exit) {+ uint32_t peer_seq;+ uint32_t peer_status;+ htp_fence_read(peer_fence, &peer_seq, &peer_status);+ if ((int32_t)(peer_seq - fence_seq_exit) >= 0) {+ if (peer_status > HTP_STATUS_OK) {+ FARF(ERROR, "ggml-hex: allreduce exit peer %u failed with status %u\n", j, peer_status);+ htp_fence_write(my_fence, fence_seq_exit, peer_status);+ htp_trace_event_stop(tr0, HTP_TRACE_EVT_FENCE, (uint16_t) fence_seq_exit);+ return peer_status;+ }break;}if (++spins > HTP_FENCE_TIMEOUT) {- FARF(ERROR, "ggml-hex: allreduce exit fence-wait TIMEOUT: rank %u waiting on %u (fence %p seq %u)\n", rank, j, peer_fence, fence_seq_exit);+ FARF(ERROR, "ggml-hex: allreduce exit fence-wait TIMEOUT : rank %u waiting on %u fence %p seq 0x%x peer-seq 0x%x\n",+ rank, j, peer_fence, fence_seq_exit, peer_seq);+ htp_fence_write(my_fence, fence_seq_exit, HTP_STATUS_INTERNAL_ERR);+ htp_trace_event_stop(tr0, HTP_TRACE_EVT_FENCE, (uint16_t) fence_seq_exit);return HTP_STATUS_INTERNAL_ERR;}hex_pause();@@ -394,5 +453,5 @@ int op_allreduce(struct htp_ops_context * octx) {htp_trace_event_stop(tr0, HTP_TRACE_EVT_FENCE, (uint16_t) fence_seq_exit);- return HTP_STATUS_OK;+ return octx->status;}diff --git a/ggml/src/ggml-hexagon/htp/allreduce-ops.h b/ggml/src/ggml-hexagon/htp/allreduce-ops.hindex de447d87e9..0aed2b8b7e 100644--- a/ggml/src/ggml-hexagon/htp/allreduce-ops.h+++ b/ggml/src/ggml-hexagon/htp/allreduce-ops.h@@ -2,6 +2,8 @@#define ALLREDUCE_OPS_H#include <stdint.h>+#include <stddef.h>+#include <stdbool.h>#define HTP_ALLREDUCE_MAX_RANKS 4@@ -15,6 +17,15 @@ enum htp_allreduce_kernel_type {HTP_ALLREDUCE_KERNEL_DMA_2D,};+static inline size_t htp_allreduce_vtcm_buffer_count(+ uint32_t n_ranks,+ uint32_t n_threads,+ bool has_add,+ bool is_row_bcast+) {+ return (size_t) (n_ranks + 1) * n_threads + (has_add ? (is_row_bcast ? 1 : n_threads) : 0);+}+struct htp_allreduce_kernel_params {int32_t rank;int32_t n_ranks;diff --git a/ggml/src/ggml-hexagon/htp/argsort-ops.c b/ggml/src/ggml-hexagon/htp/argsort-ops.cindex 774faef5f3..e3c49e763d 100644--- a/ggml/src/ggml-hexagon/htp/argsort-ops.c+++ b/ggml/src/ggml-hexagon/htp/argsort-ops.c@@ -11,9 +11,10 @@#include "hvx-utils.h"#include "hex-dma.h"+#include "hex-common.h"#include "htp-ctx.h"#include "htp-ops.h"-#include "htp-ops.h"+#include "htp-tensor.h"#ifndef MIN#define MIN(a, b) ((a) < (b) ? (a) : (b))@@ -22,6 +23,9 @@struct htp_argsort_context {struct htp_ops_context * octx;uint32_t nrows_per_thread;+ uint32_t total_rows;+ uint32_t row_start;+ uint32_t row_end;uint8_t * vtcm_base;size_t vtcm_per_thread;};@@ -336,10 +340,9 @@ static void htp_argsort_f32_##ne00##_##order_name(unsigned int n, unsigned int iconst struct htp_tensor * src0 = octx->src[0]; \const struct htp_tensor * dst = octx->dst; \uint8_t * spad = actx->vtcm_base + actx->vtcm_per_thread * i; \- uint32_t total_rows = src0->ne[1] * src0->ne[2] * src0->ne[3]; \uint32_t rows_per_thread = actx->nrows_per_thread; \- uint32_t start_row = rows_per_thread * i; \- uint32_t end_row = MIN(start_row + rows_per_thread, total_rows); \+ uint32_t start_row = actx->row_start + rows_per_thread * i; \+ uint32_t end_row = MIN(start_row + rows_per_thread, actx->row_end); \size_t values_size = hex_round_up(ne00 * sizeof(float), 128); \float * values_buf = (float *) spad; \int32_t * indices_buf = (int32_t *) (spad + values_size); \@@ -386,9 +389,6 @@ static void htp_argsort_f32_fallback(unsigned int n, unsigned int i, void * data// Dimensionsuint32_t ne00 = src0->ne[0];- uint32_t ne01 = src0->ne[1];- uint32_t ne02 = src0->ne[2];- uint32_t ne03 = src0->ne[3];uint32_t nb01 = src0->nb[1];@@ -398,10 +398,9 @@ static void htp_argsort_f32_fallback(unsigned int n, unsigned int i, void * dataenum ggml_sort_order order = (enum ggml_sort_order) octx->op_params[0];// Rows to process- uint32_t total_rows = ne01 * ne02 * ne03;uint32_t rows_per_thread = actx->nrows_per_thread;- uint32_t start_row = rows_per_thread * i;- uint32_t end_row = MIN(start_row + rows_per_thread, total_rows);+ uint32_t start_row = actx->row_start + rows_per_thread * i;+ uint32_t end_row = MIN(start_row + rows_per_thread, actx->row_end);size_t values_size = hex_round_up(ne00 * sizeof(float), 128);uint32_t num_vec_ind_values = hmx_ceil_div(ne00, VLEN/(sizeof(int32_t)));@@ -451,8 +450,28 @@ int op_argsort(struct htp_ops_context * octx) {return HTP_STATUS_NO_SUPPORT;}- const uint32_t total_rows = octx->src[0]->ne[1] * octx->src[0]->ne[2] * octx->src[0]->ne[3];- const uint32_t n_threads = MIN(total_rows, octx->n_threads);+ const struct htp_tensor * src0 = octx->src[0];+ const struct htp_tensor * dst = octx->dst;++ const uint32_t total_rows = src0->ne[1] * src0->ne[2] * src0->ne[3];+ const size_t dst_row_size = dst->ne[0] * sizeof(int32_t);++ uint32_t row_start = 0;+ uint32_t row_end = total_rows;+ if (octx->ctx->mdev.count > 1) {+ uint32_t rows_per_chunk = 0;+ htp_tensor_mdev_rows_per_chunk(dst, sizeof(int32_t), (uint32_t) dst_row_size, &rows_per_chunk);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_rows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ row_end = range.start + range.count;+ }++ const uint32_t nrows = row_end - row_start;+ if (nrows == 0) {+ return HTP_STATUS_OK;+ }++ const uint32_t n_threads = octx->n_threads;// Allocate scratchpad// We need 1 row of float + 1 row of int32 per thread.@@ -478,7 +497,10 @@ int op_argsort(struct htp_ops_context * octx) {struct htp_argsort_context actx;actx.octx = octx;- actx.nrows_per_thread = (total_rows + n_threads - 1) / n_threads;+ actx.nrows_per_thread = fastdiv(nrows + n_threads - 1, &octx->n_threads_div);+ actx.total_rows = nrows;+ actx.row_start = row_start;+ actx.row_end = row_end;actx.vtcm_base = (uint8_t *) octx->ctx->vtcm_base;actx.vtcm_per_thread = spad_per_thread;@@ -508,7 +530,7 @@ int op_argsort(struct htp_ops_context * octx) {}// Run jobs- worker_pool_run_func(octx->ctx->worker_pool, job_func, &actx, n_threads);+ work_queue_run(octx->ctx->work_queue, job_func, &actx, n_threads);return HTP_STATUS_OK;}diff --git a/ggml/src/ggml-hexagon/htp/binary-ops.c b/ggml/src/ggml-hexagon/htp/binary-ops.cindex db61779635..bfa849e0ed 100644--- a/ggml/src/ggml-hexagon/htp/binary-ops.c+++ b/ggml/src/ggml-hexagon/htp/binary-ops.c@@ -13,9 +13,10 @@#define GGML_COMMON_DECL_C#include "ggml-common.h"+#include "hex-common.h"+#include "hex-profile.h"#include "htp-ctx.h"#include "htp-ops.h"-#include "htp-ops.h"#include "htp-tensor.h"#ifndef MIN@@ -36,6 +37,8 @@ struct htp_binary_context {uint32_t block_max;uint32_t nrows_per_thread;+ uint32_t total_rows;+ uint32_t row_start;size_t src0_row_size_aligned;size_t src1_row_size_aligned;size_t dst_row_size_aligned;@@ -48,27 +51,27 @@ struct htp_binary_context {const struct htp_tensor * src0 = octx->src[0]; \const struct htp_tensor * src1 = octx->src[1]; \const struct htp_tensor * dst = octx->dst; \- \- const uint32_t ne00 = src0->ne[0]; \- const uint32_t ne01 = src0->ne[1]; \- const uint32_t ne02 = src0->ne[2]; \- const uint32_t ne03 = src0->ne[3]; \- \- const uint32_t ne10 = src1->ne[0]; \- const uint32_t ne11 = src1->ne[1]; \- const uint32_t ne12 = src1->ne[2]; \- const uint32_t ne13 = src1->ne[3]; \- \- const uint32_t nb01 = src0->nb[1]; \- const uint32_t nb02 = src0->nb[2]; \- const uint32_t nb03 = src0->nb[3]; \- \- const uint32_t nb11 = src1->nb[1]; \- const uint32_t nb12 = src1->nb[2]; \- const uint32_t nb13 = src1->nb[3]; \- \- const uint32_t nb1 = dst->nb[1]; \- const uint32_t nb2 = dst->nb[2]; \+ \+ const uint32_t ne00 = src0->ne[0]; \+ const uint32_t ne01 = src0->ne[1]; \+ const uint32_t ne02 = src0->ne[2]; \+ const uint32_t ne03 = src0->ne[3]; \+ \+ const uint32_t ne10 = src1->ne[0]; \+ const uint32_t ne11 = src1->ne[1]; \+ const uint32_t ne12 = src1->ne[2]; \+ const uint32_t ne13 = src1->ne[3]; \+ \+ const uint32_t nb01 = src0->nb[1]; \+ const uint32_t nb02 = src0->nb[2]; \+ const uint32_t nb03 = src0->nb[3]; \+ \+ const uint32_t nb11 = src1->nb[1]; \+ const uint32_t nb12 = src1->nb[2]; \+ const uint32_t nb13 = src1->nb[3]; \+ \+ const uint32_t nb1 = dst->nb[1]; \+ const uint32_t nb2 = dst->nb[2]; \const uint32_t nb3 = dst->nb[3];static inline uint32_t calc_block_size(struct htp_binary_context * bctx, uint32_t ir, uint32_t end_row, uint32_t ne01, uint32_t ne02) {@@ -93,87 +96,87 @@ static inline uint32_t calc_block_size(struct htp_binary_context * bctx, uint32_}// Macro for scalar op switch-#define COMPUTE_SCALAR_OP(DST, SRC, VAL, TYPE, N) \- if(TYPE == HTP_TYPE_F32) { \- switch (octx->op) { \- case HTP_OP_ADD: hvx_add_scalar_f32_aa(DST, SRC, *(float *)VAL, N); break; \- case HTP_OP_SUB: hvx_sub_scalar_f32_aa(DST, SRC, *(float *)VAL, N); break; \- case HTP_OP_MUL: hvx_mul_scalar_f32_aa(DST, SRC, *(float *)VAL, N); break; \+#define COMPUTE_SCALAR_OP(DST, SRC, VAL, TYPE, N) \+ if(TYPE == HTP_TYPE_F32) { \+ switch (octx->op) { \+ case HTP_OP_ADD: hvx_add_scalar_f32_aa(DST, SRC, *(float *)VAL, N); break; \+ case HTP_OP_SUB: hvx_sub_scalar_f32_aa(DST, SRC, *(float *)VAL, N); break; \+ case HTP_OP_MUL: hvx_mul_scalar_f32_aa(DST, SRC, *(float *)VAL, N); break; \case HTP_OP_DIV: hvx_mul_scalar_f32_aa(DST, SRC, 1.0f / (*(float *)VAL), N); break; \- default: break; \- } \- } \- else { \- switch (octx->op) { \- case HTP_OP_ADD: hvx_add_scalar_f16_aa(DST, SRC, *(_Float16 *)VAL, N); break; \- case HTP_OP_SUB: hvx_sub_scalar_f16_aa(DST, SRC, *(_Float16 *)VAL, N); break; \- case HTP_OP_MUL: hvx_mul_scalar_f16_aa(DST, SRC, *(_Float16 *)VAL, N); break; \- case HTP_OP_DIV: hvx_div_scalar_f16_aa(DST, SRC, *(_Float16 *)VAL, N); break; \- default: break; \- } \+ default: break; \+ } \+ } \+ else { \+ switch (octx->op) { \+ case HTP_OP_ADD: hvx_add_scalar_f16_aa(DST, SRC, *(_Float16 *)VAL, N); break; \+ case HTP_OP_SUB: hvx_sub_scalar_f16_aa(DST, SRC, *(_Float16 *)VAL, N); break; \+ case HTP_OP_MUL: hvx_mul_scalar_f16_aa(DST, SRC, *(_Float16 *)VAL, N); break; \+ case HTP_OP_DIV: hvx_div_scalar_f16_aa(DST, SRC, *(_Float16 *)VAL, N); break; \+ default: break; \+ } \}// Macro for vector op switch (All Aligned)-#define COMPUTE_VECTOR_OP_AAA(DST, SRC0, SRC1, TYPE, N) \- if(TYPE == HTP_TYPE_F32) { \- switch (octx->op) { \+#define COMPUTE_VECTOR_OP_AAA(DST, SRC0, SRC1, TYPE, N) \+ if(TYPE == HTP_TYPE_F32) { \+ switch (octx->op) { \case HTP_OP_ADD: hvx_add_f32_aaa(DST, SRC0, SRC1, N); break; \case HTP_OP_SUB: hvx_sub_f32_aaa(DST, SRC0, SRC1, N); break; \case HTP_OP_MUL: hvx_mul_f32_aaa(DST, SRC0, SRC1, N); break; \case HTP_OP_DIV: hvx_div_f32_aaa(DST, SRC0, SRC1, N); break; \- default: break; \- } \- } \- else { \- switch (octx->op) { \+ default: break; \+ } \+ } \+ else { \+ switch (octx->op) { \case HTP_OP_ADD: hvx_add_f16_aaa(DST, SRC0, SRC1, N); break; \case HTP_OP_SUB: hvx_sub_f16_aaa(DST, SRC0, SRC1, N); break; \case HTP_OP_MUL: hvx_mul_f16_aaa(DST, SRC0, SRC1, N); break; \case HTP_OP_DIV: hvx_div_f16_aaa(DST, SRC0, SRC1, N); break; \- default: break; \- } \+ default: break; \+ } \}// Macro for vector op switch (Dst Aligned, Src0 Aligned, Src1 Unaligned)-#define COMPUTE_VECTOR_OP_AAU(DST, SRC0, SRC1, TYPE, N) \- if(TYPE == HTP_TYPE_F32) { \- switch (octx->op) { \+#define COMPUTE_VECTOR_OP_AAU(DST, SRC0, SRC1, TYPE, N) \+ if(TYPE == HTP_TYPE_F32) { \+ switch (octx->op) { \case HTP_OP_ADD: hvx_add_f32_aau(DST, SRC0, SRC1, N); break; \case HTP_OP_SUB: hvx_sub_f32_aau(DST, SRC0, SRC1, N); break; \case HTP_OP_MUL: hvx_mul_f32_aau(DST, SRC0, SRC1, N); break; \case HTP_OP_DIV: hvx_div_f32_aau(DST, SRC0, SRC1, N); break; \- default: break; \- } \- } \- else { \- switch (octx->op) { \+ default: break; \+ } \+ } \+ else { \+ switch (octx->op) { \case HTP_OP_ADD: hvx_add_f16_aau(DST, SRC0, SRC1, N); break; \case HTP_OP_SUB: hvx_sub_f16_aau(DST, SRC0, SRC1, N); break; \case HTP_OP_MUL: hvx_mul_f16_aau(DST, SRC0, SRC1, N); break; \case HTP_OP_DIV: hvx_div_f16_aau(DST, SRC0, SRC1, N); break; \- default: break; \- } \+ default: break; \+ } \}// Macro for vector op switch (All Unaligned - generic loop used in element repeat)-#define COMPUTE_VECTOR_OP_UUU(DST, SRC0, SRC1, TYPE, N) \- if(TYPE == HTP_TYPE_F32) { \- switch (octx->op) { \+#define COMPUTE_VECTOR_OP_UUU(DST, SRC0, SRC1, TYPE, N) \+ if(TYPE == HTP_TYPE_F32) { \+ switch (octx->op) { \case HTP_OP_ADD: hvx_add_f32_uuu(DST, SRC0, SRC1, N); break; \case HTP_OP_SUB: hvx_sub_f32_uuu(DST, SRC0, SRC1, N); break; \case HTP_OP_MUL: hvx_mul_f32_uuu(DST, SRC0, SRC1, N); break; \case HTP_OP_DIV: hvx_div_f32_uuu(DST, SRC0, SRC1, N); break; \- default: break; \- } \- } \- else { \- switch (octx->op) { \+ default: break; \+ } \+ } \+ else { \+ switch (octx->op) { \case HTP_OP_ADD: hvx_add_f16_uuu(DST, SRC0, SRC1, N); break; \case HTP_OP_SUB: hvx_sub_f16_uuu(DST, SRC0, SRC1, N); break; \case HTP_OP_MUL: hvx_mul_f16_uuu(DST, SRC0, SRC1, N); break; \case HTP_OP_DIV: hvx_div_f16_uuu(DST, SRC0, SRC1, N); break; \- default: break; \- } \+ default: break; \+ } \}// 1. Scalar src1 (ne10 == 1)@@ -184,9 +187,8 @@ static void binary_job_scalar(unsigned int nth, unsigned int ith, void * data) {const uint32_t src0_type = octx->src[0]->type;const uint32_t row_size_bytes = (src0_type == HTP_TYPE_F32) ? ne00 * sizeof(float) : ne00 * sizeof(_Float16);- const uint32_t total_rows = ne01 * ne02 * ne03;- const uint32_t start_row = bctx->nrows_per_thread * ith;- const uint32_t end_row = MIN(start_row + bctx->nrows_per_thread, total_rows);+ const uint32_t start_row = bctx->row_start + bctx->nrows_per_thread * ith;+ const uint32_t end_row = MIN(start_row + bctx->nrows_per_thread, bctx->row_start + bctx->total_rows);if (start_row >= end_row) return;FARF(HIGH, "binary-scalar: %d/%d (%u:%u) row-size %u (%u)", ith, nth, start_row, end_row, nb01, bctx->dst_row_size_aligned);@@ -222,6 +224,8 @@ static void binary_job_scalar(unsigned int nth, unsigned int ith, void * data) {}// Main loop+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+for (uint32_t ir = start_row; ir < end_row; ) {uint32_t current_block_size = calc_block_size(bctx, ir, end_row, ne01, ne02);@@ -242,12 +246,14 @@ static void binary_job_scalar(unsigned int nth, unsigned int ith, void * data) {uint8_t * src1_ptr = (uint8_t *)src1->data + i13 * nb13 + i12 * nb12 + i11 * nb11;uint32_t s1_stride = (ne11 == 1) ? 0 : nb11;+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);for (uint32_t r = 0; r < current_block_size; r++) {uint8_t * r_src0 = s0_spad + r * bctx->src0_row_size_aligned;uint8_t * r_dst = d_spad + r * bctx->dst_row_size_aligned;COMPUTE_SCALAR_OP(r_dst, r_src0, src1_ptr, src0_type, ne00);src1_ptr += s1_stride;}+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);uint8_t * dst_curr = (uint8_t *)dst->data + i03 * nb3 + i02 * nb2 + i01 * nb1;dma_queue_push(q, dma_make_ptr(dst_curr, d_spad), nb1, bctx->dst_row_size_aligned, row_size_bytes, current_block_size);@@ -266,6 +272,7 @@ static void binary_job_scalar(unsigned int nth, unsigned int ith, void * data) {}ir += current_block_size;}+dma_queue_flush(q);}@@ -277,9 +284,8 @@ static void binary_job_vector_same_shape(unsigned int nth, unsigned int ith, voiconst uint32_t src0_type = octx->src[0]->type;const uint32_t row_size_bytes = (src0_type == HTP_TYPE_F32) ? ne00 * sizeof(float) : ne00 * sizeof(_Float16);- const uint32_t total_rows = ne01 * ne02 * ne03;- const uint32_t start_row = bctx->nrows_per_thread * ith;- const uint32_t end_row = MIN(start_row + bctx->nrows_per_thread, total_rows);+ const uint32_t start_row = bctx->row_start + bctx->nrows_per_thread * ith;+ const uint32_t end_row = MIN(start_row + bctx->nrows_per_thread, bctx->row_start + bctx->total_rows);if (start_row >= end_row) return;FARF(HIGH, "binary-same-shape: %d/%d (%u:%u) row-size %u (%u)", ith, nth, start_row, end_row, nb01, bctx->dst_row_size_aligned);@@ -323,18 +329,22 @@ static void binary_job_vector_same_shape(unsigned int nth, unsigned int ith, voispad_idx ^= 1;}+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+for (uint32_t ir = start_row; ir < end_row; ) {uint32_t current_block_size = calc_block_size(bctx, ir, end_row, ne01, ne02);uint8_t * d_spad = (uint8_t *) dma_queue_pop(q).src;uint8_t * s0_spad = (uint8_t *) dma_queue_pop(q).dst;uint8_t * s1_spad = (uint8_t *) dma_queue_pop(q).dst;+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);for (uint32_t r = 0; r < current_block_size; r++) {uint8_t * r_src0 = s0_spad + r * bctx->src0_row_size_aligned;uint8_t * r_src1 = s1_spad + r * bctx->src1_row_size_aligned;uint8_t * r_dst = d_spad + r * bctx->dst_row_size_aligned;COMPUTE_VECTOR_OP_AAA(r_dst, r_src0, r_src1, src0_type, ne00);}+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);uint32_t i03, i02, i01, rem;i03 = fastdiv(ir, &bctx->src0_dim12_div);@@ -366,6 +376,7 @@ static void binary_job_vector_same_shape(unsigned int nth, unsigned int ith, voi}ir += current_block_size;}+dma_queue_flush(q);}@@ -377,9 +388,8 @@ static void binary_job_vector_row_broadcast(unsigned int nth, unsigned int ith,const uint32_t src0_type = octx->src[0]->type;const uint32_t row_size_bytes = (src0_type == HTP_TYPE_F32) ? ne00 * sizeof(float) : ne00 * sizeof(_Float16);- const uint32_t total_rows = ne01 * ne02 * ne03;- const uint32_t start_row = bctx->nrows_per_thread * ith;- const uint32_t end_row = MIN(start_row + bctx->nrows_per_thread, total_rows);+ const uint32_t start_row = bctx->row_start + bctx->nrows_per_thread * ith;+ const uint32_t end_row = MIN(start_row + bctx->nrows_per_thread, bctx->row_start + bctx->total_rows);if (start_row >= end_row) return;FARF(HIGH, "binary-row-bcast: %d/%d (%u:%u) row-size %u (%u)", ith, nth, start_row, end_row, nb01, bctx->dst_row_size_aligned);@@ -416,17 +426,21 @@ static void binary_job_vector_row_broadcast(unsigned int nth, unsigned int ith,spad_idx ^= 1;}+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+for (uint32_t ir = start_row; ir < end_row; ) {uint32_t current_block_size = calc_block_size(bctx, ir, end_row, ne01, ne02);uint8_t * d_spad = (uint8_t *) dma_queue_pop(q).src;uint8_t * s0_spad = (uint8_t *) dma_queue_pop(q).dst;+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);for (uint32_t r = 0; r < current_block_size; r++) {uint8_t * r_src0 = s0_spad + r * bctx->src0_row_size_aligned;uint8_t * r_src1 = (uint8_t *)s1_ptr; // Constantuint8_t * r_dst = d_spad + r * bctx->dst_row_size_aligned;COMPUTE_VECTOR_OP_AAA(r_dst, r_src0, r_src1, src0_type, ne00);}+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);uint32_t i03 = fastdiv(ir, &bctx->src0_dim12_div);uint32_t rem = ir - i03 * (ne02 * ne01);@@ -447,6 +461,7 @@ static void binary_job_vector_row_broadcast(unsigned int nth, unsigned int ith,}ir += current_block_size;}+dma_queue_flush(q);}@@ -458,9 +473,8 @@ static void binary_job_vector_complex(unsigned int nth, unsigned int ith, void *const uint32_t src0_type = octx->src[0]->type;const uint32_t row_size_bytes = (src0_type == HTP_TYPE_F32) ? ne00 * sizeof(float) : ne00 * sizeof(_Float16);- const uint32_t total_rows = ne01 * ne02 * ne03;- const uint32_t start_row = bctx->nrows_per_thread * ith;- const uint32_t end_row = MIN(start_row + bctx->nrows_per_thread, total_rows);+ const uint32_t start_row = bctx->row_start + bctx->nrows_per_thread * ith;+ const uint32_t end_row = MIN(start_row + bctx->nrows_per_thread, bctx->row_start + bctx->total_rows);if (start_row >= end_row) return;FARF(HIGH, "binary-complex: %d/%d (%u:%u) row-size %u (%u)", ith, nth, start_row, end_row, nb01, bctx->dst_row_size_aligned);@@ -493,6 +507,8 @@ static void binary_job_vector_complex(unsigned int nth, unsigned int ith, void *spad_idx ^= 1;}+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+for (uint32_t ir = start_row; ir < end_row; ) {uint32_t current_block_size = calc_block_size(bctx, ir, end_row, ne01, ne02);uint8_t * d_spad = (uint8_t *) dma_queue_pop(q).src;@@ -503,6 +519,7 @@ static void binary_job_vector_complex(unsigned int nth, unsigned int ith, void *uint32_t i02 = fastdiv(rem, &bctx->src0_dim1_div);uint32_t i01 = rem - i02 * ne01;+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);for (uint32_t r = 0; r < current_block_size; r++) {uint32_t r_i01 = i01 + r;uint32_t i13 = fastmodulo(i03, ne13, &bctx->src1_dim3_div);@@ -516,6 +533,7 @@ static void binary_job_vector_complex(unsigned int nth, unsigned int ith, void *// Read src1 from DDR (unaligned)COMPUTE_VECTOR_OP_AAU(r_dst, r_src0, r_src1, src0_type, ne00);}+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);uint8_t * dst_curr = (uint8_t *)dst->data + i03 * nb3 + i02 * nb2 + i01 * nb1;dma_queue_push(q, dma_make_ptr(dst_curr, d_spad), nb1, bctx->dst_row_size_aligned, row_size_bytes, current_block_size);@@ -532,6 +550,7 @@ static void binary_job_vector_complex(unsigned int nth, unsigned int ith, void *}ir += current_block_size;}+dma_queue_flush(q);}@@ -544,9 +563,8 @@ static void binary_job_element_repeat(unsigned int nth, unsigned int ith, void *const uint32_t src0_type = octx->src[0]->type;const uint32_t elem_size_bytes = (src0_type == HTP_TYPE_F32) ? sizeof(float) : sizeof(_Float16);const uint32_t row_size_bytes = ne00 * elem_size_bytes;;- const uint32_t total_rows = ne01 * ne02 * ne03;- const uint32_t start_row = bctx->nrows_per_thread * ith;- const uint32_t end_row = MIN(start_row + bctx->nrows_per_thread, total_rows);+ const uint32_t start_row = bctx->row_start + bctx->nrows_per_thread * ith;+ const uint32_t end_row = MIN(start_row + bctx->nrows_per_thread, bctx->row_start + bctx->total_rows);if (start_row >= end_row) return;uint8_t * src0_spad_base = octx->src0_spad.data + (ith * octx->src0_spad.size_per_thread);@@ -579,6 +597,8 @@ static void binary_job_element_repeat(unsigned int nth, unsigned int ith, void *spad_idx ^= 1;}+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+for (uint32_t ir = start_row; ir < end_row; ) {uint32_t current_block_size = calc_block_size(bctx, ir, end_row, ne01, ne02);uint8_t * d_spad = (uint8_t *) dma_queue_pop(q).src;@@ -589,6 +609,7 @@ static void binary_job_element_repeat(unsigned int nth, unsigned int ith, void *uint32_t i02 = fastdiv(rem, &bctx->src0_dim1_div);uint32_t i01 = rem - i02 * ne01;+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);for (uint32_t r = 0; r < current_block_size; r++) {uint32_t r_i01 = i01 + r;uint32_t i13 = fastmodulo(i03, ne13, &bctx->src1_dim3_div);@@ -606,6 +627,7 @@ static void binary_job_element_repeat(unsigned int nth, unsigned int ith, void *COMPUTE_VECTOR_OP_UUU(r_dst + c * elem_size_bytes, r_src0 + c * elem_size_bytes, r_src1_row, src0_type, len);}}+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);uint8_t * dst_curr = (uint8_t *)dst->data + i03 * nb3 + i02 * nb2 + i01 * nb1;dma_queue_push(q, dma_make_ptr(dst_curr, d_spad), nb1, bctx->dst_row_size_aligned, row_size_bytes, current_block_size);@@ -622,6 +644,7 @@ static void binary_job_element_repeat(unsigned int nth, unsigned int ith, void *}ir += current_block_size;}+dma_queue_flush(q);}@@ -650,9 +673,8 @@ static void binary_job_add_id(unsigned int nth, unsigned int ith, void * data) {const uint32_t nb2 = dst->nb[2];const uint32_t nb3 = dst->nb[3];- const uint32_t total_rows = ne01 * ne02 * ne03;- const uint32_t start_row = bctx->nrows_per_thread * ith;- const uint32_t end_row = MIN(start_row + bctx->nrows_per_thread, total_rows);+ const uint32_t start_row = bctx->row_start + bctx->nrows_per_thread * ith;+ const uint32_t end_row = MIN(start_row + bctx->nrows_per_thread, bctx->row_start + bctx->total_rows);if (start_row >= end_row) return;uint8_t * src0_spad_base = octx->src0_spad.data + (ith * octx->src0_spad.size_per_thread);@@ -683,6 +705,8 @@ static void binary_job_add_id(unsigned int nth, unsigned int ith, void * data) {spad_idx ^= 1;}+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+for (uint32_t ir = start_row; ir < end_row; ) {uint32_t current_block_size = calc_block_size(bctx, ir, end_row, ne01, ne02);uint8_t * d_spad = (uint8_t *) dma_queue_pop(q).src;@@ -693,6 +717,7 @@ static void binary_job_add_id(unsigned int nth, unsigned int ith, void * data) {uint32_t i02 = fastdiv(rem, &bctx->src0_dim1_div);uint32_t i01 = rem - i02 * ne01;+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);for (uint32_t r = 0; r < current_block_size; r++) {uint32_t r_i01 = i01 + r; // linear within block since we split at ne01@@ -704,6 +729,7 @@ static void binary_job_add_id(unsigned int nth, unsigned int ith, void * data) {hvx_add_f32_aau(r_dst, r_src0, r_src1, ne00);}+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);uint8_t * dst_curr = (uint8_t *)dst->data + i03 * nb3 + i02 * nb2 + i01 * nb1;dma_queue_push(q, dma_make_ptr(dst_curr, d_spad), nb1, bctx->dst_row_size_aligned, ne00 * sizeof(float), current_block_size);@@ -720,6 +746,7 @@ static void binary_job_add_id(unsigned int nth, unsigned int ith, void * data) {}ir += current_block_size;}+dma_queue_flush(q);}@@ -729,15 +756,31 @@ static int execute_op_binary(struct htp_ops_context * octx) {const struct htp_tensor * dst = octx->dst;const uint32_t src0_nrows = src0->ne[1] * src0->ne[2] * src0->ne[3];- const uint32_t n_threads = MIN(octx->n_threads, src0_nrows);- // Use packed row sizes for VTCM allocation+ // Use packed row sizes for VTCM allocation and alignmentconst uint32_t src0_type = octx->src[0]->type;const size_t elem_size = (src0_type == HTP_TYPE_F32) ? sizeof(float) : sizeof(_Float16);const size_t src0_row_size = src0->ne[0] * elem_size;const size_t src1_row_size = src1->ne[0] * elem_size;const size_t dst_row_size = dst->ne[0] * elem_size;+ uint32_t row_start = 0;+ uint32_t nrows = src0_nrows;++ if (octx->ctx->mdev.count > 1) {+ uint32_t rows_per_chunk = 0;+ htp_tensor_mdev_rows_per_chunk(dst, (uint32_t) elem_size, (uint32_t) dst_row_size, &rows_per_chunk);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(src0_nrows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ nrows = range.count;+ }++ if (nrows == 0) {+ return HTP_STATUS_OK;+ }++ const uint32_t n_threads = octx->n_threads;+size_t src0_row_size_aligned = hex_round_up(src0_row_size, VLEN);size_t src1_row_size_aligned = hex_round_up(src1_row_size, VLEN);size_t dst_row_size_aligned = hex_round_up(dst_row_size, VLEN);@@ -815,7 +858,9 @@ static int execute_op_binary(struct htp_ops_context * octx) {struct htp_binary_context bctx;bctx.octx = octx;- bctx.nrows_per_thread = (src0_nrows + n_threads - 1) / n_threads;+ bctx.nrows_per_thread = fastdiv(nrows + n_threads - 1, &octx->n_threads_div);+ bctx.total_rows = nrows;+ bctx.row_start = row_start;bctx.block_max = rows_per_buffer;bctx.src0_row_size_aligned = src0_row_size_aligned;bctx.src1_row_size_aligned = src1_row_size_aligned;@@ -850,7 +895,7 @@ static int execute_op_binary(struct htp_ops_context * octx) {dma_queue_pop(q);}- worker_pool_run_func(octx->ctx->worker_pool, worker_func, &bctx, n_threads);+ work_queue_run(octx->ctx->work_queue, worker_func, &bctx, n_threads);return HTP_STATUS_OK;}@@ -870,4 +915,3 @@ int op_binary(struct htp_ops_context * octx) {return HTP_STATUS_NO_SUPPORT;}-diff --git a/ggml/src/ggml-hexagon/htp/concat-ops.c b/ggml/src/ggml-hexagon/htp/concat-ops.cindex 51d39e8d98..966e867b39 100644--- a/ggml/src/ggml-hexagon/htp/concat-ops.c+++ b/ggml/src/ggml-hexagon/htp/concat-ops.c@@ -1,5 +1,8 @@+#include "hex-common.h"+#include "hex-profile.h"#include "htp-ctx.h"#include "htp-ops.h"+#include "htp-tensor.h"#include "hexagon_types.h"#include "hexagon_protos.h"#include "hvx_hexagon_protos.h"@@ -13,6 +16,10 @@ struct htp_concat_context {struct htp_ops_context * octx;uint32_t dim;uint32_t nrows_per_thread;+ uint32_t row_start;+ uint32_t nrows;+ uint32_t elem_start;+ uint32_t nelems;struct fastdiv_values div_ne0;struct fastdiv_values div_ne1;struct fastdiv_values div_ne2;@@ -28,10 +35,10 @@ static void concat_2d_f32_transposed(unsigned int nth, unsigned int ith, void *const uint32_t src0_ne0 = src0->ne[0];const uint32_t src1_ne0 = src1->ne[0];- const uint32_t ne1 = dst->ne[1];- const uint32_t start_i = ith * cctx->nrows_per_thread;- const uint32_t end_i = (start_i + cctx->nrows_per_thread < ne1) ? (start_i + cctx->nrows_per_thread) : ne1;+ const uint32_t row_end = cctx->row_start + cctx->nrows;+ const uint32_t start_i = cctx->row_start + ith * cctx->nrows_per_thread;+ const uint32_t end_i = (start_i + cctx->nrows_per_thread < row_end) ? (start_i + cctx->nrows_per_thread) : row_end;if (start_i >= end_i) return;dma_queue * q = octx->ctx->dma[ith];@@ -51,6 +58,8 @@ static void concat_2d_f32_transposed(unsigned int nth, unsigned int ith, void *const uint32_t spad0_row_bytes = hex_round_up((src0_ne0 + src1_ne0_padded) * sizeof(float), VLEN);uint32_t mu = src1_ne0_padded * spad1_stride;+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+for (uint32_t i = start_i; i < end_i; i += block_i) {uint32_t current_block_i = (end_i - i < block_i) ? (end_i - i) : block_i;@@ -66,6 +75,7 @@ static void concat_2d_f32_transposed(unsigned int nth, unsigned int ith, void *HVX_Vector * vtcm_tmp = (HVX_Vector *)(spad1_base + src1_ne0_padded * spad1_stride);+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) i);for (uint32_t j = 0; j < src1_ne0_padded; j += 32) {#pragma unroll(4)for (uint32_t ii = 0; ii < current_block_i; ii++) {@@ -75,6 +85,7 @@ static void concat_2d_f32_transposed(unsigned int nth, unsigned int ith, void *hvx_vmemu(dst_ptr) = vtcm_tmp[ii];}}+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) i);dma_queue_pop(q); // src0@@ -95,10 +106,10 @@ static void concat_2d_f16_transposed(unsigned int nth, unsigned int ith, void *const uint32_t src0_ne0 = src0->ne[0];const uint32_t src1_ne0 = src1->ne[0];- const uint32_t ne1 = dst->ne[1];- const uint32_t start_i = ith * cctx->nrows_per_thread;- const uint32_t end_i = (start_i + cctx->nrows_per_thread < ne1) ? (start_i + cctx->nrows_per_thread) : ne1;+ const uint32_t row_end = cctx->row_start + cctx->nrows;+ const uint32_t start_i = cctx->row_start + ith * cctx->nrows_per_thread;+ const uint32_t end_i = (start_i + cctx->nrows_per_thread < row_end) ? (start_i + cctx->nrows_per_thread) : row_end;if (start_i >= end_i) return;dma_queue * q = octx->ctx->dma[ith];@@ -118,6 +129,8 @@ static void concat_2d_f16_transposed(unsigned int nth, unsigned int ith, void *const uint32_t spad0_row_bytes = hex_round_up((src0_ne0 + src1_ne0_padded) * sizeof(__fp16), VLEN);uint32_t mu = src1_ne0_padded * spad1_stride;+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+for (uint32_t i = start_i; i < end_i; i += block_i) {uint32_t current_block_i = (end_i - i < block_i) ? (end_i - i) : block_i;@@ -133,6 +146,7 @@ static void concat_2d_f16_transposed(unsigned int nth, unsigned int ith, void *HVX_Vector * vtcm_tmp = (HVX_Vector *)(spad1_base + src1_ne0_padded * spad1_stride);+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) i);for (uint32_t j = 0; j < src1_ne0_padded; j += 64) {#pragma unroll(4)for (uint32_t ii = 0; ii < current_block_i; ii++) {@@ -142,6 +156,7 @@ static void concat_2d_f16_transposed(unsigned int nth, unsigned int ith, void *hvx_vmemu(dst_ptr) = vtcm_tmp[ii];}}+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) i);dma_queue_pop(q); // src0@@ -164,11 +179,14 @@ static void concat_generic(unsigned int nth, unsigned int ith, void * data) {const uint32_t type_size = (dst->type == HTP_TYPE_F32 || dst->type == HTP_TYPE_I32) ? 4 : 2;const uint32_t ne[4] = {dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3]};- const uint32_t total_elements = ne[0] * ne[1] * ne[2] * ne[3];- const uint32_t chunk_size = (total_elements + nth - 1) / nth;- const uint32_t start_idx = MIN(ith * chunk_size, total_elements);- const uint32_t end_idx = MIN(start_idx + chunk_size, total_elements);+ // Per-device element range aligned to prevent false sharing+ const uint32_t elem_start = cctx->elem_start;+ const uint32_t nelems = cctx->nelems;+ const uint32_t chunk_size = (nelems + nth - 1) / nth;++ const uint32_t start_idx = MIN(elem_start + ith * chunk_size, elem_start + nelems);+ const uint32_t end_idx = MIN(start_idx + chunk_size, elem_start + nelems);// Naive scalar element-wise copyfor (uint32_t idx = start_idx; idx < end_idx; idx++) {@@ -236,13 +254,28 @@ int op_concat(struct htp_ops_context * octx) {void (*worker_func)(unsigned int, unsigned int, void *) = concat_generic;if (dim == 0 && is_2d && is_src1_transposed && !is_src0_transposed) {- n_threads = MIN(dst->ne[1], n_threads);- if (n_threads < 1) {- n_threads = 1;+ const uint32_t total_rows = dst->ne[1];+ const size_t dst_data_row_size = dst->ne[0] * type_size;+ uint32_t row_start = 0;+ uint32_t nrows = total_rows;+ if (octx->ctx->mdev.count > 1) {+ uint32_t rows_per_chunk = 0;+ htp_tensor_mdev_rows_per_chunk(dst, type_size, (uint32_t) dst_data_row_size, &rows_per_chunk);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_rows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ nrows = range.count;}++ if (nrows == 0) {+ return HTP_STATUS_OK;+ }++ cctx.row_start = row_start;+ cctx.nrows = nrows;+uint32_t block_i = (type_size == 4) ? 32 : 64;- cctx.nrows_per_thread = hmx_ceil_div(dst->ne[1], n_threads);+ cctx.nrows_per_thread = fastdiv(nrows + n_threads - 1, &octx->n_threads_div);// Allocate VTCMuint32_t spad1_stride = block_i * type_size;@@ -270,8 +303,26 @@ int op_concat(struct htp_ops_context * octx) {} else {worker_func = concat_2d_f16_transposed;}+ } else {+ const uint32_t total_elements = dst->ne[0] * dst->ne[1] * dst->ne[2] * dst->ne[3];+ uint32_t elem_start = 0;+ uint32_t nelems = total_elements;+ if (octx->ctx->mdev.count > 1) {+ const uint32_t elems_per_chunk = HEX_L2_LINE_SIZE / type_size;+ const bool can_split = htp_tensor_mdev_data_aligned(dst) && htp_tensor_is_contiguous(dst, type_size) && !htp_tensor_is_permuted(dst);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_elements, can_split ? elems_per_chunk : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ elem_start = range.start;+ nelems = range.count;+ }++ if (nelems == 0) {+ return HTP_STATUS_OK;+ }++ cctx.elem_start = elem_start;+ cctx.nelems = nelems;}- worker_pool_run_func(octx->ctx->worker_pool, worker_func, &cctx, n_threads);+ work_queue_run(octx->ctx->work_queue, worker_func, &cctx, n_threads);return HTP_STATUS_OK;}diff --git a/ggml/src/ggml-hexagon/htp/cpy-ops.c b/ggml/src/ggml-hexagon/htp/cpy-ops.cindex b151b757f4..7f01a8c1e0 100644--- a/ggml/src/ggml-hexagon/htp/cpy-ops.c+++ b/ggml/src/ggml-hexagon/htp/cpy-ops.c@@ -16,6 +16,7 @@#include "htp-ops.h"#include "hvx-utils.h"#include "htp-tensor.h"+#include "htp-fence.h"struct htp_copy_context {struct htp_ops_context * octx;@@ -29,7 +30,23 @@ struct htp_copy_context {uint32_t src0_blocks_per_row;uint32_t dst_blocks_per_row;+ uint32_t elem_start;+ uint32_t nelem;+ uint32_t elem_per_thread;+uint32_t src0_nrows_per_thread;+ uint32_t row_start;+ uint32_t nrows;++ struct fastdiv_values div_ne01;+ struct fastdiv_values div_ne02_ne01;++ struct fastdiv_values div_ne0;+ struct fastdiv_values div_ne1_ne0;+ struct fastdiv_values div_ne2_ne1_ne0;+ struct fastdiv_values div_ne00;+ struct fastdiv_values div_ne01_ne00;+ struct fastdiv_values div_ne02_ne01_ne00;};#define cpy_preamble \@@ -54,131 +71,113 @@ struct htp_copy_context {const uint32_t nb0 = dst->nb[0]; \const uint32_t nb1 = dst->nb[1]; \const uint32_t nb2 = dst->nb[2]; \- const uint32_t nb3 = dst->nb[3]; \- \- const uint32_t nr = ne01;--#define DEFINE_CPY_SAMESHAPE(NAME, ELEM_TYPE, ELEM_SIZE) \-static void cpy_thread_##NAME##_sameshape(unsigned int nth, unsigned int ith, void * data) { \- struct htp_copy_context * ct = (struct htp_copy_context *) data; \- struct htp_ops_context * octx = ct->octx; \- cpy_preamble; \- const uint32_t dr = ct->src0_nrows_per_thread; \- const uint32_t ir0 = dr * ith; \- const uint32_t ir1 = (ir0 + dr) < nr ? (ir0 + dr) : nr; \- if (ir0 >= nr) return; \- for (uint32_t i03 = 0; i03 < ne03; i03++) { \- for (uint32_t i02 = 0; i02 < ne02; i02++) { \- _Pragma("unroll(4)") \- for (uint32_t i01 = ir0; i01 < ir1; i01++) { \- uint8_t* dst_ptr = (uint8_t*) dst->data + i01*nb1 + i02*nb2 + i03*nb3; \- uint8_t* src0_ptr = (uint8_t*) src0->data + i01*nb01 + i02*nb02 + i03*nb03; \- hex_l2fetch(src0_ptr, ne00 * ELEM_SIZE, nb01, 2); \- hvx_copy_uu(dst_ptr, src0_ptr, ne00, ELEM_SIZE); \- } \- } \- } \+ const uint32_t nb3 = dst->nb[3];++#define DEFINE_CPY_SAMESHAPE(NAME, ELEM_TYPE, ELEM_SIZE) \+static void cpy_thread_##NAME##_sameshape(unsigned int nth, unsigned int ith, void * data) { \+ struct htp_copy_context * ct = (struct htp_copy_context *) data; \+ struct htp_ops_context * octx = ct->octx; \+ cpy_preamble; \+ const uint32_t dr = ct->src0_nrows_per_thread; \+ const uint32_t ir0 = ct->row_start + dr * ith; \+ const uint32_t ir1 = MIN(ir0 + dr, ct->row_start + ct->nrows); \+ if (ir0 >= ir1) return; \+ const bool contiguous = (nb01 == ne00 * ELEM_SIZE) && (nb1 == nb01) && \+ (nb02 == ne01 * nb01) && (nb2 == nb02) && \+ (nb03 == ne02 * nb02) && (nb3 == nb03); \+ const uint32_t ne02_ne01 = ne02 * ne01; \+ uint32_t i03 = fastdiv(ir0, &ct->div_ne02_ne01); \+ uint32_t rem = ir0 - i03 * ne02_ne01; \+ uint32_t i02 = fastdiv(rem, &ct->div_ne01); \+ uint32_t i01 = rem - i02 * ne01; \+ uint8_t * dst_ptr = (uint8_t *) dst->data + i01*nb1 + i02*nb2 + i03*nb3; \+ uint8_t * src0_ptr = (uint8_t *) src0->data + i01*nb01 + i02*nb02 + i03*nb03; \+ if (contiguous) { \+ hvx_copy_uu(dst_ptr, src0_ptr, (ir1 - ir0) * ne00, ELEM_SIZE); \+ return; \+ } \+ for (uint32_t r = ir0; r < ir1; r++) { \+ hex_l2fetch(src0_ptr, ne00 * ELEM_SIZE, nb01, 2); \+ hvx_copy_uu(dst_ptr, src0_ptr, ne00, ELEM_SIZE); \+ dst_ptr += nb1; \+ src0_ptr += nb01; \+ if (++i01 == ne01) { \+ i01 = 0; \+ if (++i02 == ne02) { \+ i02 = 0; \+ i03++; \+ } \+ dst_ptr = (uint8_t *) dst->data + i02*nb2 + i03*nb3; \+ src0_ptr = (uint8_t *) src0->data + i02*nb02 + i03*nb03; \+ } \+ } \}DEFINE_CPY_SAMESHAPE(f32, float, 4)DEFINE_CPY_SAMESHAPE(f16, __fp16, 2)-#define DEFINE_CPY_RESHAPE(NAME, ELEM_TYPE, ELEM_SIZE) \-static void cpy_thread_##NAME##_reshape(unsigned int nth, unsigned int ith, void * data) { \- struct htp_copy_context * ct = (struct htp_copy_context *) data; \- struct htp_ops_context * octx = ct->octx; \- cpy_preamble; \- const uint32_t dr = ct->src0_nrows_per_thread; \- const uint32_t ir0 = dr * ith; \- const uint32_t ir1 = (ir0 + dr) < nr ? (ir0 + dr) : nr; \- if (ir0 >= nr) return; \- const bool src0_contig = (nb00 == ELEM_SIZE) && \- (nb01 == ne00 * nb00) && \- (nb02 == ne01 * nb01) && \- (nb03 == ne02 * nb02); \- const bool dst_contig = (nb0 == ELEM_SIZE) && \- (nb1 == ne0 * nb0) && \- (nb2 == ne1 * nb1) && \- (nb3 == ne2 * nb2); \- if (src0_contig && dst_contig) { \- for (int64_t i03 = 0; i03 < ne03; i03++) { \- for (int64_t i02 = 0; i02 < ne02; i02++) { \- uint8_t * src_ptr = (uint8_t *) src0->data + i03*nb03 + i02*nb02 + ir0*nb01; \- uint32_t flat = ((i03*ne02 + i02)*ne01 + ir0) * ne00; \- uint8_t * dst_ptr = (uint8_t *) dst->data + flat * ELEM_SIZE; \- hvx_copy_uu(dst_ptr, src_ptr, (ir1 - ir0) * ne00, ELEM_SIZE); \- } \- } \- return; \- } \- const bool reshape_flat_fast = (ne03 == 1 && ne2 == 1 && ne3 == 1) && \- (ne0 == ne00 * ne01) && (ne1 == ne02) && \- (nb00 == ELEM_SIZE) && (nb0 == ELEM_SIZE); \- if (reshape_flat_fast) { \- for (uint32_t i02 = 0; i02 < ne02; i02++) { \- for (uint32_t i01 = ir0; i01 < ir1; i01++) { \- uint8_t * src0_ptr = (uint8_t *) src0->data + i01 * nb01 + i02 * nb02; \- uint8_t * dst_ptr = (uint8_t *) dst->data + i01 * ne00 * ELEM_SIZE + i02 * nb1; \- hvx_copy_uu(dst_ptr, src0_ptr, ne00, ELEM_SIZE); \- } \- } \- return; \- } \- int64_t k10 = 0; \- int64_t i11 = 0; \- int64_t i12 = 0; \- int64_t i13 = 0; \- const int64_t nk00 = ct->src0_blocks_per_row; \- const int64_t nk0 = ct->dst_blocks_per_row; \- for (int64_t i03 = 0; i03 < ne03; i03++) { \- for (int64_t i02 = 0; i02 < ne02; i02++) { \- k10 += nk00 * ir0; \- while (k10 >= nk0) { \- k10 -= nk0; \- if (++i11 == ne1) { \- i11 = 0; \- if (++i12 == ne2) { \- i12 = 0; \- if (++i13 == ne3) { \- i13 = 0; \- } \- } \- } \- } \- for (int64_t i01 = ir0; i01 < ir1; i01++) { \- for (int64_t k00 = 0; k00 < nk00; k00++) { \- const char * src0_ptr = ((char *) src0->data + k00*nb00 + i01*nb01 + i02*nb02 + i03*nb03); \- char * dst_ptr = ((char *) dst->data + k10*nb0 + i11*nb1 + i12*nb2 + i13*nb3); \- memcpy(dst_ptr, src0_ptr, ELEM_SIZE); \- if (++k10 == nk0) { \- k10 = 0; \- if (++i11 == ne1) { \- i11 = 0; \- if (++i12 == ne2) { \- i12 = 0; \- if (++i13 == ne3) { \- i13 = 0; \- } \- } \- } \- } \- } \- } \- k10 += nk00 * (ne01 - ir1); \- while (k10 >= nk0) { \- k10 -= nk0; \- if (++i11 == ne1) { \- i11 = 0; \- if (++i12 == ne2) { \- i12 = 0; \- if (++i13 == ne3) { \- i13 = 0; \- } \- } \- } \- } \- } \- } \+#define DEFINE_CPY_RESHAPE(NAME, ELEM_TYPE, ELEM_SIZE) \+static void cpy_thread_##NAME##_reshape(unsigned int nth, unsigned int ith, void * data) { \+ struct htp_copy_context * ct = (struct htp_copy_context *) data; \+ struct htp_ops_context * octx = ct->octx; \+ cpy_preamble; \+ const uint32_t th_nelem = ct->elem_per_thread; \+ const uint32_t th_start = ct->elem_start + ith * th_nelem; \+ const uint32_t th_end = MIN(th_start + th_nelem, ct->elem_start + ct->nelem); \+ if (th_start >= th_end) return; \+ \+ const uint32_t ne01_ne00 = ne01 * ne00; \+ const uint32_t ne02_ne01_ne00 = ne02 * ne01_ne00; \+ const uint32_t ne1_ne0 = ne1 * ne0; \+ const uint32_t ne2_ne1_ne0 = ne2 * ne1_ne0; \+ \+ uint32_t e = th_start; \+ uint32_t i13 = fastdiv(e, &ct->div_ne2_ne1_ne0); \+ uint32_t rem = e - i13 * ne2_ne1_ne0; \+ uint32_t i12 = fastdiv(rem, &ct->div_ne1_ne0); \+ uint32_t rem2 = rem - i12 * ne1_ne0; \+ uint32_t i11 = fastdiv(rem2, &ct->div_ne0); \+ uint32_t i10 = rem2 - i11 * ne0; \+ \+ uint32_t i03 = fastdiv(e, &ct->div_ne02_ne01_ne00); \+ uint32_t rem_s = e - i03 * ne02_ne01_ne00; \+ uint32_t i02 = fastdiv(rem_s, &ct->div_ne01_ne00); \+ uint32_t rem2_s = rem_s - i02 * ne01_ne00; \+ uint32_t i01 = fastdiv(rem2_s, &ct->div_ne00); \+ uint32_t i00 = rem2_s - i01 * ne00; \+ \+ char * dst_ptr = (char *) dst->data + i10*nb0 + i11*nb1 + i12*nb2 + i13*nb3; \+ const char * src0_ptr = (const char *) src0->data + i00*nb00 + i01*nb01 + i02*nb02 + i03*nb03; \+ \+ for (; e < th_end; e++) { \+ *((ELEM_TYPE *) dst_ptr) = *((const ELEM_TYPE *) src0_ptr); \+ \+ dst_ptr += nb0; \+ if (++i10 == ne0) { \+ i10 = 0; \+ if (++i11 == ne1) { \+ i11 = 0; \+ if (++i12 == ne2) { \+ i12 = 0; \+ i13++; \+ } \+ } \+ dst_ptr = (char *) dst->data + i11*nb1 + i12*nb2 + i13*nb3; \+ } \+ \+ src0_ptr += nb00; \+ if (++i00 == ne00) { \+ i00 = 0; \+ if (++i01 == ne01) { \+ i01 = 0; \+ if (++i02 == ne02) { \+ i02 = 0; \+ i03++; \+ } \+ } \+ src0_ptr = (const char *) src0->data + i01*nb01 + i02*nb02 + i03*nb03; \+ } \+ } \}DEFINE_CPY_RESHAPE(f32, float, 4)@@ -189,22 +188,33 @@ static void cpy_thread_f16_f32_sameshape(unsigned int nth, unsigned int ith, voistruct htp_ops_context * octx = ct->octx;cpy_preamble;- // parallelize by src0 rowsconst uint32_t dr = ct->src0_nrows_per_thread;- const uint32_t ir0 = dr * ith;- const uint32_t ir1 = (ir0 + dr) < nr ? (ir0 + dr) : nr;- if (ir0 >= nr) return;-- // copy by rows- for (uint32_t i03 = 0; i03 < ne03; i03++) {- for (uint32_t i02 = 0; i02 < ne02; i02++) {- #pragma unroll(2)- for (uint32_t i01 = ir0; i01 < ir1; i01++) {- uint8_t* dst_ptr = (uint8_t*) dst->data + i01*nb1 + i02*nb2 + i03*nb3;- uint8_t* src0_ptr = (uint8_t*) src0->data + i01*nb01 + i02*nb02 + i03*nb03;- hex_l2fetch(src0_ptr, ne00 * sizeof(float), nb01, 2);- hvx_copy_f16_f32_uu(dst_ptr, src0_ptr, ne00);+ const uint32_t ir0 = ct->row_start + dr * ith;+ const uint32_t ir1 = MIN(ir0 + dr, ct->row_start + ct->nrows);+ if (ir0 >= ir1) return;++ const uint32_t ne02_ne01 = ne02 * ne01;+ uint32_t i03 = fastdiv(ir0, &ct->div_ne02_ne01);+ uint32_t rem = ir0 - i03 * ne02_ne01;+ uint32_t i02 = fastdiv(rem, &ct->div_ne01);+ uint32_t i01 = rem - i02 * ne01;++ uint8_t* dst_ptr = (uint8_t*) dst->data + i01*nb1 + i02*nb2 + i03*nb3;+ uint8_t* src0_ptr = (uint8_t*) src0->data + i01*nb01 + i02*nb02 + i03*nb03;++ for (uint32_t r = ir0; r < ir1; r++) {+ hex_l2fetch(src0_ptr, ne00 * sizeof(float), nb01, 2);+ hvx_copy_f16_f32_uu(dst_ptr, src0_ptr, ne00);+ dst_ptr += nb1;+ src0_ptr += nb01;+ if (++i01 == ne01) {+ i01 = 0;+ if (++i02 == ne02) {+ i02 = 0;+ i03++;}+ dst_ptr = (uint8_t*) dst->data + i02*nb2 + i03*nb3;+ src0_ptr = (uint8_t*) src0->data + i02*nb02 + i03*nb03;}}}@@ -214,22 +224,33 @@ static void cpy_thread_f32_f16_sameshape(unsigned int nth, unsigned int ith, voistruct htp_ops_context * octx = ct->octx;cpy_preamble;- // parallelize by src0 rowsconst uint32_t dr = ct->src0_nrows_per_thread;- const uint32_t ir0 = dr * ith;- const uint32_t ir1 = (ir0 + dr) < nr ? (ir0 + dr) : nr;- if (ir0 >= nr) return;-- // copy by rows- for (uint32_t i03 = 0; i03 < ne03; i03++) {- for (uint32_t i02 = 0; i02 < ne02; i02++) {- #pragma unroll(2)- for (uint32_t i01 = ir0; i01 < ir1; i01++) {- uint8_t* dst_ptr = (uint8_t*) dst->data + i01*nb1 + i02*nb2 + i03*nb3;- uint8_t* src0_ptr = (uint8_t*) src0->data + i01*nb01 + i02*nb02 + i03*nb03;- hex_l2fetch(src0_ptr, ne00 * sizeof(__fp16), nb01, 2);- hvx_copy_f32_f16_uu(dst_ptr, src0_ptr, ne00);+ const uint32_t ir0 = ct->row_start + dr * ith;+ const uint32_t ir1 = MIN(ir0 + dr, ct->row_start + ct->nrows);+ if (ir0 >= ir1) return;++ const uint32_t ne02_ne01 = ne02 * ne01;+ uint32_t i03 = fastdiv(ir0, &ct->div_ne02_ne01);+ uint32_t rem = ir0 - i03 * ne02_ne01;+ uint32_t i02 = fastdiv(rem, &ct->div_ne01);+ uint32_t i01 = rem - i02 * ne01;++ uint8_t* dst_ptr = (uint8_t*) dst->data + i01*nb1 + i02*nb2 + i03*nb3;+ uint8_t* src0_ptr = (uint8_t*) src0->data + i01*nb01 + i02*nb02 + i03*nb03;++ for (uint32_t r = ir0; r < ir1; r++) {+ hex_l2fetch(src0_ptr, ne00 * sizeof(__fp16), nb01, 2);+ hvx_copy_f32_f16_uu(dst_ptr, src0_ptr, ne00);+ dst_ptr += nb1;+ src0_ptr += nb01;+ if (++i01 == ne01) {+ i01 = 0;+ if (++i02 == ne02) {+ i02 = 0;+ i03++;}+ dst_ptr = (uint8_t*) dst->data + i02*nb2 + i03*nb3;+ src0_ptr = (uint8_t*) src0->data + i02*nb02 + i03*nb03;}}}@@ -250,15 +271,19 @@ static inline void cpy_dma_sametype_sameshape(dma_queue * q = octx->ctx->dma[0];if (contiguous_outer) {- dma_queue_push(q, dma_make_ptr((void *) dst->data, (const void *) src0->data), nb1, nb01, ne00 * elem_size, ne01 * ne02 * ne03);- dma_queue_pop(q);+ if (!dma_queue_push(q, dma_make_ptr((void *) dst->data, (const void *) src0->data), nb1, nb01, ne00 * elem_size, ne01 * ne02 * ne03)) {+ dma_queue_flush(q);+ dma_queue_push(q, dma_make_ptr((void *) dst->data, (const void *) src0->data), nb1, nb01, ne00 * elem_size, ne01 * ne02 * ne03);+ }+ dma_queue_flush(q);return;}for (uint32_t i03 = 0; i03 < ne03; i03++) {for (uint32_t i02 = 0; i02 < ne02; i02++) {- uint8_t* dst_ptr = (uint8_t*) dst->data + i02*nb2 + i03*nb3;- uint8_t* src0_ptr = (uint8_t*) src0->data + i02*nb02 + i03*nb03;+ uint8_t * dst_ptr = (uint8_t *) dst->data + i02 * nb2 + i03 * nb3;+ uint8_t * src0_ptr = (uint8_t *) src0->data + i02 * nb02 + i03 * nb03;+if (!dma_queue_push(q, dma_make_ptr(dst_ptr, src0_ptr), nb1, nb01, ne00 * elem_size, ne01)) {dma_queue_flush(q);dma_queue_push(q, dma_make_ptr(dst_ptr, src0_ptr), nb1, nb01, ne00 * elem_size, ne01);@@ -269,10 +294,9 @@ static inline void cpy_dma_sametype_sameshape(dma_queue_flush(q);}-int op_cpy(struct htp_ops_context * octx) {+static int exec_cpy(struct htp_ops_context * octx, bool * use_dma) {cpy_preamble;-- const uint32_t n_threads = MIN(nr, octx->n_threads);+ *use_dma = false;struct htp_copy_context ct;ct.octx = octx;@@ -296,59 +320,117 @@ int op_cpy(struct htp_ops_context * octx) {}const bool sametype = (src0->type == dst->type);- const bool transposed = (nb00 > nb01) || (nb0 > nb1);+ const bool transposed = (nb00 > nb01) || (nb0 > nb1) ||+ (nb00 != ct.src0_type_size) || (nb0 != ct.dst_type_size) ||+ (nb01 < ne00 * ct.src0_type_size) || (nb1 < ne0 * ct.dst_type_size);const bool sameshape = !transposed && (ne00 == ne0 && ne01 == ne1 && ne02 == ne2 && ne03 == ne3);- ct.src0_nrows_per_thread = (nr + n_threads - 1) / n_threads;+ const uint32_t n_threads = octx->n_threads;- worker_callback_t copy_fun = NULL;- bool use_dma = false;+ const bool dst_is_contiguous = htp_tensor_is_contiguous(dst, ct.dst_type_size);- if (sametype && sameshape) {- use_dma = true;- } else if (sameshape) {- /**/ if (dst->type == HTP_TYPE_F16 && src0->type == HTP_TYPE_F32)- copy_fun = cpy_thread_f16_f32_sameshape;- else if (dst->type == HTP_TYPE_F32 && src0->type == HTP_TYPE_F16)- copy_fun = cpy_thread_f32_f16_sameshape;- else- return HTP_STATUS_NO_SUPPORT;- } else if (sametype) {- if (src0->type == HTP_TYPE_F32) {- copy_fun = cpy_thread_f32_reshape;+ if (sameshape) {+ const uint32_t total_rows = ne01 * ne02 * ne03;+ const uint32_t row_size = ne00 * ct.dst_type_size;++ ct.div_ne01 = init_fastdiv_values(ne01);+ ct.div_ne02_ne01 = init_fastdiv_values(ne02 * ne01);++ uint32_t row_start = 0;+ uint32_t nrows = total_rows;++ if (octx->ctx->mdev.count > 1) {+ const uint32_t rows_per_chunk = (row_size > 0) ? (HEX_L2_LINE_SIZE / hex_gcd_u32(row_size, HEX_L2_LINE_SIZE)) : 1;+ const bool can_split = htp_tensor_mdev_data_aligned(dst) && dst_is_contiguous;+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_rows, can_split ? rows_per_chunk : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ nrows = range.count;+ }++ if (nrows == 0) {+ return HTP_STATUS_OK;+ }++ ct.row_start = row_start;+ ct.nrows = nrows;+ ct.src0_nrows_per_thread = fastdiv(nrows + n_threads - 1, &octx->n_threads_div);++ if (sametype && octx->ctx->mdev.count <= 1) {+ *use_dma = true;+ cpy_dma_sametype_sameshape(octx, dst, src0, ct.src0_type_size, ne00, ne01, ne02, ne03, nb01, nb02, nb03, nb1, nb2, nb3);} else {- copy_fun = cpy_thread_f16_reshape;+ work_queue_func_t copy_fun = NULL;+ if (sametype) {+ copy_fun = (src0->type == HTP_TYPE_F32) ? cpy_thread_f32_sameshape : cpy_thread_f16_sameshape;+ } else if (dst->type == HTP_TYPE_F16 && src0->type == HTP_TYPE_F32) {+ copy_fun = cpy_thread_f16_f32_sameshape;+ } else if (dst->type == HTP_TYPE_F32 && src0->type == HTP_TYPE_F16) {+ copy_fun = cpy_thread_f32_f16_sameshape;+ } else {+ return HTP_STATUS_NO_SUPPORT;+ }+ work_queue_run(octx->ctx->work_queue, copy_fun, &ct, n_threads);+ }+ } else if (sametype) {+ const uint32_t total_elems = ne0 * ne1 * ne2 * ne3;+ const uint32_t elems_per_line = (ct.dst_type_size == 4) ? 32 : 64;++ ct.div_ne0 = init_fastdiv_values(ne0);+ ct.div_ne1_ne0 = init_fastdiv_values(ne1 * ne0);+ ct.div_ne2_ne1_ne0 = init_fastdiv_values(ne2 * ne1 * ne0);+ ct.div_ne00 = init_fastdiv_values(ne00);+ ct.div_ne01_ne00 = init_fastdiv_values(ne01 * ne00);+ ct.div_ne02_ne01_ne00 = init_fastdiv_values(ne02 * ne01 * ne00);++ uint32_t elem_start = 0;+ uint32_t nelem = total_elems;++ if (octx->ctx->mdev.count > 1) {+ const bool can_split = htp_tensor_mdev_data_aligned(dst) && dst_is_contiguous;+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_elems, can_split ? elems_per_line : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ elem_start = range.start;+ nelem = range.count;+ }++ if (nelem == 0) {+ return HTP_STATUS_OK;}++ ct.elem_start = elem_start;+ ct.nelem = nelem;+ ct.elem_per_thread = fastdiv(nelem + n_threads - 1, &octx->n_threads_div);++ work_queue_func_t copy_fun = (src0->type == HTP_TYPE_F32) ? cpy_thread_f32_reshape : cpy_thread_f16_reshape;+ work_queue_run(octx->ctx->work_queue, copy_fun, &ct, n_threads);} else {return HTP_STATUS_NO_SUPPORT;}- FARF(HIGH, "cpy-%s-%s: (%ux%ux%ux%u) -> (%ux%ux%ux%u) : use_dma=%d n_threads %u\n",- src0->type == HTP_TYPE_F32 ? "f32" : "f16", dst->type == HTP_TYPE_F32 ? "f32" : "f16",- ne00, ne01, ne02, ne03, ne0, ne1, ne2, ne3, use_dma, n_threads);+ return HTP_STATUS_OK;+}- if (use_dma) {- cpy_dma_sametype_sameshape(octx, dst, src0, ct.src0_type_size, ne00, ne01, ne02, ne03, nb01, nb02, nb03, nb1, nb2, nb3);- } else {- worker_pool_run_func(octx->ctx->worker_pool, copy_fun, &ct, n_threads);- }+int op_cpy(struct htp_ops_context * octx) {+ bool use_dma = false;+ int status = exec_cpy(octx, &use_dma);++ htp_ops_context_set_status(octx, status);- const struct htp_tensor *sync = octx->src[1];- if (sync && (sync->flags & HTP_TENSOR_FENCE)) {+ if (octx->op == HTP_OP_CPY_FENCE) {if (!use_dma) {- // htp_tensor_flush_all(octx->ctx, octx->dsts, 1);- qurt_mem_cache_clean((qurt_addr_t) 0, 0, QURT_MEM_CACHE_FLUSH_INVALIDATE_ALL, QURT_MEM_DCACHE);+ htp_flush_dirty_ranges(octx->ctx);}- atomic_uint * sync_fence = (atomic_uint *) sync->data;- const uint32_t seq = (uint32_t) octx->op_params[0];+ htp_mdev_group_barrier(octx);- atomic_store(&sync_fence[0], seq);- asm volatile ("syncht" : : : "memory");- Q6_dccleaninva_A((void *) sync_fence);+ if (octx->ctx->mdev.idx == 0) {+ const struct htp_tensor * sync = octx->src[1];+ const uint32_t seq = (uint32_t) octx->op_params[0];+ atomic_uint * sync_fence = (atomic_uint *) (uintptr_t) sync->data;+ htp_fence_write(sync_fence, seq, octx->status);- FARF(HIGH, "ggml-hex: sync-release : fence %p seq %u\n", sync_fence, seq);+ FARF(HIGH, "ggml-hex: sync-release : fence %p seq 0x%x status %d\n", sync_fence, seq, octx->status);+ }}- return HTP_STATUS_OK;+ return octx->status;}diff --git a/ggml/src/ggml-hexagon/htp/cumsum-ops.c b/ggml/src/ggml-hexagon/htp/cumsum-ops.cindex 2d45c39f23..971fa3bccb 100644--- a/ggml/src/ggml-hexagon/htp/cumsum-ops.c+++ b/ggml/src/ggml-hexagon/htp/cumsum-ops.c@@ -7,6 +7,8 @@#define GGML_COMMON_DECL_C#include "ggml-common.h"+#include "hex-common.h"+#include "hex-profile.h"#include "htp-ctx.h"#include "htp-ops.h"#include "htp-tensor.h"@@ -17,25 +19,25 @@#define htp_cumsum_tensors_preamble \const struct htp_tensor * restrict src0 = octx->src[0]; \const struct htp_tensor * restrict dst = octx->dst; \- \- const uint32_t ne00 = src0->ne[0]; \- const uint32_t ne01 = src0->ne[1]; \- const uint32_t ne02 = src0->ne[2]; \- const uint32_t ne03 = src0->ne[3]; \- \- const uint32_t ne0 = dst->ne[0]; \- const uint32_t ne1 = dst->ne[1]; \- const uint32_t ne2 = dst->ne[2]; \- const uint32_t ne3 = dst->ne[3]; \- \- const uint32_t nb00 = src0->nb[0]; \- const uint32_t nb01 = src0->nb[1]; \- const uint32_t nb02 = src0->nb[2]; \- const uint32_t nb03 = src0->nb[3]; \- \- const uint32_t nb0 = dst->nb[0]; \- const uint32_t nb1 = dst->nb[1]; \- const uint32_t nb2 = dst->nb[2]; \+ \+ const uint32_t ne00 = src0->ne[0]; \+ const uint32_t ne01 = src0->ne[1]; \+ const uint32_t ne02 = src0->ne[2]; \+ const uint32_t ne03 = src0->ne[3]; \+ \+ const uint32_t ne0 = dst->ne[0]; \+ const uint32_t ne1 = dst->ne[1]; \+ const uint32_t ne2 = dst->ne[2]; \+ const uint32_t ne3 = dst->ne[3]; \+ \+ const uint32_t nb00 = src0->nb[0]; \+ const uint32_t nb01 = src0->nb[1]; \+ const uint32_t nb02 = src0->nb[2]; \+ const uint32_t nb03 = src0->nb[3]; \+ \+ const uint32_t nb0 = dst->nb[0]; \+ const uint32_t nb1 = dst->nb[1]; \+ const uint32_t nb2 = dst->nb[2]; \const uint32_t nb3 = dst->nb[3];struct htp_cumsum_context {@@ -46,6 +48,7 @@ struct htp_cumsum_context {size_t dst_row_size_aligned;uint32_t rows_per_thread;uint32_t total_rows;+ uint32_t row_start;};#define htp_cumsum_preamble \@@ -116,11 +119,8 @@ static inline void hvx_cumsum_row_f32(const float * restrict src, float * restristatic void cumsum_thread_f32_dma(unsigned int nth, unsigned int ith, void * data) {htp_cumsum_preamble;- uint64_t t1, t2;- t1 = HAP_perf_get_qtimer_count();-- const uint32_t ir0 = cctx->rows_per_thread * ith;- const uint32_t ir1 = MIN(ir0 + cctx->rows_per_thread, cctx->total_rows);+ const uint32_t ir0 = cctx->row_start + cctx->rows_per_thread * ith;+ const uint32_t ir1 = MIN(ir0 + cctx->rows_per_thread, cctx->row_start + cctx->total_rows);if (ir0 >= ir1) {return;@@ -149,11 +149,15 @@ static void cumsum_thread_f32_dma(unsigned int nth, unsigned int ith, void * datsrc_row_size_aligned, src_row_size, 1);}+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+for (uint32_t ir = ir0; ir < ir1; ir++) {float * dst_spad_row = (float *) dma_queue_pop(dma_queue).src;float * src_spad_row = (float *) dma_queue_pop(dma_queue).dst;+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);hvx_cumsum_row_f32(src_spad_row, dst_spad_row, ne00);+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);dma_queue_push_vtcm_to_ddr(dma_queue,dma_make_ptr(dst_data + (ir * dst_row_size), (uint8_t *) dst_spad_row),@@ -168,12 +172,10 @@ static void cumsum_thread_f32_dma(unsigned int nth, unsigned int ith, void * dat}dma_queue_flush(dma_queue);- t2 = HAP_perf_get_qtimer_count();- FARF(HIGH, "cumsum-f32-dma %d/%d: %ux%ux%ux%u (%u:%u) -> %ux%ux%ux%u usec %u\n",+ FARF(HIGH, "cumsum-f32-dma %d/%d: %ux%ux%ux%u (%u:%u) -> %ux%ux%ux%u\n",ith, nth, src0->ne[0], src0->ne[1], src0->ne[2], src0->ne[3], ir0, ir1,- dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3],- (unsigned) HAP_perf_qtimer_count_to_us(t2 - t1));+ dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3]);}// ---------------------------------------------------------------------------@@ -183,14 +185,14 @@ static void cumsum_thread_f32_dma(unsigned int nth, unsigned int ith, void * datstatic void cumsum_thread_f32(unsigned int nth, unsigned int ith, void * data) {htp_cumsum_preamble;- uint64_t t1, t2;- t1 = HAP_perf_get_qtimer_count();-const uint8_t * src_data = (const uint8_t *) src0->data;uint8_t * dst_data = (uint8_t *) dst->data;- const uint32_t ir0 = cctx->rows_per_thread * ith;- const uint32_t ir1 = MIN(ir0 + cctx->rows_per_thread, cctx->total_rows);+ const uint32_t ir0 = cctx->row_start + cctx->rows_per_thread * ith;+ const uint32_t ir1 = MIN(ir0 + cctx->rows_per_thread, cctx->row_start + cctx->total_rows);++ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir0);for (uint32_t ir = ir0; ir < ir1; ir++) {const float * restrict src_row = (const float *) (src_data + ir * cctx->src_row_size);@@ -198,12 +200,11 @@ static void cumsum_thread_f32(unsigned int nth, unsigned int ith, void * data) {hvx_cumsum_row_f32(src_row, dst_row, ne00);}- t2 = HAP_perf_get_qtimer_count();+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir0);- FARF(HIGH, "cumsum-f32 %d/%d: %ux%ux%ux%u (%u:%u) -> %ux%ux%ux%u usec %u\n",+ FARF(HIGH, "cumsum-f32 %d/%d: %ux%ux%ux%u (%u:%u) -> %ux%ux%ux%u\n",ith, nth, src0->ne[0], src0->ne[1], src0->ne[2], src0->ne[3], ir0, ir1,- dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3],- (unsigned) HAP_perf_qtimer_count_to_us(t2 - t1));+ dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3]);}int op_cumsum_f32(struct htp_ops_context * octx) {@@ -214,8 +215,25 @@ int op_cumsum_f32(struct htp_ops_context * octx) {return HTP_STATUS_OK;}- const uint32_t total_rows = src0->ne[1] * src0->ne[2] * src0->ne[3];- const uint32_t n_threads = MIN(octx->n_threads, total_rows);+ const uint32_t total_rows = src0->ne[1] * src0->ne[2] * src0->ne[3];+ const size_t dst_data_row_size = dst->ne[0] * sizeof(float);++ uint32_t row_start = 0;+ uint32_t nrows = total_rows;++ if (octx->ctx->mdev.count > 1) {+ uint32_t rows_per_chunk = 0;+ htp_tensor_mdev_rows_per_chunk(dst, sizeof(float), (uint32_t) dst_data_row_size, &rows_per_chunk);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_rows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ nrows = range.count;+ }++ if (nrows == 0) {+ return HTP_STATUS_OK;+ }++ const uint32_t n_threads = octx->n_threads;const size_t src_row_size = src0->nb[1];const size_t dst_row_size = dst->nb[1];@@ -240,14 +258,15 @@ int op_cumsum_f32(struct htp_ops_context * octx) {.dst_row_size = dst_row_size,.src_row_size_aligned = src_row_size_aligned,.dst_row_size_aligned = dst_row_size_aligned,- .rows_per_thread = (total_rows + n_threads - 1) / n_threads,- .total_rows = total_rows,+ .rows_per_thread = fastdiv(nrows + n_threads - 1, &octx->n_threads_div),+ .total_rows = nrows,+ .row_start = row_start,};if (octx->ctx->vtcm_size < spad_per_thread * n_threads) {- worker_pool_run_func(octx->ctx->worker_pool, cumsum_thread_f32, &cctx, n_threads);+ work_queue_run(octx->ctx->work_queue, cumsum_thread_f32, &cctx, n_threads);} else {- worker_pool_run_func(octx->ctx->worker_pool, cumsum_thread_f32_dma, &cctx, n_threads);+ work_queue_run(octx->ctx->work_queue, cumsum_thread_f32_dma, &cctx, n_threads);}return HTP_STATUS_OK;diff --git a/ggml/src/ggml-hexagon/htp/diag-ops.c b/ggml/src/ggml-hexagon/htp/diag-ops.cindex 9b3194d908..a69fd89d38 100644--- a/ggml/src/ggml-hexagon/htp/diag-ops.c+++ b/ggml/src/ggml-hexagon/htp/diag-ops.c@@ -5,8 +5,11 @@#define GGML_COMMON_DECL_C#include "ggml-common.h"+#include "hex-common.h"+#include "hex-profile.h"#include "htp-ctx.h"#include "htp-ops.h"+#include "htp-tensor.h"#include "hvx-types.h"#include "hex-utils.h"#include "hvx-copy.h"@@ -15,17 +18,17 @@#define htp_diag_tensors_preamble \const struct htp_tensor * restrict src0 = octx->src[0]; \const struct htp_tensor * restrict dst = octx->dst; \- \- const uint32_t ne02 = src0->ne[2]; \- \- const uint32_t ne0 = dst->ne[0]; \- const uint32_t ne1 = dst->ne[1]; \- \- const uint32_t nb02 = src0->nb[2]; \- const uint32_t nb03 = src0->nb[3]; \- \- const uint32_t nb1 = dst->nb[1]; \- const uint32_t nb2 = dst->nb[2]; \+ \+ const uint32_t ne02 = src0->ne[2]; \+ \+ const uint32_t ne0 = dst->ne[0]; \+ const uint32_t ne1 = dst->ne[1]; \+ \+ const uint32_t nb02 = src0->nb[2]; \+ const uint32_t nb03 = src0->nb[3]; \+ \+ const uint32_t nb1 = dst->nb[1]; \+ const uint32_t nb2 = dst->nb[2]; \const uint32_t nb3 = dst->nb[3];struct htp_diag_context {@@ -36,6 +39,7 @@ struct htp_diag_context {size_t dst_row_size_aligned;uint32_t batches_per_thread;uint32_t total_batches;+ uint32_t batch_start;};#define htp_diag_preamble \@@ -57,11 +61,8 @@ static void diag_thread_f32_dma(unsigned int nth, unsigned int ith, void * data)htp_diag_preamble;dma_queue * dma_queue = octx->ctx->dma[ith];- uint64_t t1, t2;- t1 = HAP_perf_get_qtimer_count();-- const uint32_t ib0 = dctx->batches_per_thread * ith;- const uint32_t ib1 = MIN(ib0 + dctx->batches_per_thread, dctx->total_batches);+ const uint32_t ib0 = dctx->batch_start + dctx->batches_per_thread * ith;+ const uint32_t ib1 = MIN(ib0 + dctx->batches_per_thread, dctx->batch_start + dctx->total_batches);if (ib0 >= ib1) {return;@@ -79,6 +80,8 @@ static void diag_thread_f32_dma(unsigned int nth, unsigned int ith, void * data)uint8_t * src_spad = octx->src0_spad.data + (ith * src_batch_size_aligned);uint8_t * dst_spad = octx->dst_spad.data + (ith * dst_row_size_aligned);+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+for (uint32_t ib = ib0; ib < ib1; ib++) {const uint32_t i3 = ib / ne02;const uint32_t i2 = ib % ne02;@@ -96,7 +99,9 @@ static void diag_thread_f32_dma(unsigned int nth, unsigned int ith, void * data)for (uint32_t i1 = 0; i1 < ne1; i1++) {// Compute row in VTCM+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) (ib * ne1 + i1));hvx_diag_row_f32(src_spad_f32, dst_spad_f32, i1, ne0);+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) (ib * ne1 + i1));// Write completed row back to DDRuint8_t * dst_row = dst_data + i3 * nb3 + i2 * nb2 + i1 * nb1;@@ -107,12 +112,9 @@ static void diag_thread_f32_dma(unsigned int nth, unsigned int ith, void * data)}}- t2 = HAP_perf_get_qtimer_count();-- FARF(HIGH, "diag-f32-dma %d/%d: %ux%ux%ux%u (%u:%u) -> %ux%ux%ux%u usec %u\n",+ FARF(HIGH, "diag-f32-dma %d/%d: %ux%ux%ux%u (%u:%u) -> %ux%ux%ux%u\n",ith, nth, src0->ne[0], src0->ne[1], src0->ne[2], src0->ne[3], ib0, ib1,- dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3],- (unsigned) HAP_perf_qtimer_count_to_us(t2 - t1));+ dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3]);}// ---------------------------------------------------------------------------@@ -122,14 +124,14 @@ static void diag_thread_f32_dma(unsigned int nth, unsigned int ith, void * data)static void diag_thread_f32(unsigned int nth, unsigned int ith, void * data) {htp_diag_preamble;- uint64_t t1, t2;- t1 = HAP_perf_get_qtimer_count();-const uint8_t * src_data = (const uint8_t *) src0->data;uint8_t * dst_data = (uint8_t *) dst->data;- const uint32_t ib0 = dctx->batches_per_thread * ith;- const uint32_t ib1 = MIN(ib0 + dctx->batches_per_thread, dctx->total_batches);+ const uint32_t ib0 = dctx->batch_start + dctx->batches_per_thread * ith;+ const uint32_t ib1 = MIN(ib0 + dctx->batches_per_thread, dctx->batch_start + dctx->total_batches);++ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ib0);for (uint32_t ib = ib0; ib < ib1; ib++) {const uint32_t i3 = ib / ne02;@@ -143,12 +145,11 @@ static void diag_thread_f32(unsigned int nth, unsigned int ith, void * data) {}}- t2 = HAP_perf_get_qtimer_count();+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ib0);- FARF(HIGH, "diag-f32 %d/%d: %ux%ux%ux%u (%u:%u) -> %ux%ux%ux%u usec %u\n",+ FARF(HIGH, "diag-f32 %d/%d: %ux%ux%ux%u (%u:%u) -> %ux%ux%ux%u\n",ith, nth, src0->ne[0], src0->ne[1], src0->ne[2], src0->ne[3], ib0, ib1,- dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3],- (unsigned) HAP_perf_qtimer_count_to_us(t2 - t1));+ dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3]);}int op_diag_f32(struct htp_ops_context * octx) {@@ -160,7 +161,36 @@ int op_diag_f32(struct htp_ops_context * octx) {}const uint32_t total_batches = src0->ne[2] * src0->ne[3];- const uint32_t n_threads = MIN(octx->n_threads, total_batches);+ const size_t dst_batch_size = dst->ne[1] * dst->nb[1];++ uint32_t batch_start = 0;+ uint32_t nbatches = total_batches;++ if (octx->ctx->mdev.count > 1) {+ bool can_split = htp_tensor_mdev_data_aligned(dst) && (dst->ne[0] == 1 || dst->nb[0] == sizeof(float)) && !htp_tensor_is_permuted(dst);+ uint32_t batches_per_chunk = 1;+ if (can_split) {+ if (dst->ne[2] > 1 && (dst->nb[2] & (HTP_TENSOR_MDEV_LINE_SIZE - 1)) == 0 &&+ (dst->ne[3] <= 1 || (dst->nb[3] & (HTP_TENSOR_MDEV_LINE_SIZE - 1)) == 0)) {+ batches_per_chunk = 1;+ } else if (dst->nb[2] == dst_batch_size &&+ (dst->ne[3] <= 1 || dst->nb[3] == dst->nb[2] * dst->ne[2])) {+ batches_per_chunk = (dst_batch_size > 0) ? (HEX_L2_LINE_SIZE / hex_gcd_u32(dst_batch_size, HEX_L2_LINE_SIZE)) : 1;+ } else {+ can_split = false;+ }+ }++ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_batches, can_split ? batches_per_chunk : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ batch_start = range.start;+ nbatches = range.count;+ }++ if (nbatches == 0) {+ return HTP_STATUS_OK;+ }++ const uint32_t n_threads = octx->n_threads;const size_t src_batch_size = src0->ne[0] * sizeof(float);const size_t dst_row_size = dst->ne[0] * sizeof(float);@@ -185,14 +215,15 @@ int op_diag_f32(struct htp_ops_context * octx) {.dst_row_size = dst_row_size,.src_batch_size_aligned = src_batch_size_aligned,.dst_row_size_aligned = dst_row_size_aligned,- .batches_per_thread = (total_batches + n_threads - 1) / n_threads,- .total_batches = total_batches,+ .batches_per_thread = fastdiv(nbatches + n_threads - 1, &octx->n_threads_div),+ .total_batches = nbatches,+ .batch_start = batch_start,};if (octx->ctx->vtcm_size < spad_per_thread * n_threads) {- worker_pool_run_func(octx->ctx->worker_pool, diag_thread_f32, &dctx, n_threads);+ work_queue_run(octx->ctx->work_queue, diag_thread_f32, &dctx, n_threads);} else {- worker_pool_run_func(octx->ctx->worker_pool, diag_thread_f32_dma, &dctx, n_threads);+ work_queue_run(octx->ctx->work_queue, diag_thread_f32_dma, &dctx, n_threads);}return HTP_STATUS_OK;diff --git a/ggml/src/ggml-hexagon/htp/fill-ops.c b/ggml/src/ggml-hexagon/htp/fill-ops.cindex 3ccfbe74ee..1f6eaafada 100644--- a/ggml/src/ggml-hexagon/htp/fill-ops.c+++ b/ggml/src/ggml-hexagon/htp/fill-ops.c@@ -3,10 +3,11 @@#pragma clang diagnostic ignored "-Wunused-but-set-variable"#include <HAP_farf.h>-#include <HAP_perf.h>-#include <string.h>+#include "hex-common.h"+#include "hex-profile.h"+#include "hvx-copy.h"#include "hvx-utils.h"@@ -14,28 +15,30 @@#include "ggml-common.h"#include "htp-ctx.h"#include "htp-ops.h"+#include "htp-tensor.h"// ggml op_params layout for FILL:// op_params[0] (as float) - the scalar fill value-#define fill_preamble \+#define fill_preamble \const struct htp_tensor * dst = octx->dst; \- \- const uint32_t ne0 = dst->ne[0]; \- const uint32_t ne1 = dst->ne[1]; \- const uint32_t ne2 = dst->ne[2]; \- const uint32_t ne3 = dst->ne[3]; \- \- const uint32_t nb1 = dst->nb[1]; \- const uint32_t nb2 = dst->nb[2]; \- const uint32_t nb3 = dst->nb[3]; \- \+ \+ const uint32_t ne0 = dst->ne[0]; \+ const uint32_t ne1 = dst->ne[1]; \+ const uint32_t ne2 = dst->ne[2]; \+ const uint32_t ne3 = dst->ne[3]; \+ \+ const uint32_t nb1 = dst->nb[1]; \+ const uint32_t nb2 = dst->nb[2]; \+ const uint32_t nb3 = dst->nb[3]; \+ \const uint32_t nr = ne1 * ne2 * ne3;struct htp_fill_context {struct htp_ops_context * octx;uint32_t nrows_per_thread;uint32_t total_rows; // ne1 * ne2 * ne3+ uint32_t row_start;bool opt_path;HVX_Vector splat_vec;uint32_t elem_size;@@ -47,10 +50,15 @@ static void fill_thread(unsigned int nth, unsigned int ith, void * data) {fill_preamble;// Parallelise over the flat row index spanning ne1*ne2*ne3- const uint32_t ir0 = fctx->nrows_per_thread * ith;- const uint32_t ir1 = MIN(ir0 + fctx->nrows_per_thread, fctx->total_rows);+ const uint32_t ir0 = fctx->row_start + fctx->nrows_per_thread * ith;+ const uint32_t ir1 = MIN(ir0 + fctx->nrows_per_thread, fctx->row_start + fctx->total_rows);- uint64_t t1 = HAP_perf_get_qtimer_count();+ if (ir0 >= ir1) {+ return;+ }++ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir0);if (fctx->opt_path) {// Opt path: tensor is fully contiguous, treat as flat array@@ -69,9 +77,8 @@ static void fill_thread(unsigned int nth, unsigned int ith, void * data) {}}- uint64_t t2 = HAP_perf_get_qtimer_count();- FARF(HIGH, "fill %u/%u: rows %u:%u usec %u\n",- ith, nth, ir0, ir1, (unsigned) HAP_perf_qtimer_count_to_us(t2 - t1));+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir1);+ FARF(HIGH, "fill %u/%u: rows %u:%u\n", ith, nth, ir0, ir1);}int op_fill(struct htp_ops_context * octx) {@@ -85,8 +92,23 @@ int op_fill(struct htp_ops_context * octx) {return HTP_STATUS_OK;}+ uint32_t row_start = 0;+ uint32_t nrows = nr;++ if (octx->ctx->mdev.count > 1) {+ const uint32_t row_size = nb1;+ const uint32_t rows_per_chunk = (row_size > 0) ? (HEX_L2_LINE_SIZE / hex_gcd_u32(row_size, HEX_L2_LINE_SIZE)) : 1;+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(nr, htp_tensor_mdev_data_aligned(dst) ? rows_per_chunk : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ nrows = range.count;+ }++ if (nrows == 0) {+ return HTP_STATUS_OK;+ }+// nr = ne1*ne2*ne3 (flat row count across all outer dims); parallelise over it.- const uint32_t n_threads = MIN(nr, octx->n_threads);+ const uint32_t n_threads = octx->n_threads;// Optimize if fully contiguous: skip stride arithmetic, treat as flat arrayconst bool opt_path = (nb2 == nb1 * ne1) && (nb3 == nb2 * ne2);@@ -99,8 +121,9 @@ int op_fill(struct htp_ops_context * octx) {struct htp_fill_context fctx = {.octx = octx,- .nrows_per_thread = (nr + n_threads - 1) / n_threads,- .total_rows = nr,+ .nrows_per_thread = fastdiv(nrows + n_threads - 1, &octx->n_threads_div),+ .total_rows = nrows,+ .row_start = row_start,.opt_path = opt_path,};@@ -117,7 +140,7 @@ int op_fill(struct htp_ops_context * octx) {return HTP_STATUS_NO_SUPPORT;}- worker_pool_run_func(octx->ctx->worker_pool, fill_thread, &fctx, n_threads);+ work_queue_run(octx->ctx->work_queue, fill_thread, &fctx, n_threads);return HTP_STATUS_OK;}diff --git a/ggml/src/ggml-hexagon/htp/flash-attn-ops.c b/ggml/src/ggml-hexagon/htp/flash-attn-ops.cindex c76b4d3a3a..8a1caba22b 100644--- a/ggml/src/ggml-hexagon/htp/flash-attn-ops.c+++ b/ggml/src/ggml-hexagon/htp/flash-attn-ops.c@@ -5,7 +5,6 @@#include <assert.h>#include <HAP_compute_res.h>#include <HAP_farf.h>-#include <HAP_perf.h>#include <math.h>#include <stdbool.h>#include <stdatomic.h>@@ -75,6 +74,7 @@ struct htp_fa_context {uint32_t qrows;uint32_t qrows_per_thread;+ uint32_t qrow_start;bool is_q_fp32;@@ -89,8 +89,6 @@ struct htp_fa_context {const struct htp_tensor * k;const struct htp_tensor * v;-- uint64_t t_start;};struct hmx_fa_context {@@ -206,10 +204,9 @@ static void flash_attn_ext_f16_thread(unsigned int nth, unsigned int ith, void *const uint32_t nb3 = dst->nb[3];// total rows in q- const uint32_t nr = factx->qrows;- const uint32_t dr = factx->qrows_per_thread;- const uint32_t ir0 = dr * ith;- const uint32_t ir1 = MIN(ir0 + dr, nr);+ const uint32_t dr = factx->qrows_per_thread;+ const uint32_t ir0 = factx->qrow_start + dr * ith;+ const uint32_t ir1 = MIN(ir0 + dr, factx->qrow_start + factx->qrows);if (ir0 >= ir1) return;@@ -1888,6 +1885,24 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {const uint32_t n_threads = factx.n_threads;const uint32_t G = factx.G;+ // Multi-device: split Q blocks across devices+ const uint32_t n_q_blocks = (neq1 + Br - 1) / Br;+ uint32_t q_start_min = 0;+ uint32_t q_start_max = neq1;++ if (octx->ctx->mdev.count > 1) {+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(n_q_blocks, htp_tensor_mdev_data_aligned(dst) ? 1 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ const uint32_t block_start = range.start;+ const uint32_t block_end = range.start + range.count;++ if (block_start >= block_end) {+ return HTP_STATUS_OK;+ }++ q_start_min = block_start * Br;+ q_start_max = MIN(block_end * Br, neq1);+ }+// ======== VTCM allocation (GQA-aware) ========// K/V row sizes drive the DMA descriptors (not the VTCM layout) and are used// throughout the KV loop below.@@ -1977,7 +1992,7 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {// ======== Main loop ========for (uint32_t ib3 = 0; ib3 < neq3; ++ib3) {const uint32_t im3 = mask ? fastmodulo(ib3, mask->ne[3], &factx.src3_div3) : 0;- for (uint32_t q_start = 0; q_start < neq1; q_start += Br) {+ for (uint32_t q_start = q_start_min; q_start < q_start_max; q_start += Br) {const uint32_t n_rows_q = hex_smin(Br, neq1 - q_start);const size_t n_rows_g = n_rows_q * G;const size_t g_br_actual = hex_align_up(n_rows_g, HMX_FP16_TILE_N_ROWS);@@ -1991,8 +2006,9 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {// 1. Push Q and KV DMAs for the very first iteration.// Subsequent iterations are enqueued early at the end of the previous iteration.- if (ib3 == 0 && q_start == 0 && kv_head == 0) {- const uint8_t * q_ptr = (const uint8_t *) q->data;+ if (ib3 == 0 && q_start == q_start_min && kv_head == 0) {+ const uint8_t * q_ptr = (const uint8_t *) q->data + q_start * q->nb[1] ++ (kv_head * factx.G) * q->nb[2] + ib3 * q->nb[3];const size_t q_row_bytes = q_transposed ? n_rows_q * q_row_bytes_trans_factor : q_row_bytes_untransposed;const size_t n_rows = q_transposed ? factx.G : n_rows_q;dma_queue_push(dma, dma_make_ptr(factx.vtcm_q_dma, q_ptr), q_row_bytes, hex_smax(q_src_stride, q_row_bytes), q_row_bytes, n_rows);@@ -2311,8 +2327,8 @@ int hmx_flash_attn_ext(struct htp_ops_context * octx) {if (next_kv_head >= n_kv_heads) {next_kv_head = 0;next_q_start = q_start + Br;- if (next_q_start >= neq1) {- next_q_start = 0;+ if (next_q_start >= q_start_max) {+ next_q_start = q_start_min;next_ib3 = ib3 + 1;}}@@ -2398,6 +2414,10 @@ int op_flash_attn_ext(struct htp_ops_context * octx) {return HTP_STATUS_NO_SUPPORT;}+ if (!htp_ops_context_set_n_threads(octx, kparams->n_threads)) {+ return HTP_STATUS_INVAL_PARAMS;+ }+if (kparams->kernel_type == HTP_FA_KERNEL_HMX) {return hmx_flash_attn_ext(octx);}@@ -2407,8 +2427,6 @@ int op_flash_attn_ext(struct htp_ops_context * octx) {factx.k = k;factx.v = v;- factx.t_start = HAP_perf_get_qtimer_count();-factx.src0_div21 = kparams->u.hvx.src0_div21;factx.src0_div1 = kparams->u.hvx.src0_div1;@@ -2451,8 +2469,30 @@ int op_flash_attn_ext(struct htp_ops_context * octx) {}// total rows in q- factx.qrows = kparams->qrows;- factx.qrows_per_thread = kparams->qrows_per_thread;+ const uint32_t neq1 = q->ne[1];+ const uint32_t neq2 = q->ne[2];+ const uint32_t neq3 = q->ne[3];+ const uint32_t total_qrows = neq1 * neq2 * neq3;++ uint32_t qrow_start = 0;+ uint32_t qrows = total_qrows;++ if (octx->ctx->mdev.count > 1) {+ const bool can_split = htp_tensor_mdev_data_aligned(dst) && ((dst->nb[1] & (HTP_TENSOR_MDEV_LINE_SIZE - 1)) == 0);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_qrows, can_split ? 1 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ qrow_start = range.start;+ qrows = range.count;+ }++ if (qrows == 0) {+ return HTP_STATUS_OK;+ }++ const uint32_t n_threads = octx->n_threads;++ factx.qrows = qrows;+ factx.qrow_start = qrow_start;+ factx.qrows_per_thread = fastdiv(qrows + n_threads - 1, &octx->n_threads_div);size_t size_vkq_acc = hex_round_up(v->ne[0] * sizeof(float), 128); // VKQ32@@ -2461,18 +2501,18 @@ int op_flash_attn_ext(struct htp_ops_context * octx) {uint8_t * vtcm_cur = octx->ctx->vtcm_base;- factx.spad_q = vtcm_seq_alloc(&vtcm_cur, size_q_block * octx->n_threads);- factx.spad_k = vtcm_seq_alloc(&vtcm_cur, factx.size_k_block * 2 * octx->n_threads);- factx.spad_v = vtcm_seq_alloc(&vtcm_cur, factx.size_v_block * 2 * octx->n_threads);- factx.spad_m = vtcm_seq_alloc(&vtcm_cur, (mask ? factx.size_m_block * HVX_FA_DMA_CACHE_SIZE : 0) * octx->n_threads);- factx.spad_a = vtcm_seq_alloc(&vtcm_cur, size_vkq_acc * octx->n_threads);+ factx.spad_q = vtcm_seq_alloc(&vtcm_cur, size_q_block * n_threads);+ factx.spad_k = vtcm_seq_alloc(&vtcm_cur, factx.size_k_block * 2 * n_threads);+ factx.spad_v = vtcm_seq_alloc(&vtcm_cur, factx.size_v_block * 2 * n_threads);+ factx.spad_m = vtcm_seq_alloc(&vtcm_cur, (mask ? factx.size_m_block * HVX_FA_DMA_CACHE_SIZE : 0) * n_threads);+ factx.spad_a = vtcm_seq_alloc(&vtcm_cur, size_vkq_acc * n_threads);if ((size_t) (vtcm_cur - octx->ctx->vtcm_base) > octx->ctx->vtcm_size) {return HTP_STATUS_VTCM_TOO_SMALL;}if (!(octx->flags & HTP_OPFLAGS_SKIP_COMPUTE)) {- work_queue_run(octx->ctx->work_queue, flash_attn_ext_f16_thread, &factx, octx->n_threads);+ work_queue_run(octx->ctx->work_queue, flash_attn_ext_f16_thread, &factx, n_threads);}return HTP_STATUS_OK;diff --git a/ggml/src/ggml-hexagon/htp/flash-attn-ops.h b/ggml/src/ggml-hexagon/htp/flash-attn-ops.hindex c4d1906316..0278454114 100644--- a/ggml/src/ggml-hexagon/htp/flash-attn-ops.h+++ b/ggml/src/ggml-hexagon/htp/flash-attn-ops.h@@ -51,6 +51,7 @@ struct htp_fa_kernel_params {uint32_t qrows;uint32_t qrows_per_thread;+ uint32_t qrow_start;float m0;float m1;uint32_t n_head_log2;diff --git a/ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c b/ggml/src/ggml-hexagon/htp/gated-delta-net-ops.cindex 9665521529..0b6529571d 100644--- a/ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c+++ b/ggml/src/ggml-hexagon/htp/gated-delta-net-ops.c@@ -4,10 +4,13 @@#include "hvx-utils.h"#include "hex-fastdiv.h"+#include "hex-common.h"+#include "hex-profile.h"#define GGML_COMMON_DECL_C#include "ggml-common.h"#include "htp-ctx.h"+#include "htp-tensor.h"#ifndef MIN#define MIN(a, b) ((a) < (b) ? (a) : (b))@@ -22,6 +25,8 @@ struct htp_gdn_context {size_t state_bytes;uint8_t * vtcm_base;size_t vtcm_per_thread;+ uint32_t row_start;+ uint32_t nrows;};static inline HVX_Vector gdn_mul_dot_f32(float * restrict dst, const float * restrict mul, const float * restrict dot, uint32_t n) {@@ -586,8 +591,9 @@ static void gated_delta_net_f32_pp_thread(unsigned int nth, unsigned int ith, voconst uint32_t n_seqs = v->ne[3];const uint32_t K = octx->op_params[0];- const uint32_t total_rows = H * n_seqs;- if (ith >= total_rows) {+ const uint32_t row_end = gctx->row_start + gctx->nrows;++ if (ith >= gctx->nrows) {return;}@@ -621,11 +627,11 @@ static void gated_delta_net_f32_pp_thread(unsigned int nth, unsigned int ith, voconst uint64_t state_seq_stride = state->nb[3] / sizeof(float);const uint64_t state_size_per_snap = (uint64_t) S_v * S_v * H * n_seqs;- uint32_t ir_prefetch = ith;+ uint32_t ir_prefetch = gctx->row_start + ith;int spad_idx = 0;// Prefetch preamble (up to 2 steps)- for (int k = 0; k < 2 && ir_prefetch < total_rows; k++) {+ for (int k = 0; k < 2 && ir_prefetch < row_end; k++) {const uint32_t piv1 = fastmodulo(ir_prefetch, H, &fd_H);const uint32_t piv3 = fastdiv(ir_prefetch, &fd_H);const float * ps_in = state_in_base + (uint64_t) piv3 * state_seq_stride + (uint64_t) piv1 * S_v * S_v;@@ -646,8 +652,11 @@ static void gated_delta_net_f32_pp_thread(unsigned int nth, unsigned int ith, vospad_idx ^= 1;}+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) (gctx->row_start + ith));+int curr_spad_idx = 0;- for (uint32_t ir = ith; ir < total_rows; ir += nth) {+ for (uint32_t ir = gctx->row_start + ith; ir < row_end; ir += nth) {dma_queue_pop(dma);dma_queue_pop(dma);@@ -812,7 +821,7 @@ static void gated_delta_net_f32_pp_thread(unsigned int nth, unsigned int ith, voS_v * sizeof(float), S_v);// Prefetch next block (if any)- if (ir_prefetch < total_rows) {+ if (ir_prefetch < row_end) {const uint32_t piv1 = fastmodulo(ir_prefetch, H, &fd_H);const uint32_t piv3 = fastdiv(ir_prefetch, &fd_H);const float * ps_in = state_in_base + (uint64_t) piv3 * state_seq_stride + (uint64_t) piv1 * S_v * S_v;@@ -828,6 +837,7 @@ static void gated_delta_net_f32_pp_thread(unsigned int nth, unsigned int ith, vocurr_spad_idx ^= 1;}dma_queue_flush(dma);+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) row_end);}@@ -847,8 +857,9 @@ static void gated_delta_net_f32_tg_thread(unsigned int nth, unsigned int ith, voconst uint32_t H = v->ne[1];const uint32_t n_seqs = v->ne[3];- const uint32_t total_rows = H * n_seqs;- if (ith >= total_rows) {+ const uint32_t row_end = gctx->row_start + gctx->nrows;++ if (ith >= gctx->nrows) {return;}@@ -881,11 +892,11 @@ static void gated_delta_net_f32_tg_thread(unsigned int nth, unsigned int ith, voconst uint64_t state_seq_stride = state->nb[3] / sizeof(float);- uint32_t ir_prefetch = ith;+ uint32_t ir_prefetch = gctx->row_start + ith;int spad_idx = 0;// Prefetch preamble (up to 2 steps)- for (int k = 0; k < 2 && ir_prefetch < total_rows; k++) {+ for (int k = 0; k < 2 && ir_prefetch < row_end; k++) {const uint32_t piv1 = fastmodulo(ir_prefetch, H, &fd_H);const uint32_t piv3 = fastdiv(ir_prefetch, &fd_H);const float * ps_in = state_in_base + (uint64_t) piv3 * state_seq_stride + (uint64_t) piv1 * S_v * S_v;@@ -906,8 +917,11 @@ static void gated_delta_net_f32_tg_thread(unsigned int nth, unsigned int ith, vospad_idx ^= 1;}+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) (gctx->row_start + ith));+int curr_spad_idx = 0;- for (uint32_t ir = ith; ir < total_rows; ir += nth) {+ for (uint32_t ir = gctx->row_start + ith; ir < row_end; ir += nth) {dma_queue_pop(dma);dma_queue_pop(dma);@@ -1057,7 +1071,7 @@ static void gated_delta_net_f32_tg_thread(unsigned int nth, unsigned int ith, voS_v * sizeof(float), S_v);// Prefetch next block (if any)- if (ir_prefetch < total_rows) {+ if (ir_prefetch < row_end) {const uint32_t piv1 = fastmodulo(ir_prefetch, H, &fd_H);const uint32_t piv3 = fastdiv(ir_prefetch, &fd_H);const float * ps_in = state_in_base + (uint64_t) piv3 * state_seq_stride + (uint64_t) piv1 * S_v * S_v;@@ -1073,6 +1087,7 @@ static void gated_delta_net_f32_tg_thread(unsigned int nth, unsigned int ith, vocurr_spad_idx ^= 1;}dma_queue_flush(dma);+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) row_end);}@@ -1085,10 +1100,6 @@ int op_gated_delta_net(struct htp_ops_context * octx) {const struct htp_tensor * state = octx->src[5];const struct htp_tensor * dst = octx->dst;- if (!q || !k || !v || !g || !beta || !state || !dst) {- return HTP_STATUS_INVAL_PARAMS;- }-if (q->type != HTP_TYPE_F32 || k->type != HTP_TYPE_F32 || v->type != HTP_TYPE_F32 ||g->type != HTP_TYPE_F32 || beta->type != HTP_TYPE_F32 || state->type != HTP_TYPE_F32 ||dst->type != HTP_TYPE_F32) {@@ -1124,16 +1135,37 @@ int op_gated_delta_net(struct htp_ops_context * octx) {return HTP_STATUS_OK;}+ const uint32_t total_rows = H * n_seqs;++ uint32_t row_start = 0;+ uint32_t nrows = total_rows;++ if (octx->ctx->mdev.count > 1) {+ const uint32_t head_bytes = S_v * sizeof(float);+ const uint32_t rows_per_chunk = (head_bytes > 0) ? (HEX_L2_LINE_SIZE / hex_gcd_u32(head_bytes, HEX_L2_LINE_SIZE)) : 1;+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_rows, htp_tensor_mdev_data_aligned(dst) ? rows_per_chunk : 0,+ octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ nrows = range.count;+ }++ if (nrows == 0) {+ return HTP_STATUS_OK;+ }++ const uint32_t n_threads = octx->n_threads;+struct htp_gdn_context gctx;gctx.octx = octx;- gctx.rows_per_thread = (H * n_seqs + octx->n_threads - 1) / octx->n_threads;+ gctx.row_start = row_start;+ gctx.nrows = nrows;+ gctx.rows_per_thread = fastdiv(nrows + n_threads - 1, &octx->n_threads_div);gctx.state_bytes = (size_t) S_v * S_v * sizeof(float);size_t state_aligned = (size_t) S_v * S_v * sizeof(float);state_aligned = (state_aligned + 127) & ~(size_t)127;- assert(octx->ctx->vtcm_base != NULL);- assert(octx->ctx->vtcm_size >= 2 * state_aligned * octx->n_threads);+ assert(octx->ctx->vtcm_size >= 2 * state_aligned * n_threads);gctx.vtcm_base = octx->ctx->vtcm_base;gctx.vtcm_per_thread = 2 * state_aligned;@@ -1148,9 +1180,9 @@ int op_gated_delta_net(struct htp_ops_context * octx) {gctx.vtcm_per_thread * octx->n_threads, octx->n_threads);if (n_tokens == 1) {- worker_pool_run_func(octx->ctx->worker_pool, gated_delta_net_f32_tg_thread, &gctx, octx->n_threads);+ work_queue_run(octx->ctx->work_queue, gated_delta_net_f32_tg_thread, &gctx, n_threads);} else {- worker_pool_run_func(octx->ctx->worker_pool, gated_delta_net_f32_pp_thread, &gctx, octx->n_threads);+ work_queue_run(octx->ctx->work_queue, gated_delta_net_f32_pp_thread, &gctx, n_threads);}return HTP_STATUS_OK;diff --git a/ggml/src/ggml-hexagon/htp/get-rows-ops.c b/ggml/src/ggml-hexagon/htp/get-rows-ops.cindex a87962d229..d294ba57a0 100644--- a/ggml/src/ggml-hexagon/htp/get-rows-ops.c+++ b/ggml/src/ggml-hexagon/htp/get-rows-ops.c@@ -10,6 +10,7 @@#define GGML_COMMON_DECL_C#include "ggml-common.h"+#include "hex-common.h"#include "htp-ctx.h"#include "htp-ops.h"#include "htp-tensor.h"@@ -23,9 +24,12 @@ struct get_rows_context {const struct htp_get_rows_kernel_params * kparams;struct htp_get_rows_vtcm_layout vtcm_layout;uint8_t * vtcm_base;+ uint32_t task_start;+ uint32_t tasks;+ uint32_t tasks_per_thread;};-#define get_rows_preamble \+#define get_rows_preamble \const uint32_t ne00 = octx->src[0]->ne[0]; \const uint32_t ne01 = octx->src[0]->ne[1]; \const uint32_t ne02 = octx->src[0]->ne[2]; \@@ -61,12 +65,12 @@ static void get_rows_thread_st_##IDX_TYPE(unsigned int nth, unsigned int ith, vostruct htp_ops_context * octx = grctx->octx; \const struct htp_get_rows_kernel_params * kparams = grctx->kparams; \get_rows_preamble; \- const uint32_t dr = kparams->tasks_per_thread; \- const uint32_t ir0 = dr * ith; \- if (ir0 >= kparams->total_tasks) { \+ const uint32_t dr = grctx->tasks_per_thread; \+ const uint32_t ir0 = grctx->task_start + dr * ith; \+ if (ir0 >= grctx->task_start + grctx->tasks) { \return; \} \- const uint32_t ir1 = MIN(ir0 + dr, kparams->total_tasks); \+ const uint32_t ir1 = MIN(ir0 + dr, grctx->task_start + grctx->tasks); \const uint32_t row_size_bytes = htp_tensor_get_row_size(octx->src[0]->type, ne00); \dma_queue * dma_queue = octx->ctx->dma[ith]; \for (uint32_t i = ir0; i < ir1; ++i) { \@@ -101,12 +105,12 @@ static void get_rows_thread_##TYPE_NAME##_##IDX_TYPE(unsigned int nth, unsignedconst struct htp_get_rows_kernel_params * kparams = grctx->kparams; \get_rows_preamble; \struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \- const uint32_t dr = kparams->tasks_per_thread; \- const uint32_t ir0 = dr * ith; \- if (ir0 >= kparams->total_tasks) { \+ const uint32_t dr = grctx->tasks_per_thread; \+ const uint32_t ir0 = grctx->task_start + dr * ith; \+ if (ir0 >= grctx->task_start + grctx->tasks) { \return; \} \- const uint32_t ir1 = MIN(ir0 + dr, kparams->total_tasks); \+ const uint32_t ir1 = MIN(ir0 + dr, grctx->task_start + grctx->tasks); \const uint32_t chunks_per_row = kparams->chunks_per_row; \const uint32_t chunk_size = kparams->chunk_size; \dma_queue * dma_queue = octx->ctx->dma[ith]; \@@ -225,13 +229,41 @@ int op_get_rows(struct htp_ops_context * octx) {return HTP_STATUS_OK;}+ const struct htp_tensor * dst = octx->dst;+ const uint32_t total_tasks = kparams->total_tasks;+ const size_t dst_row_size = htp_tensor_get_row_size(dst->type, dst->ne[0]);++ uint32_t task_start = 0;+ uint32_t tasks = total_tasks;++ if (octx->ctx->mdev.count > 1) {+ uint32_t tasks_per_chunk = 1;+ htp_tensor_mdev_rows_per_chunk(dst, dst_row_size / dst->ne[0], (uint32_t) dst_row_size, &tasks_per_chunk);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_tasks, tasks_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ task_start = range.start;+ tasks = range.count;+ }++ if (tasks == 0) {+ return HTP_STATUS_OK;+ }++ if (!htp_ops_context_set_n_threads(octx, (uint32_t) kparams->n_threads)) {+ return HTP_STATUS_INVAL_PARAMS;+ }++ const uint32_t n_threads = octx->n_threads;+struct get_rows_context grctx;grctx.octx = octx;grctx.kparams = kparams;grctx.vtcm_base = (uint8_t *)octx->ctx->vtcm_base;+ grctx.task_start = task_start;+ grctx.tasks = tasks;+ grctx.tasks_per_thread = fastdiv(tasks + n_threads - 1, &octx->n_threads_div);const uint32_t ne00 = octx->src[0]->ne[0];- htp_get_rows_vtcm_layout_build(&grctx.vtcm_layout, octx->src[0]->type, ne00, kparams->n_threads);+ htp_get_rows_vtcm_layout_build(&grctx.vtcm_layout, octx->src[0]->type, ne00, n_threads);const bool is_i32 = (octx->src[1]->type == HTP_TYPE_I32);@@ -247,14 +279,14 @@ int op_get_rows(struct htp_ops_context * octx) {}}- FARF(HIGH, "get-rows: (%ux%ux%ux%u) x (%ux%ux%ux%u) -> (%ux%ux%ux%u) : src0-vtcm-size %zu dst-vtcm-size %zu use_dma=%d n_threads %d\n",+ FARF(HIGH, "get-rows: (%ux%ux%ux%u) x (%ux%ux%ux%u) -> (%ux%ux%ux%u) : src0-vtcm-size %zu dst-vtcm-size %zu use-dma %d n-threads %d\n",octx->src[0]->ne[0], octx->src[0]->ne[1], octx->src[0]->ne[2], octx->src[0]->ne[3],octx->src[1]->ne[0], octx->src[1]->ne[1], octx->src[1]->ne[2], octx->src[1]->ne[3],octx->dst->ne[0], octx->dst->ne[1], octx->dst->ne[2], octx->dst->ne[3],- grctx.vtcm_layout.src0_bytes_per_thread * kparams->n_threads,- grctx.vtcm_layout.dst_bytes_per_thread * kparams->n_threads,- kparams->use_dma, kparams->n_threads);+ grctx.vtcm_layout.src0_bytes_per_thread * n_threads,+ grctx.vtcm_layout.dst_bytes_per_thread * n_threads,+ kparams->use_dma, n_threads);- work_queue_run(octx->ctx->work_queue, q_func, &grctx, kparams->n_threads);+ work_queue_run(octx->ctx->work_queue, q_func, &grctx, n_threads);return HTP_STATUS_OK;}diff --git a/ggml/src/ggml-hexagon/htp/hex-common.h b/ggml/src/ggml-hexagon/htp/hex-common.hindex 4714486a04..e6a52540d5 100644--- a/ggml/src/ggml-hexagon/htp/hex-common.h+++ b/ggml/src/ggml-hexagon/htp/hex-common.h@@ -77,4 +77,13 @@ static inline bool hex_add_overflow(size_t a, size_t b, size_t *out) {return false;}+static inline uint32_t hex_gcd_u32(uint32_t a, uint32_t b) {+ while (b != 0) {+ uint32_t t = b;+ b = a % b;+ a = t;+ }+ return a;+}+#endif // HEX_COMMON_Hdiff --git a/ggml/src/ggml-hexagon/htp/hex-utils.h b/ggml/src/ggml-hexagon/htp/hex-utils.hindex 1b39650300..853f1c1b2d 100644--- a/ggml/src/ggml-hexagon/htp/hex-utils.h+++ b/ggml/src/ggml-hexagon/htp/hex-utils.h@@ -39,7 +39,6 @@ static inline void hex_l2fetch_block(const void * addr, size_t size) {#define HEX_L2_LINE_SIZE 128#define HEX_L2_BLOCK_SIZE (HEX_L2_LINE_SIZE * 4) // flush granularity (lines per loop iteration)-#define HEX_L2_FLUSH_IL_THRESHOLD 1024 // inline flush threshold#define HEX_L2_FLUSH_WQ_THRESHOLD (4 * 1024)#define HEX_L2_FLUSH_ALL_THRESHOLD (4 * 1024 * 1024)diff --git a/ggml/src/ggml-hexagon/htp/hmx-utils.h b/ggml/src/ggml-hexagon/htp/hmx-utils.hindex 2a61ca7349..ad295cb7df 100644--- a/ggml/src/ggml-hexagon/htp/hmx-utils.h+++ b/ggml/src/ggml-hexagon/htp/hmx-utils.h@@ -27,7 +27,7 @@ static inline void hmx_init_column_scales(void *out_scales, HVX_Vector v_scale)// vscatter offsets for fused dequant+transpose: write K-values directly to [K][N] tile.// word[i] = i*128 maps K-row-pair i to byte offset i*128.// Column offset (n*4) is added at runtime. Entries 0..15 cover one tile (region 2047);-// entries 16..31 cover the next adjacent tile (region 4095) — pick region size at the+// entries 16..31 cover the next adjacent tile (region 4095) - pick region size at the// call site to scatter into one tile (masked) or two contiguous tiles (unmasked).static const int32_t hmx_transpose_scatter_offsets[32] __attribute__((aligned(VLEN))) = {0 * 128, 1 * 128, 2 * 128, 3 * 128, 4 * 128, 5 * 128, 6 * 128, 7 * 128, 8 * 128, 9 * 128, 10 * 128,@@ -198,16 +198,16 @@ static inline void hmx_interleave_cols_to_tiles(__fp16 * restrict tiles_out,}// --- HMX inline asm macros for load-store packetization ----#define HMX_LOAD_MPY_F16(act, wt, range) \- "{\n" \+#define HMX_LOAD_MPY_F16(act, wt, range) \+ "{\n" \" activation.hf = mxmem(" act ", " range ")\n" \- " weight.hf = mxmem(" wt ", " range ")\n" \+ " weight.hf = mxmem(" wt ", " range ")\n" \"}\n"-#define HMX_LOAD_MPY_DEEP_F16(act, wt, range) \- "{\n" \+#define HMX_LOAD_MPY_DEEP_F16(act, wt, range) \+ "{\n" \" activation.hf = mxmem(" act ", " range "):deep\n" \- " weight.hf = mxmem(" wt ", " range ")\n" \+ " weight.hf = mxmem(" wt ", " range ")\n" \"}\n"#define HMX_STORE_AFTER_F16(out, scale_reg) \diff --git a/ggml/src/ggml-hexagon/htp/htp-ctx.h b/ggml/src/ggml-hexagon/htp/htp-ctx.hindex c8a909d619..3b60c8bdb0 100644--- a/ggml/src/ggml-hexagon/htp/htp-ctx.h+++ b/ggml/src/ggml-hexagon/htp/htp-ctx.h@@ -19,7 +19,7 @@#endif#define HTP_MAX_MMAPS 16-#define HTP_MAX_DIRTY_RANGES 16+#define HTP_MAX_DIRTY_RANGES 32// Memory mappingstruct htp_mmap {@@ -29,6 +29,11 @@ struct htp_mmap {uint32_t reserved;};+struct htp_dirty_range {+ uint32_t start;+ uint32_t end;+};+// Scratchpad statestruct htp_spad {const struct htp_tensor * src; // original src of the data (for reuse)@@ -38,6 +43,14 @@ struct htp_spad {uint32_t size_per_thread; // size per thread};+struct htp_mdev_group {+ uint16_t idx;+ uint16_t count;+ struct fastdiv_values count_div;+ uint8_t * fence_base;+ uint32_t fence_seq;+};+struct htp_context;// Context while processing an Op@@ -65,8 +78,10 @@ struct htp_ops_context {struct htp_spad src3_spad;struct htp_spad dst_spad;- uint32_t n_threads;- uint32_t flags;+ uint32_t flags;+ uint32_t n_threads;+ struct fastdiv_values n_threads_div;+ int status;};// Main context for htp DSP backend@@ -76,6 +91,7 @@ struct htp_context {struct htp_mmap mmap[HTP_MAX_MMAPS];dma_queue_t dma[HTP_MAX_NTHREADS];dma_queue_t dma_cached[HTP_MAX_NTHREADS];+ struct htp_thread_trace trace[HTP_MAX_NTHREADS + 1];work_queue_t work_queue;hmx_queue_t hmx_queue;@@ -88,7 +104,6 @@ struct htp_context {bool hmx_enabled;bool etm;uint32_t profiler;- struct htp_thread_trace trace[HTP_MAX_NTHREADS + 1];uint8_t * vtcm_base;size_t vtcm_size;@@ -97,16 +112,13 @@ struct htp_context {atomic_bool vtcm_needs_release;uint64_t max_vmem;- struct htp_dirty_range {- uint32_t start;- uint32_t end;- uint32_t bi;- } dirty_ranges[HTP_MAX_DIRTY_RANGES];+ struct htp_dirty_range dirty_ranges[HTP_MAX_DIRTY_RANGES];// Persistent DDR scratchpad for MUL_MAT_ID mappingsvoid * ddr_spad_base;size_t ddr_spad_size;+ struct htp_mdev_group mdev;struct htp_ops_context octx;qurt_thread_t main_thread;@@ -115,6 +127,27 @@ struct htp_context {size_t footprint;};+static inline bool htp_ops_context_set_n_threads(struct htp_ops_context * octx, uint32_t n_threads) {+ if (n_threads == 0 || n_threads > octx->ctx->n_threads) {+ return false;+ }++ if (n_threads != octx->n_threads) {+ octx->n_threads = n_threads;+ octx->n_threads_div = n_threads == octx->ctx->n_threads+ ? octx->ctx->n_threads_div+ : init_fastdiv_values(n_threads);+ }++ return true;+}++static inline void htp_ops_context_set_status(struct htp_ops_context * octx, int status) {+ if (status > HTP_STATUS_OK && octx->status == HTP_STATUS_OK) {+ octx->status = status;+ }+}+int op_matmul(struct htp_ops_context * octx);int op_matmul_id(struct htp_ops_context * octx);int op_matmul_nx(struct htp_ops_context * octx);diff --git a/ggml/src/ggml-hexagon/htp/htp-fence.h b/ggml/src/ggml-hexagon/htp/htp-fence.hnew file mode 100644index 0000000000..7450b5de53--- /dev/null+++ b/ggml/src/ggml-hexagon/htp/htp-fence.h@@ -0,0 +1,89 @@+#ifndef HTP_FENCE_H+#define HTP_FENCE_H++#include <stdatomic.h>+#include <stdint.h>++#include <HAP_farf.h>++#include "hex-utils.h"+#include "htp-ops.h"+#include "htp-ctx.h"++static inline atomic_uint * htp_mdev_fence_slot(const void * fence_base, uint32_t idx) {+ return (atomic_uint *) ((const uint8_t *) fence_base + (size_t) idx * HTP_FENCE_SLOT_SIZE);+}++static inline void htp_fence_write(void * fence_ptr, uint32_t seq, uint32_t status) {+ atomic_uint * fence = (atomic_uint *) fence_ptr;+ atomic_store(&fence[1], status);+ atomic_store(&fence[0], seq);+ asm volatile ("syncht" : : : "memory");+ Q6_dccleaninva_A((void *) fence);+}++static inline void htp_fence_read(const void * fence_ptr, uint32_t * seq, uint32_t * status) {+ const atomic_uint * fence = (const atomic_uint *) fence_ptr;+ Q6_dccleaninva_A((void *) fence);+ asm volatile ("syncht" : : : "memory");+ *seq = atomic_load(&fence[0]);+ *status = atomic_load(&fence[1]);+}++static inline void htp_mdev_group_barrier(struct htp_ops_context * octx) {+ struct htp_context * ctx = octx->ctx;+ if (ctx->mdev.count <= 1) {+ return;+ }++ const uint32_t seq = ++ctx->mdev.fence_seq;++ struct htp_thread_trace * tr = &ctx->trace[0];+ htp_trace_event_start(tr, HTP_TRACE_EVT_FENCE, (uint16_t) seq);++ const uint32_t mdev_idx = ctx->mdev.idx;+ const uint32_t mdev_count = ctx->mdev.count;++ uint8_t * fence_base = ctx->mdev.fence_base;+ atomic_uint * my_fence = htp_mdev_fence_slot(fence_base, mdev_idx);+ htp_fence_write(my_fence, seq, octx->status);++ for (uint32_t d = 0; d < mdev_count; d++) {+ if (d == mdev_idx) continue;+ atomic_uint * peer_fence = htp_mdev_fence_slot(fence_base, d);+ uint64_t spins = 0;+ while (1) {+ uint32_t peer_seq;+ uint32_t peer_status;+ htp_fence_read(peer_fence, &peer_seq, &peer_status);+ if ((int32_t)(peer_seq - seq) >= 0) {+ if (peer_status > HTP_STATUS_OK) {+ FARF(ERROR, "ggml-hex: mdev %u peer %u failed with status %u : seq 0x%08x\n",+ mdev_idx, d, peer_status, seq);+ htp_ops_context_set_status(octx, peer_status);+ }+ break;+ }+ if (++spins == 10000) {+ FARF(ALWAYS, "ggml-hex: mdev %u waiting for mdev %u : seq 0x%08x (b %u op %u) my-fence %p peer-fence %p peer-seq 0x%08x (diff %d)\n",+ mdev_idx, d, seq, seq >> 12, seq & 0xfff, my_fence, peer_fence, peer_seq, (int32_t)(peer_seq - seq));+ }+ if (spins > HTP_FENCE_TIMEOUT) {+ FARF(ERROR, "ggml-hex: mdev %u timeout waiting for mdev %u : seq 0x%08x (b %u op %u) peer-fence %p peer-seq 0x%08x\n",+ mdev_idx, d, seq, seq >> 12, seq & 0xfff, peer_fence, peer_seq);+ htp_ops_context_set_status(octx, HTP_STATUS_INTERNAL_ERR);+ break;+ }+ hex_pause();+ }+ }+ asm volatile ("syncht" : : : "memory");++ if (octx->status > HTP_STATUS_OK) {+ htp_fence_write(my_fence, seq, octx->status);+ }++ htp_trace_event_stop(tr, HTP_TRACE_EVT_FENCE, (uint16_t) seq);+}++#endif // HTP_FENCE_Hdiff --git a/ggml/src/ggml-hexagon/htp/htp-ops.h b/ggml/src/ggml-hexagon/htp/htp-ops.hindex 12a61b67f2..869b19b8c2 100644--- a/ggml/src/ggml-hexagon/htp/htp-ops.h+++ b/ggml/src/ggml-hexagon/htp/htp-ops.h@@ -77,6 +77,7 @@ enum htp_op_code {HTP_OP_GET_ROWS,HTP_OP_SCALE,HTP_OP_CPY,+ HTP_OP_CPY_FENCE,HTP_OP_ARGSORT,HTP_OP_SQR,HTP_OP_SQRT,@@ -100,6 +101,7 @@ enum htp_op_code {HTP_OP_ALLREDUCE,HTP_OP_ALLREDUCE_ADD,HTP_OP_GLU_SWIGLU_CLAMP,+ HTP_OP_MDEV_GROUP,HTP_OP_INVALID};@@ -114,6 +116,7 @@ enum htp_op_code {#define HTP_OP_MAX_TENSORS 8192 // must stay under 64K (uint16)#define HTP_FENCE_TIMEOUT (1000000000ULL)+#define HTP_FENCE_SLOT_SIZE 128#define HTP_OP_MAX_VMEM_DEFAULT (3355443200u)@@ -214,30 +217,26 @@ struct htp_prof_desc {};struct htp_opbatch_req {- uint32_t id; // Batch id+ uint64_t seq; // Sequence numberuint32_t n_bufs; // Number of buffersuint32_t n_tensors; // Number of tensorsuint32_t n_ops; // Number of opsuint32_t n_traces; // Number of trace descriptors per thread- uint32_t pad; // unused- uint64_t seq; // Sequence number// struct htp_buf_desc bufs[]; -- dspqueue buf 0// struct htp_tensor tensors[]; -- dspqueue buf 0// struct htp_op_desc ops[]; -- dspqueue buf 0};struct htp_opbatch_rsp {- uint32_t id; // Batch id- uint32_t status; // HTP_STATUS_...- uint32_t n_bufs; // Number of buffers- uint32_t n_tensors; // Number of tensors- uint32_t n_ops; // Number of op profile descriptors- uint32_t n_traces[HTP_MAX_NTHREADS + 1];- uint32_t usecs; // Number of usec- uint32_t pad; // align to 8 bytes+ uint64_t seq; // Sequence numberuint64_t cycles_start; // Start cycle counteruint64_t cycles_stop; // Stop cycle counter- uint64_t seq; // Sequence number+ uint32_t status; // HTP_STATUS_...+ uint32_t n_bufs; // Number of buffers+ uint32_t n_tensors; // Number of tensors+ uint32_t n_ops; // Number of op profile descriptors+ uint32_t usecs; // Number of usec+ uint32_t n_traces[HTP_MAX_NTHREADS + 1];// struct htp_prof_desc profs[]; -- dspqueue buf 0};diff --git a/ggml/src/ggml-hexagon/htp/htp-tensor.c b/ggml/src/ggml-hexagon/htp/htp-tensor.cindex ae377c9221..760ccd8313 100644--- a/ggml/src/ggml-hexagon/htp/htp-tensor.c+++ b/ggml/src/ggml-hexagon/htp/htp-tensor.c@@ -20,7 +20,7 @@ struct l2flush_range {struct l2flush_multi_task {struct htp_thread_trace * trace;- struct l2flush_range ranges[HTP_OP_MAX_INPUTS];+ struct l2flush_range ranges[HTP_MAX_DIRTY_RANGES];uint32_t n_ranges;uint32_t total_blocks;uint32_t blocks_per_thread;@@ -73,6 +73,27 @@ static void l2flush_multi_worker(unsigned int n, unsigned int i, void * data) {htp_trace_event_stop(tr, HTP_TRACE_EVT_L2FLUSH, gb_first);}+static void merge_dirty_ranges(struct htp_context * ctx) {+ for (uint32_t i = 0; i < HTP_MAX_DIRTY_RANGES; i++) {+ struct htp_dirty_range * r = &ctx->dirty_ranges[i];+ if (!r->start) continue;++ for (uint32_t j = 0; j < HTP_MAX_DIRTY_RANGES;) {+ struct htp_dirty_range * s = &ctx->dirty_ranges[j];+ if (i == j || !s->start || r->end < s->start || s->end < r->start) {+ j++;+ continue;+ }++ r->start = MIN(r->start, s->start);+ r->end = MAX(r->end, s->end);+ s->start = 0;+ s->end = 0;+ j = 0;+ }+ }+}+void htp_tensor_dirty_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n) {const struct htp_tensor * pending[HTP_OP_MAX_OUTPUTS];uint32_t n_pending = 0;@@ -83,11 +104,6 @@ void htp_tensor_dirty_all(struct htp_context * ctx, const struct htp_tensor * cocontinue;}- if (t->size <= HEX_L2_FLUSH_IL_THRESHOLD) {- hex_l2flush((void *) (uintptr_t) t->data, t->size);- continue;- }-uint32_t t_start = t->data;uint32_t t_end = t_start + t->size;@@ -110,6 +126,8 @@ void htp_tensor_dirty_all(struct htp_context * ctx, const struct htp_tensor * co}}+ merge_dirty_ranges(ctx);+if (n_pending == 0) {return;}@@ -132,8 +150,8 @@ void htp_tensor_dirty_all(struct htp_context * ctx, const struct htp_tensor * costruct htp_dirty_range * r = &ctx->dirty_ranges[idx];r->start = pending[i]->data;r->end = pending[i]->data + pending[i]->size;- r->bi = pending[i]->bi;}+ merge_dirty_ranges(ctx);return;}@@ -151,12 +169,12 @@ void htp_tensor_dirty_all(struct htp_context * ctx, const struct htp_tensor * costruct htp_dirty_range * r = &ctx->dirty_ranges[i];r->start = pending[i]->data;r->end = pending[i]->data + pending[i]->size;- r->bi = pending[i]->bi;}+ merge_dirty_ranges(ctx);return;}- if (total_evict_size > HEX_L2_FLUSH_WQ_THRESHOLD && ctx->n_threads > 1 && n_evict <= HTP_OP_MAX_INPUTS) {+ if (total_evict_size > HEX_L2_FLUSH_WQ_THRESHOLD && ctx->n_threads > 1 && n_evict <= HTP_MAX_DIRTY_RANGES) {struct l2flush_multi_task task;task.trace = ctx->trace;task.n_ranges = n_evict;@@ -195,7 +213,6 @@ void htp_tensor_dirty_all(struct htp_context * ctx, const struct htp_tensor * costruct htp_dirty_range * r = &ctx->dirty_ranges[idx];r->start = pending[i]->data;r->end = pending[i]->data + pending[i]->size;- r->bi = pending[i]->bi;}for (uint32_t i = 0; i < n_empty; i++) {@@ -203,8 +220,9 @@ void htp_tensor_dirty_all(struct htp_context * ctx, const struct htp_tensor * costruct htp_dirty_range * r = &ctx->dirty_ranges[idx];r->start = pending[n_evict + i]->data;r->end = pending[n_evict + i]->data + pending[n_evict + i]->size;- r->bi = pending[n_evict + i]->bi;}++ merge_dirty_ranges(ctx);}static void make_tensor_clean(struct htp_context * ctx, const struct htp_tensor * t) {@@ -242,17 +260,50 @@ static inline bool is_tensor_dirty(struct htp_context * ctx, const struct htp_tereturn false;}-void htp_tensor_flush_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n) {- const struct htp_tensor * dirty_tensors[HTP_OP_MAX_INPUTS];- uint32_t n_dirty = 0;+static void flush_dirty_ranges(struct htp_context * ctx, const struct htp_dirty_range * ranges, uint32_t n_ranges, uint64_t total_dirty) {+ if (total_dirty >= HEX_L2_FLUSH_WQ_THRESHOLD && ctx->n_threads > 1) {+ struct l2flush_multi_task task;+ task.trace = ctx->trace;+ task.n_ranges = n_ranges;++ uint32_t block_acc = 0;+ for (uint32_t i = 0; i < n_ranges; i++) {+ const struct htp_dirty_range * r = &ranges[i];+ struct l2flush_range * rg = &task.ranges[i];+ rg->start = hex_align_down((size_t) r->start, HEX_L2_LINE_SIZE);+ rg->end = hex_align_up((size_t) r->end, HEX_L2_LINE_SIZE);+ rg->block_first = block_acc;+ rg->n_blocks = (rg->end - rg->start + HEX_L2_BLOCK_SIZE - 1) / HEX_L2_BLOCK_SIZE;+ block_acc += rg->n_blocks;+ }++ task.total_blocks = block_acc;+ task.blocks_per_thread = fastdiv(block_acc + ctx->n_threads - 1, &ctx->n_threads_div);++ work_queue_run(ctx->work_queue, l2flush_multi_worker, &task, ctx->n_threads);+ } else {+ struct htp_thread_trace * tr = &ctx->trace[0];+ htp_trace_event_start(tr, HTP_TRACE_EVT_L2FLUSH, 0);+ for (uint32_t i = 0; i < n_ranges; i++) {+ const struct htp_dirty_range * r = &ranges[i];+ hex_l2flush((void *) (uintptr_t) r->start, r->end - r->start);+ }+ htp_trace_event_stop(tr, HTP_TRACE_EVT_L2FLUSH, 0);+ }+}++void htp_flush_dirty_ranges(struct htp_context * ctx) {+ struct htp_dirty_range ranges[HTP_MAX_DIRTY_RANGES];+ uint32_t n_ranges = 0;uint64_t total_dirty = 0;- for (uint32_t i = 0; i < n; i++) {- const struct htp_tensor * t = tensors[i];- if (t && !(t->flags & (HTP_TENSOR_WEIGHT | HTP_TENSOR_FENCE)) && is_tensor_dirty(ctx, t)) {- dirty_tensors[n_dirty++] = t;- total_dirty += t->size;+ for (uint32_t i = 0; i < HTP_MAX_DIRTY_RANGES; i++) {+ const struct htp_dirty_range * r = &ctx->dirty_ranges[i];+ if (!r->start) {+ continue;}+ ranges[n_ranges++] = *r;+ total_dirty += r->end - r->start;}if (total_dirty == 0) {@@ -264,37 +315,37 @@ void htp_tensor_flush_all(struct htp_context * ctx, const struct htp_tensor * coreturn;}- if (total_dirty >= HEX_L2_FLUSH_WQ_THRESHOLD && ctx->n_threads > 1) {- struct l2flush_multi_task task;- task.trace = ctx->trace;- task.n_ranges = 0;+ flush_dirty_ranges(ctx, ranges, n_ranges, total_dirty);+ memset(ctx->dirty_ranges, 0, sizeof(ctx->dirty_ranges));+}- uint32_t block_acc = 0;- for (uint32_t i = 0; i < n_dirty; i++) {- const struct htp_tensor * t = dirty_tensors[i];- make_tensor_clean(ctx, t);+void htp_tensor_flush_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n) {+ const struct htp_tensor * dirty_tensors[HTP_OP_MAX_INPUTS];+ struct htp_dirty_range ranges[HTP_OP_MAX_INPUTS];+ uint32_t n_dirty = 0;+ uint64_t total_dirty = 0;- struct l2flush_range * rg = &task.ranges[task.n_ranges++];- rg->start = hex_align_down((size_t) t->data, HEX_L2_LINE_SIZE);- rg->end = hex_align_up((size_t) t->data + t->size, HEX_L2_LINE_SIZE);- rg->block_first = block_acc;- rg->n_blocks = (rg->end - rg->start + HEX_L2_BLOCK_SIZE - 1) / HEX_L2_BLOCK_SIZE;- block_acc += rg->n_blocks;+ for (uint32_t i = 0; i < n; i++) {+ const struct htp_tensor * t = tensors[i];+ if (t && is_tensor_dirty(ctx, t)) {+ dirty_tensors[n_dirty++] = t;+ ranges[n_dirty - 1].start = t->data;+ ranges[n_dirty - 1].end = t->data + t->size;+ total_dirty += t->size;}+ }- task.total_blocks = block_acc;- task.blocks_per_thread = fastdiv(block_acc + ctx->n_threads - 1, &ctx->n_threads_div);+ if (total_dirty == 0) {+ return;+ }- work_queue_run(ctx->work_queue, l2flush_multi_worker, &task, ctx->n_threads);+ if (total_dirty > HEX_L2_FLUSH_ALL_THRESHOLD) {+ flush_all_dcache(ctx);return;}- struct htp_thread_trace * tr = &ctx->trace[0];+ flush_dirty_ranges(ctx, ranges, n_dirty, total_dirty);for (uint32_t i = 0; i < n_dirty; i++) {- const struct htp_tensor * t = dirty_tensors[i];- htp_trace_event_start(tr, HTP_TRACE_EVT_L2FLUSH, t->ti);- hex_l2flush((void *) (uintptr_t) t->data, t->size);- htp_trace_event_stop(tr, HTP_TRACE_EVT_L2FLUSH, t->ti);- make_tensor_clean(ctx, t);+ make_tensor_clean(ctx, dirty_tensors[i]);}}diff --git a/ggml/src/ggml-hexagon/htp/htp-tensor.h b/ggml/src/ggml-hexagon/htp/htp-tensor.hindex c9cadbae3f..3afff69170 100644--- a/ggml/src/ggml-hexagon/htp/htp-tensor.h+++ b/ggml/src/ggml-hexagon/htp/htp-tensor.h@@ -2,8 +2,20 @@#define HTP_TENSOR_H#include <stdint.h>+#include <stdbool.h>#include "htp-ops.h"#include "hex-bitmap.h"+#include "hex-common.h"+#include "hex-fastdiv.h"++enum {+ HTP_TENSOR_MDEV_LINE_SIZE = 128,+};++struct htp_tensor_mdev_range {+ uint32_t start;+ uint32_t count;+};static inline void * htp_tensor_data(const struct htp_tensor * t) {return (void *) (uintptr_t) t->data;@@ -13,6 +25,102 @@ static inline uint32_t * htp_tensor_flags(const struct htp_tensor * t) {return (uint32_t *) &t->flags;}+static inline bool htp_tensor_is_contiguous(const struct htp_tensor * t, uint32_t type_size) {+ uint32_t next_nb = type_size;+ if (t->ne[0] != 1 && t->nb[0] != next_nb) {+ return false;+ }+ next_nb *= t->ne[0];+ for (int i = 1; i < HTP_OP_MAX_DIMS; i++) {+ if (t->ne[i] != 1 && t->nb[i] != next_nb) {+ return false;+ }+ next_nb *= t->ne[i];+ }+ return true;+}++static inline bool htp_tensor_is_permuted(const struct htp_tensor * t) {+ return t->nb[0] > t->nb[1] || t->nb[1] > t->nb[2] || t->nb[2] > t->nb[3];+}++static inline bool htp_tensor_mdev_data_aligned(const struct htp_tensor * t) {+ return ((uintptr_t) t->data & (HTP_TENSOR_MDEV_LINE_SIZE - 1)) == 0;+}++static inline bool htp_tensor_can_row_partition(const struct htp_tensor * t, uint32_t elem_size) {+ if (!htp_tensor_mdev_data_aligned(t)) {+ return false;+ }+ if (t->ne[0] != 1 && t->nb[0] != elem_size) {+ return false;+ }+ if (htp_tensor_is_permuted(t)) {+ return false;+ }+ if (t->ne[1] > 1 && (t->nb[1] & (HTP_TENSOR_MDEV_LINE_SIZE - 1)) != 0) return false;+ if (t->ne[2] > 1 && (t->nb[2] & (HTP_TENSOR_MDEV_LINE_SIZE - 1)) != 0) return false;+ if (t->ne[3] > 1 && (t->nb[3] & (HTP_TENSOR_MDEV_LINE_SIZE - 1)) != 0) return false;+ return true;+}++static inline bool htp_tensor_mdev_rows_per_chunk(const struct htp_tensor * t, uint32_t elem_size, uint32_t row_size, uint32_t * rows_per_chunk) {+ *rows_per_chunk = 0;++ if (!htp_tensor_mdev_data_aligned(t)) {+ return false;+ }+ if (t->ne[0] != 1 && t->nb[0] != elem_size) {+ return false;+ }+ if (htp_tensor_is_permuted(t)) {+ return false;+ }+ if (t->ne[1] > 1 && (t->nb[1] & (HTP_TENSOR_MDEV_LINE_SIZE - 1)) == 0 &&+ (t->ne[2] <= 1 || (t->nb[2] & (HTP_TENSOR_MDEV_LINE_SIZE - 1)) == 0) &&+ (t->ne[3] <= 1 || (t->nb[3] & (HTP_TENSOR_MDEV_LINE_SIZE - 1)) == 0)) {+ *rows_per_chunk = 1;+ return true;+ }+ if (t->nb[1] == row_size &&+ (t->ne[2] <= 1 || t->nb[2] == t->nb[1] * t->ne[1]) &&+ (t->ne[3] <= 1 || t->nb[3] == t->nb[2] * t->ne[2])) {+ *rows_per_chunk = (row_size > 0) ? (HTP_TENSOR_MDEV_LINE_SIZE / hex_gcd_u32(row_size, HTP_TENSOR_MDEV_LINE_SIZE)) : 1;+ return true;+ }+ return false;+}++static inline struct htp_tensor_mdev_range htp_tensor_mdev_partition(uint32_t total_units, uint32_t units_per_chunk, uint32_t mdev_idx, uint32_t mdev_count, const struct fastdiv_values * mdev_count_div) {+ struct htp_tensor_mdev_range range = { 0, total_units };++ if (mdev_count <= 1) {+ return range;+ }++ if (units_per_chunk == 0) {+ range.start = (mdev_idx == 0) ? 0 : total_units;+ range.count = (mdev_idx == 0) ? total_units : 0;+ return range;+ }++ const uint32_t total_chunks = total_units / units_per_chunk;+ if (total_chunks < mdev_count) {+ range.start = (mdev_idx == 0) ? 0 : total_units;+ range.count = (mdev_idx == 0) ? total_units : 0;+ return range;+ }++ const uint32_t chunks_per_mdev = fastdiv(total_chunks + mdev_count - 1, mdev_count_div);+ range.start = MIN(mdev_idx * chunks_per_mdev * units_per_chunk, total_units);+ if (mdev_idx == mdev_count - 1) {+ range.count = total_units - range.start;+ } else {+ range.count = MIN(chunks_per_mdev * units_per_chunk, total_units - range.start);+ }+ return range;+}+static inline uint32_t htp_tensor_get_row_size(int type, uint32_t ne00) {switch (type) {case HTP_TYPE_F32: return ne00 * 4;@@ -23,6 +131,7 @@ static inline uint32_t htp_tensor_get_row_size(int type, uint32_t ne00) {}struct htp_context;+void htp_flush_dirty_ranges(struct htp_context * ctx);void htp_tensor_flush_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n);void htp_tensor_dirty_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n);diff --git a/ggml/src/ggml-hexagon/htp/hvx-arith.h b/ggml/src/ggml-hexagon/htp/hvx-arith.hindex fe5477c1be..6cbead74c7 100644--- a/ggml/src/ggml-hexagon/htp/hvx-arith.h+++ b/ggml/src/ggml-hexagon/htp/hvx-arith.h@@ -16,25 +16,25 @@#define UNUSED(x) (void)(x)#define hvx_arith_loop_body(dst_type, src0_type, src1_type, elem_size, vec_store, vec_op) \- do { \- dst_type * vdst = (dst_type *) dst; \- src0_type * vsrc0 = (src0_type *) src0; \- src1_type * vsrc1 = (src1_type *) src1; \- \- const uint32_t epv = 128 / (elem_size); \- const uint32_t nvec = n / epv; \- const uint32_t nloe = n % epv; \- \- uint32_t i = 0; \- \- _Pragma("unroll(4)") \- for (; i < nvec; i++) { \- vdst[i] = vec_op(vsrc0[i], vsrc1[i]); \- } \- if (nloe) { \- HVX_Vector v = vec_op(vsrc0[i], vsrc1[i]); \- vec_store((void *) &vdst[i], nloe * (elem_size), v); \- } \+ do { \+ dst_type * vdst = (dst_type *) dst; \+ src0_type * vsrc0 = (src0_type *) src0; \+ src1_type * vsrc1 = (src1_type *) src1; \+ \+ const uint32_t epv = 128 / (elem_size); \+ const uint32_t nvec = n / epv; \+ const uint32_t nloe = n % epv; \+ \+ uint32_t i = 0; \+ \+ _Pragma("unroll(4)") \+ for (; i < nvec; i++) { \+ vdst[i] = vec_op(vsrc0[i], vsrc1[i]); \+ } \+ if (nloe) { \+ HVX_Vector v = vec_op(vsrc0[i], vsrc1[i]); \+ vec_store((void *) &vdst[i], nloe * (elem_size), v); \+ } \} while(0)#if __HVX_ARCH__ < 79@@ -56,43 +56,43 @@#define HVX_OP_MUL_F16(a, b) hvx_vec_mul_f16_f16(a, b)// Generic macro to define alignment permutations for an op-#define DEFINE_HVX_BINARY_OP_VARIANTS(OP_NAME, OP_MACRO, ELEM_TYPE) \-static inline void OP_NAME##_aaa(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \- assert((uintptr_t) dst % 128 == 0); \- assert((uintptr_t) src0 % 128 == 0); \- assert((uintptr_t) src1 % 128 == 0); \- hvx_arith_loop_body(HVX_Vector, HVX_Vector, HVX_Vector, sizeof(ELEM_TYPE), hvx_vec_store_a, OP_MACRO); \-} \-static inline void OP_NAME##_aau(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \- assert((uintptr_t) dst % 128 == 0); \- assert((uintptr_t) src0 % 128 == 0); \- hvx_arith_loop_body(HVX_Vector, HVX_Vector, HVX_UVector, sizeof(ELEM_TYPE), hvx_vec_store_a, OP_MACRO); \-} \-static inline void OP_NAME##_aua(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \- assert((uintptr_t) dst % 128 == 0); \- assert((uintptr_t) src1 % 128 == 0); \- hvx_arith_loop_body(HVX_Vector, HVX_UVector, HVX_Vector, sizeof(ELEM_TYPE), hvx_vec_store_a, OP_MACRO); \-} \-static inline void OP_NAME##_auu(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \- assert((uintptr_t) dst % 128 == 0); \- hvx_arith_loop_body(HVX_Vector, HVX_UVector, HVX_UVector, sizeof(ELEM_TYPE), hvx_vec_store_a, OP_MACRO); \-} \-static inline void OP_NAME##_uaa(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \- assert((uintptr_t) src0 % 128 == 0); \- assert((uintptr_t) src1 % 128 == 0); \- hvx_arith_loop_body(HVX_UVector, HVX_Vector, HVX_Vector, sizeof(ELEM_TYPE), hvx_vec_store_u, OP_MACRO); \-} \-static inline void OP_NAME##_uau(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \- assert((uintptr_t) src0 % 128 == 0); \- hvx_arith_loop_body(HVX_UVector, HVX_Vector, HVX_UVector, sizeof(ELEM_TYPE), hvx_vec_store_u, OP_MACRO); \-} \-static inline void OP_NAME##_uua(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \- assert((uintptr_t) src1 % 128 == 0); \- hvx_arith_loop_body(HVX_UVector, HVX_UVector, HVX_Vector, sizeof(ELEM_TYPE), hvx_vec_store_u, OP_MACRO); \-} \-static inline void OP_NAME##_uuu(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \+#define DEFINE_HVX_BINARY_OP_VARIANTS(OP_NAME, OP_MACRO, ELEM_TYPE) \+static inline void OP_NAME##_aaa(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \+ assert((uintptr_t) dst % 128 == 0); \+ assert((uintptr_t) src0 % 128 == 0); \+ assert((uintptr_t) src1 % 128 == 0); \+ hvx_arith_loop_body(HVX_Vector, HVX_Vector, HVX_Vector, sizeof(ELEM_TYPE), hvx_vec_store_a, OP_MACRO); \+} \+static inline void OP_NAME##_aau(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \+ assert((uintptr_t) dst % 128 == 0); \+ assert((uintptr_t) src0 % 128 == 0); \+ hvx_arith_loop_body(HVX_Vector, HVX_Vector, HVX_UVector, sizeof(ELEM_TYPE), hvx_vec_store_a, OP_MACRO); \+} \+static inline void OP_NAME##_aua(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \+ assert((uintptr_t) dst % 128 == 0); \+ assert((uintptr_t) src1 % 128 == 0); \+ hvx_arith_loop_body(HVX_Vector, HVX_UVector, HVX_Vector, sizeof(ELEM_TYPE), hvx_vec_store_a, OP_MACRO); \+} \+static inline void OP_NAME##_auu(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \+ assert((uintptr_t) dst % 128 == 0); \+ hvx_arith_loop_body(HVX_Vector, HVX_UVector, HVX_UVector, sizeof(ELEM_TYPE), hvx_vec_store_a, OP_MACRO); \+} \+static inline void OP_NAME##_uaa(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \+ assert((uintptr_t) src0 % 128 == 0); \+ assert((uintptr_t) src1 % 128 == 0); \+ hvx_arith_loop_body(HVX_UVector, HVX_Vector, HVX_Vector, sizeof(ELEM_TYPE), hvx_vec_store_u, OP_MACRO); \+} \+static inline void OP_NAME##_uau(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \+ assert((uintptr_t) src0 % 128 == 0); \+ hvx_arith_loop_body(HVX_UVector, HVX_Vector, HVX_UVector, sizeof(ELEM_TYPE), hvx_vec_store_u, OP_MACRO); \+} \+static inline void OP_NAME##_uua(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \+ assert((uintptr_t) src1 % 128 == 0); \+ hvx_arith_loop_body(HVX_UVector, HVX_UVector, HVX_Vector, sizeof(ELEM_TYPE), hvx_vec_store_u, OP_MACRO); \+} \+static inline void OP_NAME##_uuu(uint8_t * dst, const uint8_t * src0, const uint8_t * src1, uint32_t n) { \hvx_arith_loop_body(HVX_UVector, HVX_UVector, HVX_UVector, sizeof(ELEM_TYPE), hvx_vec_store_u, OP_MACRO); \-} \+} \DEFINE_HVX_BINARY_OP_VARIANTS(hvx_add_f32, HVX_OP_ADD_F32, float)DEFINE_HVX_BINARY_OP_VARIANTS(hvx_sub_f32, HVX_OP_SUB_F32, float)@@ -103,25 +103,25 @@ DEFINE_HVX_BINARY_OP_VARIANTS(hvx_sub_f16, HVX_OP_SUB_F16, _Float16)DEFINE_HVX_BINARY_OP_VARIANTS(hvx_mul_f16, HVX_OP_MUL_F16, _Float16)// Dispatcher logic-#define HVX_BINARY_DISPATCHER(OP_NAME) \+#define HVX_BINARY_DISPATCHER(OP_NAME) \static inline void OP_NAME(uint8_t * restrict dst, const uint8_t * restrict src0, const uint8_t * restrict src1, const uint32_t num_elems) { \- if (hex_is_aligned((void *) dst, 128)) { \- if (hex_is_aligned((void *) src0, 128)) { \- if (hex_is_aligned((void *) src1, 128)) OP_NAME##_aaa(dst, src0, src1, num_elems); \- else OP_NAME##_aau(dst, src0, src1, num_elems); \- } else { \- if (hex_is_aligned((void *) src1, 128)) OP_NAME##_aua(dst, src0, src1, num_elems); \- else OP_NAME##_auu(dst, src0, src1, num_elems); \- } \- } else { \- if (hex_is_aligned((void *) src0, 128)) { \- if (hex_is_aligned((void *) src1, 128)) OP_NAME##_uaa(dst, src0, src1, num_elems); \- else OP_NAME##_uau(dst, src0, src1, num_elems); \- } else { \- if (hex_is_aligned((void *) src1, 128)) OP_NAME##_uua(dst, src0, src1, num_elems); \- else OP_NAME##_uuu(dst, src0, src1, num_elems); \- } \- } \+ if (hex_is_aligned((void *) dst, 128)) { \+ if (hex_is_aligned((void *) src0, 128)) { \+ if (hex_is_aligned((void *) src1, 128)) OP_NAME##_aaa(dst, src0, src1, num_elems); \+ else OP_NAME##_aau(dst, src0, src1, num_elems); \+ } else { \+ if (hex_is_aligned((void *) src1, 128)) OP_NAME##_aua(dst, src0, src1, num_elems); \+ else OP_NAME##_auu(dst, src0, src1, num_elems); \+ } \+ } else { \+ if (hex_is_aligned((void *) src0, 128)) { \+ if (hex_is_aligned((void *) src1, 128)) OP_NAME##_uaa(dst, src0, src1, num_elems); \+ else OP_NAME##_uau(dst, src0, src1, num_elems); \+ } else { \+ if (hex_is_aligned((void *) src1, 128)) OP_NAME##_uua(dst, src0, src1, num_elems); \+ else OP_NAME##_uuu(dst, src0, src1, num_elems); \+ } \+ } \}HVX_BINARY_DISPATCHER(hvx_add_f32)@@ -166,44 +166,44 @@ static inline void hvx_mul_mul_f32_aa(uint8_t * restrict dst, const uint8_t * re// Scalar Operations-#define hvx_scalar_loop_body(dst_type, src_type, elem_size, vec_store, scalar_op_macro) \- do { \- dst_type * restrict vdst = (dst_type *) dst; \- src_type * restrict vsrc = (src_type *) src; \- \- const uint32_t epv = 128 / (elem_size); \- const uint32_t nvec = n / epv; \- const uint32_t nloe = n % epv; \- \- uint32_t i = 0; \- \- _Pragma("unroll(4)") \- for (; i < nvec; i++) { \- HVX_Vector v = vsrc[i]; \- vdst[i] = scalar_op_macro(v); \- } \- if (nloe) { \- HVX_Vector v = vsrc[i]; \- v = scalar_op_macro(v); \- vec_store((void *) &vdst[i], nloe * (elem_size), v); \- } \+#define hvx_scalar_loop_body(dst_type, src_type, elem_size, vec_store, scalar_op_macro) \+ do { \+ dst_type * restrict vdst = (dst_type *) dst; \+ src_type * restrict vsrc = (src_type *) src; \+ \+ const uint32_t epv = 128 / (elem_size); \+ const uint32_t nvec = n / epv; \+ const uint32_t nloe = n % epv; \+ \+ uint32_t i = 0; \+ \+ _Pragma("unroll(4)") \+ for (; i < nvec; i++) { \+ HVX_Vector v = vsrc[i]; \+ vdst[i] = scalar_op_macro(v); \+ } \+ if (nloe) { \+ HVX_Vector v = vsrc[i]; \+ v = scalar_op_macro(v); \+ vec_store((void *) &vdst[i], nloe * (elem_size), v); \+ } \} while(0)-#define HVX_OP_ADD_SCALAR_F32(v) \- ({ \+#define HVX_OP_ADD_SCALAR_F32(v) \+ ({ \const HVX_VectorPred pred_inf = Q6_Q_vcmp_eq_VwVw(inf, v); \- HVX_Vector out = HVX_OP_ADD_F32(v, val_vec); \- Q6_V_vmux_QVV(pred_inf, inf, out); \+ HVX_Vector out = HVX_OP_ADD_F32(v, val_vec); \+ Q6_V_vmux_QVV(pred_inf, inf, out); \})#define HVX_OP_MUL_SCALAR_F32(v) HVX_OP_MUL_F32(v, val_vec)#define HVX_OP_SUB_SCALAR_F32(v) HVX_OP_SUB_F32(v, val_vec)-#define HVX_OP_ADD_SCALAR_F16(v) \- ({ \+#define HVX_OP_ADD_SCALAR_F16(v) \+ ({ \const HVX_VectorPred pred_inf = Q6_Q_vcmp_eq_VhVh(inf, v); \- HVX_Vector out = HVX_OP_ADD_F16(v, val_vec); \- Q6_V_vmux_QVV(pred_inf, inf, out); \+ HVX_Vector out = HVX_OP_ADD_F16(v, val_vec); \+ Q6_V_vmux_QVV(pred_inf, inf, out); \})#define HVX_OP_MUL_SCALAR_F16(v) HVX_OP_MUL_F16(v, val_vec)@@ -212,31 +212,31 @@ static inline void hvx_mul_mul_f32_aa(uint8_t * restrict dst, const uint8_t * re// Scalar Variants// Generic macro to define alignment permutations for an op-#define DEFINE_HVX_BINARY_SCALAR_OP_VARIANTS(OP_NAME, OP_MACRO, SPLAT_MACRO, ELEM_TYPE) \+#define DEFINE_HVX_BINARY_SCALAR_OP_VARIANTS(OP_NAME, OP_MACRO, SPLAT_MACRO, ELEM_TYPE) \static inline void OP_NAME##_aa(uint8_t * restrict dst, const uint8_t * restrict src, const ELEM_TYPE val, uint32_t n) { \- const HVX_Vector val_vec = SPLAT_MACRO(val); \- const HVX_Vector inf = SPLAT_MACRO((ELEM_TYPE)INFINITY); UNUSED(inf); \- assert((uintptr_t) dst % 128 == 0); \- assert((uintptr_t) src % 128 == 0); \- hvx_scalar_loop_body(HVX_Vector, HVX_Vector, sizeof(ELEM_TYPE), hvx_vec_store_a, OP_MACRO); \-} \+ const HVX_Vector val_vec = SPLAT_MACRO(val); \+ const HVX_Vector inf = SPLAT_MACRO((ELEM_TYPE)INFINITY); UNUSED(inf); \+ assert((uintptr_t) dst % 128 == 0); \+ assert((uintptr_t) src % 128 == 0); \+ hvx_scalar_loop_body(HVX_Vector, HVX_Vector, sizeof(ELEM_TYPE), hvx_vec_store_a, OP_MACRO); \+} \static inline void OP_NAME##_au(uint8_t * restrict dst, const uint8_t * restrict src, const ELEM_TYPE val, uint32_t n) { \- const HVX_Vector val_vec = SPLAT_MACRO(val); \- const HVX_Vector inf = SPLAT_MACRO((ELEM_TYPE)INFINITY); UNUSED(inf); \- assert((uintptr_t) dst % 128 == 0); \- hvx_scalar_loop_body(HVX_Vector, HVX_UVector, sizeof(ELEM_TYPE), hvx_vec_store_a, OP_MACRO); \-} \+ const HVX_Vector val_vec = SPLAT_MACRO(val); \+ const HVX_Vector inf = SPLAT_MACRO((ELEM_TYPE)INFINITY); UNUSED(inf); \+ assert((uintptr_t) dst % 128 == 0); \+ hvx_scalar_loop_body(HVX_Vector, HVX_UVector, sizeof(ELEM_TYPE), hvx_vec_store_a, OP_MACRO); \+} \static inline void OP_NAME##_ua(uint8_t * restrict dst, const uint8_t * restrict src, const ELEM_TYPE val, uint32_t n) { \- const HVX_Vector val_vec = SPLAT_MACRO(val); \- const HVX_Vector inf = SPLAT_MACRO((ELEM_TYPE)INFINITY); UNUSED(inf); \- assert((uintptr_t) src % 128 == 0); \- hvx_scalar_loop_body(HVX_UVector, HVX_Vector, sizeof(ELEM_TYPE), hvx_vec_store_u, OP_MACRO); \-} \+ const HVX_Vector val_vec = SPLAT_MACRO(val); \+ const HVX_Vector inf = SPLAT_MACRO((ELEM_TYPE)INFINITY); UNUSED(inf); \+ assert((uintptr_t) src % 128 == 0); \+ hvx_scalar_loop_body(HVX_UVector, HVX_Vector, sizeof(ELEM_TYPE), hvx_vec_store_u, OP_MACRO); \+} \static inline void OP_NAME##_uu(uint8_t * restrict dst, const uint8_t * restrict src, const ELEM_TYPE val, uint32_t n) { \- const HVX_Vector val_vec = SPLAT_MACRO(val); \- const HVX_Vector inf = SPLAT_MACRO((ELEM_TYPE)INFINITY); UNUSED(inf); \- hvx_scalar_loop_body(HVX_UVector, HVX_UVector, sizeof(ELEM_TYPE), hvx_vec_store_u, OP_MACRO); \-} \+ const HVX_Vector val_vec = SPLAT_MACRO(val); \+ const HVX_Vector inf = SPLAT_MACRO((ELEM_TYPE)INFINITY); UNUSED(inf); \+ hvx_scalar_loop_body(HVX_UVector, HVX_UVector, sizeof(ELEM_TYPE), hvx_vec_store_u, OP_MACRO); \+} \DEFINE_HVX_BINARY_SCALAR_OP_VARIANTS(hvx_add_scalar_f32, HVX_OP_ADD_SCALAR_F32, hvx_vec_splat_f32, float)DEFINE_HVX_BINARY_SCALAR_OP_VARIANTS(hvx_sub_scalar_f32, HVX_OP_SUB_SCALAR_F32, hvx_vec_splat_f32, float)@@ -247,17 +247,17 @@ DEFINE_HVX_BINARY_SCALAR_OP_VARIANTS(hvx_sub_scalar_f16, HVX_OP_SUB_SCALAR_F16,DEFINE_HVX_BINARY_SCALAR_OP_VARIANTS(hvx_mul_scalar_f16, HVX_OP_MUL_SCALAR_F16, hvx_vec_splat_f16, _Float16)// Dispatcher logic-#define HVX_BINARY_SCALAR_DISPATCHER(OP_NAME, ELEM_TYPE) \+#define HVX_BINARY_SCALAR_DISPATCHER(OP_NAME, ELEM_TYPE) \static inline void OP_NAME(uint8_t * restrict dst, const uint8_t * restrict src, const ELEM_TYPE val, const uint32_t num_elems) { \- if (hex_is_aligned((void *) dst, 128) && hex_is_aligned((void *) src, 128)) { \- OP_NAME##_aa(dst, src, val, num_elems); \- } else if (hex_is_aligned((void *) dst, 128)) { \- OP_NAME##_au(dst, src, val, num_elems); \- } else if (hex_is_aligned((void *) src, 128)) { \- OP_NAME##_ua(dst, src, val, num_elems); \- } else { \- OP_NAME##_uu(dst, src, val, num_elems); \- } \+ if (hex_is_aligned((void *) dst, 128) && hex_is_aligned((void *) src, 128)) { \+ OP_NAME##_aa(dst, src, val, num_elems); \+ } else if (hex_is_aligned((void *) dst, 128)) { \+ OP_NAME##_au(dst, src, val, num_elems); \+ } else if (hex_is_aligned((void *) src, 128)) { \+ OP_NAME##_ua(dst, src, val, num_elems); \+ } else { \+ OP_NAME##_uu(dst, src, val, num_elems); \+ } \}HVX_BINARY_SCALAR_DISPATCHER(hvx_add_scalar_f32, float)@@ -350,12 +350,12 @@ static inline void hvx_max_scalar_f32(uint8_t * restrict dst, const uint8_t * re// CLAMP Scalar variants-#define HVX_OP_CLAMP_SCALAR(v) \- ({ \+#define HVX_OP_CLAMP_SCALAR(v) \+ ({ \HVX_VectorPred pred_cap_right = Q6_Q_vcmp_gt_VsfVsf(v, max_vec); \HVX_VectorPred pred_cap_left = Q6_Q_vcmp_gt_VsfVsf(min_vec, v); \- HVX_Vector tmp = Q6_V_vmux_QVV(pred_cap_right, max_vec, v); \- Q6_V_vmux_QVV(pred_cap_left, min_vec, tmp); \+ HVX_Vector tmp = Q6_V_vmux_QVV(pred_cap_right, max_vec, v); \+ Q6_V_vmux_QVV(pred_cap_left, min_vec, tmp); \})static inline void hvx_clamp_scalar_f32_aa(uint8_t * restrict dst, const uint8_t * restrict src, const float min, const float max, uint32_t n) {diff --git a/ggml/src/ggml-hexagon/htp/hvx-div.h b/ggml/src/ggml-hexagon/htp/hvx-div.hindex 53ee304e74..bb7ab0519d 100644--- a/ggml/src/ggml-hexagon/htp/hvx-div.h+++ b/ggml/src/ggml-hexagon/htp/hvx-div.h@@ -219,64 +219,64 @@ static inline HVX_Vector hvx_vec_hybrid_div_f16(HVX_Vector vec1, HVX_Vector vec2} while(0)// Generic macro to define alignment permutations for an op-#define DEFINE_HVX_DIV_OP_VARIANTS(OP_NAME, OP_LOOP_BODY) \+#define DEFINE_HVX_DIV_OP_VARIANTS(OP_NAME, OP_LOOP_BODY) \static inline void OP_NAME##_aaa(uint8_t * restrict dst, const uint8_t * restrict src0, const uint8_t * restrict src1, uint32_t n) { \- assert((uintptr_t) dst % 128 == 0); \- assert((uintptr_t) src0 % 128 == 0); \- assert((uintptr_t) src1 % 128 == 0); \- OP_LOOP_BODY(HVX_Vector, HVX_Vector, HVX_Vector, hvx_vec_store_a); \-} \+ assert((uintptr_t) dst % 128 == 0); \+ assert((uintptr_t) src0 % 128 == 0); \+ assert((uintptr_t) src1 % 128 == 0); \+ OP_LOOP_BODY(HVX_Vector, HVX_Vector, HVX_Vector, hvx_vec_store_a); \+} \static inline void OP_NAME##_aau(uint8_t * restrict dst, const uint8_t * restrict src0, const uint8_t * restrict src1, uint32_t n) { \- assert((uintptr_t) dst % 128 == 0); \- assert((uintptr_t) src0 % 128 == 0); \- OP_LOOP_BODY(HVX_Vector, HVX_Vector, HVX_UVector, hvx_vec_store_a); \-} \+ assert((uintptr_t) dst % 128 == 0); \+ assert((uintptr_t) src0 % 128 == 0); \+ OP_LOOP_BODY(HVX_Vector, HVX_Vector, HVX_UVector, hvx_vec_store_a); \+} \static inline void OP_NAME##_aua(uint8_t * restrict dst, const uint8_t * restrict src0, const uint8_t * restrict src1, uint32_t n) { \- assert((uintptr_t) dst % 128 == 0); \- assert((uintptr_t) src1 % 128 == 0); \- OP_LOOP_BODY(HVX_Vector, HVX_UVector, HVX_Vector, hvx_vec_store_a); \-} \+ assert((uintptr_t) dst % 128 == 0); \+ assert((uintptr_t) src1 % 128 == 0); \+ OP_LOOP_BODY(HVX_Vector, HVX_UVector, HVX_Vector, hvx_vec_store_a); \+} \static inline void OP_NAME##_auu(uint8_t * restrict dst, const uint8_t * restrict src0, const uint8_t * restrict src1, uint32_t n) { \- assert((uintptr_t) dst % 128 == 0); \- OP_LOOP_BODY(HVX_Vector, HVX_UVector, HVX_UVector, hvx_vec_store_a); \-} \+ assert((uintptr_t) dst % 128 == 0); \+ OP_LOOP_BODY(HVX_Vector, HVX_UVector, HVX_UVector, hvx_vec_store_a); \+} \static inline void OP_NAME##_uaa(uint8_t * restrict dst, const uint8_t * restrict src0, const uint8_t * restrict src1, uint32_t n) { \- assert((uintptr_t) src0 % 128 == 0); \- assert((uintptr_t) src1 % 128 == 0); \- OP_LOOP_BODY(HVX_UVector, HVX_Vector, HVX_Vector, hvx_vec_store_u); \-} \+ assert((uintptr_t) src0 % 128 == 0); \+ assert((uintptr_t) src1 % 128 == 0); \+ OP_LOOP_BODY(HVX_UVector, HVX_Vector, HVX_Vector, hvx_vec_store_u); \+} \static inline void OP_NAME##_uau(uint8_t * restrict dst, const uint8_t * restrict src0, const uint8_t * restrict src1, uint32_t n) { \- assert((uintptr_t) src0 % 128 == 0); \- OP_LOOP_BODY(HVX_UVector, HVX_Vector, HVX_UVector, hvx_vec_store_u); \-} \+ assert((uintptr_t) src0 % 128 == 0); \+ OP_LOOP_BODY(HVX_UVector, HVX_Vector, HVX_UVector, hvx_vec_store_u); \+} \static inline void OP_NAME##_uua(uint8_t * restrict dst, const uint8_t * restrict src0, const uint8_t * restrict src1, uint32_t n) { \- assert((uintptr_t) src1 % 128 == 0); \- OP_LOOP_BODY(HVX_UVector, HVX_UVector, HVX_Vector, hvx_vec_store_u); \-} \+ assert((uintptr_t) src1 % 128 == 0); \+ OP_LOOP_BODY(HVX_UVector, HVX_UVector, HVX_Vector, hvx_vec_store_u); \+} \static inline void OP_NAME##_uuu(uint8_t * restrict dst, const uint8_t * restrict src0, const uint8_t * restrict src1, uint32_t n) { \- OP_LOOP_BODY(HVX_UVector, HVX_UVector, HVX_UVector, hvx_vec_store_u); \-} \+ OP_LOOP_BODY(HVX_UVector, HVX_UVector, HVX_UVector, hvx_vec_store_u); \+} \// Dispatcher logic-#define HVX_DIV_DISPATCHER(OP_NAME) \+#define HVX_DIV_DISPATCHER(OP_NAME) \static inline void OP_NAME(uint8_t * restrict dst, const uint8_t * restrict src0, const uint8_t * restrict src1, const uint32_t num_elems) { \- if (hex_is_aligned((void *) dst, 128)) { \- if (hex_is_aligned((void *) src0, 128)) { \- if (hex_is_aligned((void *) src1, 128)) OP_NAME##_aaa(dst, src0, src1, num_elems); \- else OP_NAME##_aau(dst, src0, src1, num_elems); \- } else { \- if (hex_is_aligned((void *) src1, 128)) OP_NAME##_aua(dst, src0, src1, num_elems); \- else OP_NAME##_auu(dst, src0, src1, num_elems); \- } \- } else { \- if (hex_is_aligned((void *) src0, 128)) { \- if (hex_is_aligned((void *) src1, 128)) OP_NAME##_uaa(dst, src0, src1, num_elems); \- else OP_NAME##_uau(dst, src0, src1, num_elems); \- } else { \- if (hex_is_aligned((void *) src1, 128)) OP_NAME##_uua(dst, src0, src1, num_elems); \- else OP_NAME##_uuu(dst, src0, src1, num_elems); \- } \- } \+ if (hex_is_aligned((void *) dst, 128)) { \+ if (hex_is_aligned((void *) src0, 128)) { \+ if (hex_is_aligned((void *) src1, 128)) OP_NAME##_aaa(dst, src0, src1, num_elems); \+ else OP_NAME##_aau(dst, src0, src1, num_elems); \+ } else { \+ if (hex_is_aligned((void *) src1, 128)) OP_NAME##_aua(dst, src0, src1, num_elems); \+ else OP_NAME##_auu(dst, src0, src1, num_elems); \+ } \+ } else { \+ if (hex_is_aligned((void *) src0, 128)) { \+ if (hex_is_aligned((void *) src1, 128)) OP_NAME##_uaa(dst, src0, src1, num_elems); \+ else OP_NAME##_uau(dst, src0, src1, num_elems); \+ } else { \+ if (hex_is_aligned((void *) src1, 128)) OP_NAME##_uua(dst, src0, src1, num_elems); \+ else OP_NAME##_uuu(dst, src0, src1, num_elems); \+ } \+ } \}DEFINE_HVX_DIV_OP_VARIANTS(hvx_div_f32, hvx_div_f32_loop_body)diff --git a/ggml/src/ggml-hexagon/htp/hvx-inverse.h b/ggml/src/ggml-hexagon/htp/hvx-inverse.hindex f2054f45ba..256a8843ba 100644--- a/ggml/src/ggml-hexagon/htp/hvx-inverse.h+++ b/ggml/src/ggml-hexagon/htp/hvx-inverse.h@@ -169,36 +169,36 @@ static inline HVX_Vector hvx_vec_inverse_f16_guard(HVX_Vector v_sf, HVX_Vector n} while(0)// Generic macro to define alignment permutations for an op-#define DEFINE_HVX_INV_OP_VARIANTS(OP_NAME, OP_LOOP_BODY) \+#define DEFINE_HVX_INV_OP_VARIANTS(OP_NAME, OP_LOOP_BODY) \static inline void OP_NAME##_aa(uint8_t * restrict dst, const uint8_t * restrict src, uint32_t n) { \- assert((uintptr_t) dst % 128 == 0); \- assert((uintptr_t) src % 128 == 0); \- OP_LOOP_BODY(HVX_Vector, HVX_Vector, hvx_vec_store_a); \-} \+ assert((uintptr_t) dst % 128 == 0); \+ assert((uintptr_t) src % 128 == 0); \+ OP_LOOP_BODY(HVX_Vector, HVX_Vector, hvx_vec_store_a); \+} \static inline void OP_NAME##_au(uint8_t * restrict dst, const uint8_t * restrict src, uint32_t n) { \- assert((uintptr_t) dst % 128 == 0); \- OP_LOOP_BODY(HVX_Vector, HVX_UVector, hvx_vec_store_a); \-} \+ assert((uintptr_t) dst % 128 == 0); \+ OP_LOOP_BODY(HVX_Vector, HVX_UVector, hvx_vec_store_a); \+} \static inline void OP_NAME##_ua(uint8_t * restrict dst, const uint8_t * restrict src, uint32_t n) { \- assert((uintptr_t) src % 128 == 0); \- OP_LOOP_BODY(HVX_UVector, HVX_Vector, hvx_vec_store_u); \-} \+ assert((uintptr_t) src % 128 == 0); \+ OP_LOOP_BODY(HVX_UVector, HVX_Vector, hvx_vec_store_u); \+} \static inline void OP_NAME##_uu(uint8_t * restrict dst, const uint8_t * restrict src, uint32_t n) { \- OP_LOOP_BODY(HVX_UVector, HVX_UVector, hvx_vec_store_u); \-} \+ OP_LOOP_BODY(HVX_UVector, HVX_UVector, hvx_vec_store_u); \+} \// Dispatcher logic-#define HVX_INV_DISPATCHER(OP_NAME) \+#define HVX_INV_DISPATCHER(OP_NAME) \static inline void OP_NAME(uint8_t * restrict dst, const uint8_t * restrict src, const uint32_t num_elems) { \- if (hex_is_aligned((void *) dst, 128) && hex_is_aligned((void *) src, 128)) { \- OP_NAME##_aa(dst, src, num_elems); \- } else if (hex_is_aligned((void *) dst, 128)) { \- OP_NAME##_au(dst, src, num_elems); \- } else if (hex_is_aligned((void *) src, 128)) { \- OP_NAME##_ua(dst, src, num_elems); \- } else { \- OP_NAME##_uu(dst, src, num_elems); \- } \+ if (hex_is_aligned((void *) dst, 128) && hex_is_aligned((void *) src, 128)) { \+ OP_NAME##_aa(dst, src, num_elems); \+ } else if (hex_is_aligned((void *) dst, 128)) { \+ OP_NAME##_au(dst, src, num_elems); \+ } else if (hex_is_aligned((void *) src, 128)) { \+ OP_NAME##_ua(dst, src, num_elems); \+ } else { \+ OP_NAME##_uu(dst, src, num_elems); \+ } \}DEFINE_HVX_INV_OP_VARIANTS(hvx_inverse_f32, hvx_inverse_f32_loop_body)diff --git a/ggml/src/ggml-hexagon/htp/hvx-scale.h b/ggml/src/ggml-hexagon/htp/hvx-scale.hindex 9b1a28f529..5d0650307e 100644--- a/ggml/src/ggml-hexagon/htp/hvx-scale.h+++ b/ggml/src/ggml-hexagon/htp/hvx-scale.h@@ -68,30 +68,30 @@ static inline void hvx_scale_f32(uint8_t * restrict dst, const uint8_t * restric}}-#define hvx_scale_offset_f32_loop_body(dst_type, src_type, vec_store) \- do { \- dst_type * restrict vdst = (dst_type *) dst; \- src_type * restrict vsrc = (src_type *) src; \- \- HVX_Vector vs = hvx_vec_splat_f32(scale); \- HVX_Vector vo = hvx_vec_splat_f32(offset); \- \- const uint32_t elem_size = sizeof(float); \- const uint32_t epv = 128 / elem_size; \- const uint32_t nvec = n / epv; \- const uint32_t nloe = n % epv; \- \- uint32_t i = 0; \- \- _Pragma("unroll(4)") \- for (; i < nvec; ++i) { \+#define hvx_scale_offset_f32_loop_body(dst_type, src_type, vec_store) \+ do { \+ dst_type * restrict vdst = (dst_type *) dst; \+ src_type * restrict vsrc = (src_type *) src; \+ \+ HVX_Vector vs = hvx_vec_splat_f32(scale); \+ HVX_Vector vo = hvx_vec_splat_f32(offset); \+ \+ const uint32_t elem_size = sizeof(float); \+ const uint32_t epv = 128 / elem_size; \+ const uint32_t nvec = n / epv; \+ const uint32_t nloe = n % epv; \+ \+ uint32_t i = 0; \+ \+ _Pragma("unroll(4)") \+ for (; i < nvec; ++i) { \HVX_Vector v = Q6_Vqf32_vadd_Vqf32Vsf(Q6_Vqf32_vmpy_VsfVsf(vsrc[i], vs), vo); \- vdst[i] = Q6_Vsf_equals_Vqf32(v); \- } \- if (nloe) { \+ vdst[i] = Q6_Vsf_equals_Vqf32(v); \+ } \+ if (nloe) { \HVX_Vector v = Q6_Vqf32_vadd_Vqf32Vsf(Q6_Vqf32_vmpy_VsfVsf(vsrc[i], vs), vo); \- vec_store((void *) &vdst[i], nloe * elem_size, Q6_Vsf_equals_Vqf32(v)); \- } \+ vec_store((void *) &vdst[i], nloe * elem_size, Q6_Vsf_equals_Vqf32(v)); \+ } \} while(0)static inline void hvx_scale_offset_f32_aa(uint8_t * restrict dst, const uint8_t * restrict src, const int n, const float scale, const float offset) {diff --git a/ggml/src/ggml-hexagon/htp/hvx-sigmoid.h b/ggml/src/ggml-hexagon/htp/hvx-sigmoid.hindex dd66dd84c9..552017309d 100644--- a/ggml/src/ggml-hexagon/htp/hvx-sigmoid.h+++ b/ggml/src/ggml-hexagon/htp/hvx-sigmoid.h@@ -68,50 +68,50 @@ static inline HVX_Vector hvx_vec_tanh_f32(HVX_Vector x) {return Q6_Vsf_equals_Vqf32(res);}-#define hvx_sigmoid_loop_body(dst_type, src_type, vec_store) \- do { \- dst_type * restrict vdst = (dst_type *) dst; \- src_type * restrict vsrc = (src_type *) src; \- \- const HVX_Vector one = hvx_vec_splat_f32(1.f); \- const HVX_Vector max_exp = hvx_vec_splat_f32(87.f); \- const HVX_Vector min_exp = hvx_vec_splat_f32(-87.f); \- \- const uint32_t epv = 128 / sizeof(float); \- const uint32_t nvec = n / epv; \- const uint32_t nloe = n % epv; \- \- uint32_t i = 0; \- \- _Pragma("unroll(4)") \- for (; i < nvec; i++) { \- vdst[i] = hvx_vec_fast_sigmoid_f32_guard(vsrc[i], one, max_exp, min_exp); \- } \- if (nloe) { \+#define hvx_sigmoid_loop_body(dst_type, src_type, vec_store) \+ do { \+ dst_type * restrict vdst = (dst_type *) dst; \+ src_type * restrict vsrc = (src_type *) src; \+ \+ const HVX_Vector one = hvx_vec_splat_f32(1.f); \+ const HVX_Vector max_exp = hvx_vec_splat_f32(87.f); \+ const HVX_Vector min_exp = hvx_vec_splat_f32(-87.f); \+ \+ const uint32_t epv = 128 / sizeof(float); \+ const uint32_t nvec = n / epv; \+ const uint32_t nloe = n % epv; \+ \+ uint32_t i = 0; \+ \+ _Pragma("unroll(4)") \+ for (; i < nvec; i++) { \+ vdst[i] = hvx_vec_fast_sigmoid_f32_guard(vsrc[i], one, max_exp, min_exp); \+ } \+ if (nloe) { \HVX_Vector tmp = hvx_vec_fast_sigmoid_f32_guard(vsrc[i], one, max_exp, min_exp); \- vec_store((void *) &vdst[i], nloe * sizeof(float), tmp); \- } \+ vec_store((void *) &vdst[i], nloe * sizeof(float), tmp); \+ } \} while(0)-#define hvx_tanh_loop_body(dst_type, src_type, vec_store) \- do { \- dst_type * restrict vdst = (dst_type *) dst; \- src_type * restrict vsrc = (src_type *) src; \- \- const uint32_t epv = 128 / sizeof(float); \- const uint32_t nvec = n / epv; \- const uint32_t nloe = n % epv; \- \- uint32_t i = 0; \- \- _Pragma("unroll(4)") \- for (; i < nvec; i++) { \- vdst[i] = hvx_vec_tanh_f32(vsrc[i]); \- } \- if (nloe) { \- HVX_Vector tmp = hvx_vec_tanh_f32(vsrc[i]); \+#define hvx_tanh_loop_body(dst_type, src_type, vec_store) \+ do { \+ dst_type * restrict vdst = (dst_type *) dst; \+ src_type * restrict vsrc = (src_type *) src; \+ \+ const uint32_t epv = 128 / sizeof(float); \+ const uint32_t nvec = n / epv; \+ const uint32_t nloe = n % epv; \+ \+ uint32_t i = 0; \+ \+ _Pragma("unroll(4)") \+ for (; i < nvec; i++) { \+ vdst[i] = hvx_vec_tanh_f32(vsrc[i]); \+ } \+ if (nloe) { \+ HVX_Vector tmp = hvx_vec_tanh_f32(vsrc[i]); \vec_store((void *) &vdst[i], nloe * sizeof(float), tmp); \- } \+ } \} while(0)static inline void hvx_sigmoid_f32_aa(uint8_t * restrict dst, const uint8_t * restrict src, uint32_t n) {diff --git a/ggml/src/ggml-hexagon/htp/im2col-ops.c b/ggml/src/ggml-hexagon/htp/im2col-ops.cindex 35fc103df8..52bbc37d1b 100644--- a/ggml/src/ggml-hexagon/htp/im2col-ops.c+++ b/ggml/src/ggml-hexagon/htp/im2col-ops.c@@ -3,11 +3,12 @@#pragma clang diagnostic ignored "-Wunused-but-set-variable"#include <HAP_farf.h>-#include <HAP_perf.h>#include <hexagon_protos.h>#include <hexagon_types.h>#include <string.h>+#include "hex-common.h"+#define GGML_COMMON_DECL_C#include "ggml-common.h"#include "htp-ctx.h"@@ -16,14 +17,19 @@#include "hex-dma.h"#include "hex-profile.h"#include "htp-vtcm.h"+#include "htp-tensor.h"struct htp_im2col_context {struct htp_ops_context * octx;+ uint32_t patch_base; // first patch index assigned to this dev+ uint32_t npatches; // number of patches assigned to this devuint32_t npatches_per_thread; // patches = N*OH*OW (pure-DDR kernel)- uint32_t pe_rows_per_thread; // N*OH rows per worker- uint32_t pe_src_row_bytes; // one output row's source: IC*KH*IW*4, rounded 256- uint32_t pe_dst_row_bytes; // one output row's dst: OW*patch_stride*2, rounded 256+ uint32_t pe_row_base; // first N*OH row index assigned to this dev (DMA path)+ uint32_t pe_nrows; // number of N*OH rows assigned to this dev (DMA path)+ uint32_t pe_rows_per_thread; // N*OH rows per worker+ uint32_t pe_src_row_bytes; // one output row's source: IC*KH*IW*4, rounded 256+ uint32_t pe_dst_row_bytes; // one output row's dst: OW*patch_stride*2, rounded 256// Patch-embed DMA path VTCM ping-pong.uint8_t * pe_vtcm_src; // base of the 2x src buffers region@@ -58,33 +64,27 @@ static inline void htp_im2col_vtcm_layout_build(struct htp_im2col_vtcm_layout *struct htp_im2col_context * ictx = (struct htp_im2col_context *) data; \struct htp_ops_context * octx = ictx->octx; \struct htp_thread_trace * restrict tr = &octx->ctx->trace[ith]; \+ const struct htp_tensor * restrict src0 = octx->src[0]; \const struct htp_tensor * restrict src1 = octx->src[1]; \const struct htp_tensor * restrict dst = octx->dst; \- const int32_t s0 = octx->op_params[0]; \- const int32_t s1 = octx->op_params[1]; \- const int32_t p0 = octx->op_params[2]; \- const int32_t p1 = octx->op_params[3]; \- const int32_t d0 = octx->op_params[4]; \- const int32_t d1 = octx->op_params[5]; \- const uint32_t N = src1->ne[3]; \- const uint32_t IC = src1->ne[2]; \- const uint32_t IH = src1->ne[1]; \- const uint32_t IW = src1->ne[0]; \- const uint32_t KH = octx->src[0]->ne[1]; \- const uint32_t KW = octx->src[0]->ne[0]; \+ const int32_t s0 = octx->op_params[0], s1 = octx->op_params[1]; \+ const int32_t p0 = octx->op_params[2], p1 = octx->op_params[3]; \+ const int32_t d0 = octx->op_params[4], d1 = octx->op_params[5]; \+ const uint32_t N = src1->ne[3], IC = src1->ne[2], IH = src1->ne[1], IW = src1->ne[0]; \+ const uint32_t KH = src0->ne[1], KW = src0->ne[0]; \const uint32_t OH = dst->ne[2]; \const uint32_t OW = dst->ne[1]; \const uint32_t patch_stride = IC * KH * KW; \const float * restrict src_data = (const float *) src1->data; \DST_CTYPE * restrict dst_data = (DST_CTYPE *) dst->data; \- const uint32_t npatches = N * OH * OW; \- const uint32_t patch_start = ictx->npatches_per_thread * ith; \- const uint32_t patch_end = MIN(patch_start + ictx->npatches_per_thread, npatches); \- if (patch_start >= patch_end) { \+ const uint32_t patch_end = ictx->patch_base + ictx->npatches; \+ const uint32_t patch_start = ictx->patch_base + ictx->npatches_per_thread * ith; \+ const uint32_t patch_stop = MIN(patch_start + ictx->npatches_per_thread, patch_end);\+ if (patch_start >= patch_stop) { \return; \} \htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, patch_start); \- for (uint32_t p = patch_start; p < patch_end; p++) { \+ for (uint32_t p = patch_start; p < patch_stop; p++) { \const uint32_t iow = p % OW; \const uint32_t ioh = (p / OW) % OH; \const uint32_t in = p / (OW * OH); \@@ -154,10 +154,10 @@ IM2COL_PATCHEMBED_BODY(im2col_patchembed_f32_thread, float, hvx_copy_f32_uu, hvxuint8_t * dst_base = ictx->pe_vtcm_dst + ith * ictx->pe_dst_size_per_thread; \float * srcb = (float *) src_base; \DST_CTYPE * dstb = (DST_CTYPE *) dst_base; \- const uint32_t nrows = N * OH; \+ const uint32_t row_end_max = ictx->pe_row_base + ictx->pe_nrows; \const uint32_t per_thread = ictx->pe_rows_per_thread; \- const uint32_t row_start = per_thread * ith; \- const uint32_t row_end = MIN(row_start + per_thread, nrows); \+ const uint32_t row_start = ictx->pe_row_base + per_thread * ith; \+ const uint32_t row_end = MIN(row_start + per_thread, row_end_max); \if (row_start >= row_end) \return; \for (uint32_t r = row_start; r < row_end; r++) { \@@ -266,26 +266,55 @@ int op_im2col(struct htp_ops_context * octx) {return HTP_STATUS_NO_SUPPORT;}- const uint32_t N = src1->ne[3];- const uint32_t OH = dst->ne[2];- const uint32_t OW = dst->ne[1];- const uint32_t npatches = N * OH * OW;- const uint32_t n_threads = MIN(octx->n_threads, npatches);+ if (octx->flags & HTP_OPFLAGS_SKIP_COMPUTE) {+ return HTP_STATUS_OK;+ }++ const uint32_t N = src1->ne[3];+ const uint32_t OH = dst->ne[2];+ const uint32_t OW = dst->ne[1];+ const uint32_t total_patches = N * OH * OW;+ const uint32_t total_rows = N * OH;++ uint32_t patch_base = 0;+ uint32_t npatches = total_patches;+ if (octx->ctx->mdev.count > 1) {+ const uint32_t patch_size = dst->nb[1];+ const uint32_t patches_per_chunk = (patch_size > 0) ? (HEX_L2_LINE_SIZE / hex_gcd_u32(patch_size, HEX_L2_LINE_SIZE)) : 1;+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_patches, htp_tensor_mdev_data_aligned(dst) ? patches_per_chunk : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ patch_base = range.start;+ npatches = range.count;+ }++ uint32_t row_base = 0;+ uint32_t nrows = total_rows;+ if (octx->ctx->mdev.count > 1) {+ const uint32_t row_size = dst->nb[2];+ const uint32_t rows_per_chunk = (row_size > 0) ? (HEX_L2_LINE_SIZE / hex_gcd_u32(row_size, HEX_L2_LINE_SIZE)) : 1;+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_rows, htp_tensor_mdev_data_aligned(dst) ? rows_per_chunk : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_base = range.start;+ nrows = range.count;+ }- if ((octx->flags & HTP_OPFLAGS_SKIP_COMPUTE) || n_threads == 0) {+ if (npatches == 0 && nrows == 0) {return HTP_STATUS_OK;}+ const uint32_t n_threads = MIN(octx->n_threads, MAX(npatches, 1));+struct htp_im2col_context ictx = { 0 };- ictx.octx = octx;- ictx.npatches_per_thread = (npatches + n_threads - 1) / n_threads;+ ictx.octx = octx;+ ictx.patch_base = patch_base;+ ictx.npatches = npatches;+ ictx.npatches_per_thread = (npatches + n_threads - 1) / n_threads;// Clean non-overlapping patch-embed -> DMA kernel (if it fits VTCM);// everything else (padding/dilation/stride edges) -> pure-DDR kernel.- if (im2col_use_patchembed_dma(octx)) {- const uint32_t nrows = N * OH;- const uint32_t pth = MIN(octx->n_threads, nrows);+ if (im2col_use_patchembed_dma(octx) && nrows > 0) {+ const uint32_t pth = MIN(octx->n_threads, nrows);if (pth > 0 && im2col_patchembed_dma_fits(octx, &ictx, pth)) {+ ictx.pe_row_base = row_base;+ ictx.pe_nrows = nrows;ictx.pe_rows_per_thread = (nrows + pth - 1) / pth;if (dst->type == HTP_TYPE_F16) {work_queue_run(octx->ctx->work_queue, im2col_patchembed_dma_thread, &ictx, pth);@@ -297,6 +326,10 @@ int op_im2col(struct htp_ops_context * octx) {// else: doesn't fit -> fall through to the pure-DDR kernel below.}+ if (npatches == 0) {+ return HTP_STATUS_OK;+ }+if (dst->type == HTP_TYPE_F16) {work_queue_run(octx->ctx->work_queue, im2col_patchembed_thread, &ictx, n_threads);} else {diff --git a/ggml/src/ggml-hexagon/htp/main.c b/ggml/src/ggml-hexagon/htp/main.cindex be54d4fe91..1d291e16b4 100644--- a/ggml/src/ggml-hexagon/htp/main.c+++ b/ggml/src/ggml-hexagon/htp/main.c@@ -34,6 +34,7 @@#include "work-queue.h"#include "hex-profile.h"#include "allreduce-ops.h"+#include "htp-fence.h"#define HMX_QUEUE_CAPACITY 16#define HMX_QUEUE_STACK_SIZE 16384@@ -710,22 +711,43 @@ static inline void profile_stop(uint32_t mode, struct profile_data * d) {static int op_fence(struct htp_ops_context * octx) {struct htp_context *ctx = octx->ctx;struct htp_thread_trace * tr = &ctx->trace[0];- const uint32_t seq = (uint32_t) octx->op_params[0];+ const uint32_t seq = (uint32_t) octx->op_params[0];+ const uint32_t mode = (uint32_t) octx->op_params[1];htp_trace_event_start(tr, HTP_TRACE_EVT_FENCE, (uint16_t) seq);const struct htp_tensor * sync = octx->src[0];- atomic_uint * sync_fence = (atomic_uint *) sync->data;+ atomic_uint * sync_fence = (atomic_uint *) (uintptr_t) sync->data;++ if (mode == 1) {+ htp_flush_dirty_ranges(ctx);++ htp_mdev_group_barrier(octx);++ if (ctx->mdev.idx == 0) {+ htp_fence_write(sync_fence, seq, octx->status);+ }+ htp_trace_event_stop(tr, HTP_TRACE_EVT_FENCE, (uint16_t) seq);+ FARF(HIGH, "ggml-hex: sync-signal : fence %p seq 0x%x status %d\n", sync_fence, seq, octx->status);+ return octx->status;+ }++ int status = HTP_STATUS_OK;uint64_t spins = 0;while (1) {- Q6_dccleaninva_A((void *) sync_fence);- asm volatile ("syncht" : : : "memory");- uint32_t val = atomic_load(&sync_fence[0]);- if ((int32_t)(val - seq) >= 0) {+ uint32_t sync_seq;+ uint32_t sync_status;+ htp_fence_read(sync_fence, &sync_seq, &sync_status);+ if ((int32_t)(sync_seq - seq) >= 0) {+ if (sync_status > HTP_STATUS_OK) {+ FARF(ERROR, "ggml-hex: sync-wait peer failed with status %u : fence %p seq 0x%x\n", sync_status, sync_fence, seq);+ status = sync_status;+ }break;}if (++spins > HTP_FENCE_TIMEOUT) {- FARF(ERROR, "ggml-hex: sync-wait TIMEOUT : fence %p spins %llu seq %u\n", sync_fence, spins, seq);+ FARF(ERROR, "ggml-hex: sync-wait TIMEOUT : fence %p spins %llu seq 0x%x\n", sync_fence, spins, seq);+ status = HTP_STATUS_INTERNAL_ERR;break;}hex_pause();@@ -733,12 +755,27 @@ static int op_fence(struct htp_ops_context * octx) {htp_trace_event_stop(tr, HTP_TRACE_EVT_FENCE, (uint16_t) seq);- FARF(HIGH, "ggml-hex: sync-done : fence %p spins %llu seq %u\n", sync_fence, spins, seq);+ FARF(HIGH, "ggml-hex: sync-done : fence %p spins %llu seq 0x%x\n", sync_fence, spins, seq);+ return status;+}++static int op_mdev_group(struct htp_ops_context * octx) {+ struct htp_context * ctx = octx->ctx;+ const struct htp_tensor * sync = octx->src[0];+ ctx->mdev.idx = (uint16_t) octx->op_params[0];+ ctx->mdev.count = (uint16_t) sync->ne[1];+ if (ctx->mdev.count > 1) {+ ctx->mdev.count_div = init_fastdiv_values(ctx->mdev.count);+ ctx->mdev.fence_base = (uint8_t *) sync->data;+ }return HTP_STATUS_OK;}static int execute_op(struct htp_ops_context * octx) {switch (octx->op) {+ case HTP_OP_MDEV_GROUP:+ return op_mdev_group(octx);+case HTP_OP_FENCE:return op_fence(octx);@@ -812,6 +849,7 @@ static int execute_op(struct htp_ops_context * octx) {return op_sum_rows(octx);case HTP_OP_CPY:+ case HTP_OP_CPY_FENCE:return op_cpy(octx);case HTP_OP_REPEAT:@@ -855,7 +893,7 @@ static int execute_op(struct htp_ops_context * octx) {}FARF(ERROR, "Unknown Op %u", octx->op);- return -1;+ return HTP_STATUS_NO_SUPPORT;}static inline bool reuse_buf(struct htp_context *ctx, uint32_t *m_reuse, struct htp_buf_desc *b) {@@ -984,11 +1022,19 @@ static void prep_tensors(struct htp_context *ctx, struct htp_buf_desc *bufs, str}}-static int proc_op_req(struct htp_ops_context * octx, struct htp_tensor *tens, uint32_t idx, struct htp_op_desc * op) {- memcpy(octx->op_params, op->params, sizeof(octx->op_params));+static void mdev_group_init(struct htp_context * ctx, const struct htp_opbatch_req * req) {+ memset(&ctx->mdev, 0, sizeof(ctx->mdev));+ ctx->mdev.fence_seq = (uint32_t)((req->seq & 0xfffff) << 12);+}++static int proc_op_req(struct htp_ops_context * octx, struct htp_buf_desc * bufs, uint32_t n_bufs,+ struct htp_tensor * tens, uint32_t idx, struct htp_op_desc * op) {+ memcpy(octx->op_params, op->params, sizeof(octx->op_params));memcpy(octx->kernel_params, op->kernel_params, sizeof(octx->kernel_params));- octx->flags = op->flags;- octx->op = op->opcode;+ octx->flags = op->flags;+ octx->op = op->opcode;+ octx->n_threads = octx->ctx->n_threads;+ octx->n_threads_div = octx->ctx->n_threads_div;FARF(HIGH, "proc-op #%u: opcode %u flags 0x%x", idx, octx->op, octx->flags);@@ -1027,9 +1073,13 @@ static int proc_op_req(struct htp_ops_context * octx, struct htp_tensor *tens, udst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3]);}+ htp_tensor_dirty_all(octx->ctx, octx->dsts, HTP_OP_MAX_OUTPUTS);++ htp_mdev_group_barrier(octx);+int status = execute_op(octx);- htp_tensor_dirty_all(octx->ctx, octx->dsts, HTP_OP_MAX_OUTPUTS);+ htp_ops_context_set_status(octx, status);octx->src0_spad.src = NULL;octx->src1_spad.src = NULL;@@ -1037,7 +1087,7 @@ static int proc_op_req(struct htp_ops_context * octx, struct htp_tensor *tens, uoctx->src3_spad.src = NULL;octx->dst_spad.src = NULL;- return status;+ return octx->status;}static void process_opbatch(struct htp_context * ctx, const struct htp_opbatch_req * req, const struct dspqueue_buffer * dbuf) {@@ -1059,7 +1109,7 @@ static void process_opbatch(struct htp_context * ctx, const struct htp_opbatch_rreturn;}- FARF(HIGH, "processing opbatch #%u: n-bufs %u n-tensors %u n-ops %u n-traces %u : m-size %u b-size %u t-size %u o-size %u", req->id,+ FARF(HIGH, "processing opbatch #%llu: n-bufs %u n-tensors %u n-ops %u n-traces %u : m-size %u b-size %u t-size %u o-size %u", (unsigned long long) req->seq,n_bufs, n_tens, n_ops, req->n_traces, dbuf->size, b_size, t_size, o_size);// Setup descriptor pointers@@ -1096,8 +1146,11 @@ static void process_opbatch(struct htp_context * ctx, const struct htp_opbatch_rstruct htp_ops_context *octx = &ctx->octx;memset(octx, 0, sizeof(*octx));- octx->n_threads = ctx->n_threads;- octx->ctx = ctx;+ octx->n_threads = ctx->n_threads;+ octx->n_threads_div = ctx->n_threads_div;+ octx->ctx = ctx;++ mdev_group_init(ctx, req);work_queue_wakeup(ctx->work_queue);if (ctx->hmx_queue) {@@ -1105,15 +1158,18 @@ static void process_opbatch(struct htp_context * ctx, const struct htp_opbatch_r}int op_status = HTP_STATUS_OK;- for (uint32_t i = 0; i < n_ops && op_status == HTP_STATUS_OK; i++) {+ octx->status = HTP_STATUS_OK;+ for (uint32_t i = 0; i < n_ops; i++) {struct profile_data prof;profile_start(ctx->profiler, &prof);- op_status = proc_op_req(octx, tens, i, &ops[i]);+ op_status = proc_op_req(octx, bufs, n_bufs, tens, i, &ops[i]);profile_stop(ctx->profiler, &prof);+ htp_ops_context_set_status(octx, op_status);+if (ctx->profiler) {pds[i].opcode = ops[i].opcode;pds[i].usecs = prof.usecs;@@ -1136,19 +1192,20 @@ static void process_opbatch(struct htp_context * ctx, const struct htp_opbatch_rqurt_mem_cache_clean((qurt_addr_t) 0, 0, QURT_MEM_CACHE_FLUSH_INVALIDATE_ALL, QURT_MEM_DCACHE);htp_trace_event_stop(&ctx->trace[0], HTP_TRACE_EVT_L2FLUSH, 0);+ htp_mdev_group_barrier(octx);+profile_stop(HTP_PROF_BASIC, &batch_prof);struct htp_opbatch_rsp rsp;memset(&rsp, 0, sizeof(rsp));- rsp.id = req->id;- rsp.status = op_status;+ rsp.seq = req->seq;+ rsp.status = octx->status;rsp.n_bufs = n_bufs;rsp.n_tensors = n_tens;rsp.n_ops = n_ops;rsp.usecs = batch_prof.usecs;rsp.cycles_start = batch_prof.cycles_start;rsp.cycles_stop = batch_prof.cycles_stop;- rsp.seq = req->seq;if (ctx->profiler == HTP_PROF_TRACE) {for (int t = 0; t <= HTP_MAX_NTHREADS; t++) {diff --git a/ggml/src/ggml-hexagon/htp/matmul-ops.c b/ggml/src/ggml-hexagon/htp/matmul-ops.cindex 2a87dd19ee..1b597dcd9f 100644--- a/ggml/src/ggml-hexagon/htp/matmul-ops.c+++ b/ggml/src/ggml-hexagon/htp/matmul-ops.c@@ -21,6 +21,7 @@#include "ggml-common.h"#include "htp-ctx.h"#include "htp-ops.h"+#include "htp-tensor.h"#include "matmul-ops.h"#include "htp-vtcm.h"@@ -89,6 +90,8 @@ struct htp_mm_context {// Precomputed valuesuint32_t src0_nrows_per_thread;+ uint32_t src0_row_start;+ uint32_t src0_row_end;uint32_t src0_row_size_padded;uint32_t src1_nrows;@@ -135,6 +138,23 @@ struct htp_mm_context {uint32_t vtcm_dst_size_per_thread;};+static int htp_mm_init_context(+ struct htp_ops_context * octx,+ const struct htp_mm_kernel_params * kparams+) {+ if (!htp_ops_context_set_n_threads(octx, (uint32_t) kparams->n_threads)) {+ return HTP_STATUS_INVAL_PARAMS;+ }++ if (kparams->n_hmx) {+ if (kparams->n_act_threads <= 0 || kparams->n_act_threads > (int32_t) octx->n_threads) {+ return HTP_STATUS_INVAL_PARAMS;+ }+ }++ return HTP_STATUS_OK;+}+// vdelta control to expand first 32 e8m0 values into 32 uint32 elementsstatic const uint8_t __attribute__((aligned(128))) expand_x32_e8m0[128] = {0x00, 0x00, 0x00, 0x00, 0x01, 0x04, 0x00, 0x00, 0x02, 0x00, 0x08, 0x08, 0x01, 0x02, 0x00, 0x04, 0x04, 0x00, 0x00,@@ -238,22 +258,24 @@ static void hvx_mm_4d(unsigned int nth, unsigned int ith, void * data) {// This is the size of the rest of the dimensions of the resultconst uint32_t nr1 = ne1 * ne2 * ne3;+ const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start;+// distribute the thread work across the inner or outer loop based on which one is largeruint32_t dr0, dr1, ith0, ith1;if (nr0 > nr1) {- dr0 = fastdiv(nr0 + nth - 1, &octx->ctx->n_threads_div);+ dr0 = fastdiv(src0_nrows + nth - 1, &octx->n_threads_div);dr1 = nr1;ith0 = ith;ith1 = 0;} else {- dr0 = nr0;- dr1 = fastdiv(nr1 + nth - 1, &octx->ctx->n_threads_div);+ dr0 = src0_nrows;+ dr1 = fastdiv(nr1 + nth - 1, &octx->n_threads_div);ith0 = 0;ith1 = ith;}- const uint32_t ir0_start = dr0 * ith0;- const uint32_t ir0_end = MIN(ir0_start + dr0, nr0);+ const uint32_t ir0_start = mmctx->src0_row_start + dr0 * ith0;+ const uint32_t ir0_end = MIN(ir0_start + dr0, mmctx->src0_row_end);const uint32_t ir1_start = dr1 * ith1;const uint32_t ir1_end = MIN(ir1_start + dr1, nr1);@@ -312,11 +334,11 @@ static void hvx_mm_4d(unsigned int nth, unsigned int ith, void * data) {static void hvx_mm_2d_repacked_##SUFFIX(unsigned int nth, unsigned int ith, void * data) { \htp_matmul_preamble; \\- const uint32_t src0_nrows = ne01 * ne02 * ne03; \+ const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start; \const uint32_t src1_nrows = ne11 * ne12 * ne13; \\- const uint32_t src0_start_row = src0_nrows_per_thread * ith; \- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, src0_nrows); \+ const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith; \+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end); \\struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \\@@ -414,10 +436,10 @@ static void hvx_mm_2d_repacked_##SUFFIX(unsigned int nth, unsigned int ith, voidstatic void hvx_mv_2d_repacked_##SUFFIX(unsigned int nth, unsigned int ith, void * data) { \htp_matmul_preamble; \\- const uint32_t src0_nrows = ne01; \+ const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start; \\- const uint32_t src0_start_row = src0_nrows_per_thread * ith; \- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, src0_nrows); \+ const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith; \+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end); \\struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \\@@ -549,12 +571,22 @@ static void hvx_mm_nx_2d_repacked_##SUFFIX(unsigned int nth, unsigned int ith, vuint32_t n_k_tiles_w = ne00 / 32; \uint32_t tile_row_stride = n_k_tiles_w * tile_size; \\- const uint32_t src0_nrows = ne01 * src_w->ne[2] * src_w->ne[3]; \- uint32_t src0_nrows_per_thread = fastdiv(src0_nrows + nth - 1, &octx->ctx->n_threads_div); \+ uint32_t src0_start_row = 0; \+ uint32_t src0_end_row = ne01; \+ if (octx->ctx->mdev.count > 1) { \+ const bool can_split = htp_tensor_can_row_partition(dst, sizeof(float)); \+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(ne01, can_split ? 32 : 0, \+ octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div); \+ src0_start_row = range.start; \+ src0_end_row = range.start + range.count; \+ } \+ \+ const uint32_t nrows = src0_end_row - src0_start_row; \+ uint32_t src0_nrows_per_thread = fastdiv(nrows + nth - 1, &octx->n_threads_div); \src0_nrows_per_thread = hex_round_up(src0_nrows_per_thread, 32); \\- const uint32_t start_row = src0_nrows_per_thread * ith; \- const uint32_t end_row = MIN(start_row + src0_nrows_per_thread, src0_nrows); \+ const uint32_t start_row = src0_start_row + src0_nrows_per_thread * ith; \+ const uint32_t end_row = MIN(start_row + src0_nrows_per_thread, src0_end_row); \if (start_row >= end_row) continue; \\uint32_t ct_start = start_row / 32; \@@ -735,11 +767,11 @@ static void hvx_mm_2d(unsigned int nth, unsigned int ith, void * data) {assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0);const uint32_t prefetch_mask = n_prefetch - 1;- const uint32_t src0_nrows = ne01 * ne02 * ne03; // src0 rows- const uint32_t src1_nrows = ne11 * ne12 * ne13; // src1 rows+ const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start; // src0 rows+ const uint32_t src1_nrows = ne11 * ne12 * ne13; // src1 rows- const uint32_t src0_start_row = src0_nrows_per_thread * ith;- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, src0_nrows);+ const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith;+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end);const uint32_t src0_end_row_x2 = src0_start_row + ((src0_end_row - src0_start_row) & ~1U);struct htp_thread_trace * tr = &octx->ctx->trace[ith];@@ -781,7 +813,7 @@ static void hvx_mm_2d(unsigned int nth, unsigned int ith, void * data) {const uint8_t * ss0 = dma_queue_pop(dma_queue).dst;htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ir0);- // Process src1 columns in pairs (2×2 tiling)+ // Process src1 columns in pairs (2x2 tiling)uint32_t ir1 = 0;for (; ir1 + 1 < src1_nrows; ir1 += 2) {const uint8_t * restrict src1_col0 = (const uint8_t *) (src1_data + (ir1+0) * src1_stride);@@ -791,7 +823,7 @@ static void hvx_mm_2d(unsigned int nth, unsigned int ith, void * data) {mmctx->vec_dot_2x2(ne00, &dst_row0[ir0], &dst_row1[ir0], ss0, ss0 + src0_stride, src1_col0, src1_col1);}- // Handle remaining src1 rows (fallback to 2×1)+ // Handle remaining src1 rows (fallback to 2x1)for (; ir1 < src1_nrows; ++ir1) {const uint8_t * restrict src1_col = (const uint8_t *) (src1_data + ir1 * src1_stride);float * restrict dst_row = (float *) (dst->data + (ir1 * dst_row_size));@@ -833,10 +865,10 @@ static void hvx_mm_2d(unsigned int nth, unsigned int ith, void * data) {static void hvx_mv_2d(unsigned int nth, unsigned int ith, void * data) {htp_matmul_preamble;- const uint32_t src0_nrows = ne01;+ const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start;- const uint32_t src0_start_row = src0_nrows_per_thread * ith;- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, src0_nrows);+ const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith;+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end);struct htp_thread_trace * tr = &octx->ctx->trace[ith];@@ -943,13 +975,10 @@ static void hvx_mm_id(unsigned int nth, unsigned int ith, void * data) {const struct htp_tensor * restrict ids = octx->src[2];- uint64_t t1, t2;- t1 = HAP_perf_get_qtimer_count();-- const uint32_t src0_nrows = ne01; // src0 rows per expert+ const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start; // src0 rows per expertconst uint32_t src1_nrows = ne11;- const uint32_t src0_start_row = src0_nrows_per_thread * ith;- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, src0_nrows);+ const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith;+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end);hvx_mm_run_quant_task(mmctx, ith);@@ -1036,9 +1065,9 @@ static void hvx_mv_id(unsigned int nth, unsigned int ith, void * data) {const struct htp_tensor * restrict ids = octx->src[2];- const uint32_t src0_nrows = ne01; // src0 rows per expert- const uint32_t src0_start_row = src0_nrows_per_thread * ith;- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, src0_nrows);+ const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start; // src0 rows per expert+ const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith;+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end);hvx_mm_run_quant_task(mmctx, ith);@@ -1143,12 +1172,22 @@ static void hvx_mv_id_nx(unsigned int nth, unsigned int ith, void * data) {const struct htp_tensor * restrict dst = octx->dsts[p];if (!src_w || !dst) continue;- const uint32_t src0_nrows = src_w->ne[1];- uint32_t src0_nrows_per_thread = fastdiv(src0_nrows + nth - 1, &octx->ctx->n_threads_div);+ const uint32_t ne01 = src_w->ne[1];+ uint32_t start_row = 0;+ uint32_t end_row = ne01;+ if (octx->ctx->mdev.count > 1) {+ const bool can_split = htp_tensor_can_row_partition(dst, sizeof(float));+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(ne01, can_split ? 32 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ start_row = range.start;+ end_row = range.start + range.count;+ }++ const uint32_t nrows = end_row - start_row;+ uint32_t src0_nrows_per_thread = fastdiv(nrows + nth - 1, &octx->n_threads_div);src0_nrows_per_thread = hex_round_up(src0_nrows_per_thread, 32);- const uint32_t src0_start_row = src0_nrows_per_thread * ith;- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, src0_nrows);+ const uint32_t src0_start_row = start_row + src0_nrows_per_thread * ith;+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, end_row);if (src0_start_row >= src0_end_row) continue;const uint8_t * restrict src0_row = (const uint8_t *) src_w->data + eid * src_w->nb[2];@@ -1227,12 +1266,22 @@ static void hvx_mm_id_nx(unsigned int nth, unsigned int ith, void * data) {const struct htp_tensor * restrict dst = octx->dsts[p];if (!src_w || !dst) continue;- const uint32_t src0_nrows = src_w->ne[1];- uint32_t src0_nrows_per_thread = fastdiv(src0_nrows + nth - 1, &octx->ctx->n_threads_div);+ const uint32_t ne01 = src_w->ne[1];+ uint32_t start_row = 0;+ uint32_t end_row = ne01;+ if (octx->ctx->mdev.count > 1) {+ const bool can_split = htp_tensor_can_row_partition(dst, sizeof(float));+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(ne01, can_split ? 32 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ start_row = range.start;+ end_row = range.start + range.count;+ }++ const uint32_t nrows = end_row - start_row;+ uint32_t src0_nrows_per_thread = fastdiv(nrows + nth - 1, &octx->n_threads_div);src0_nrows_per_thread = hex_round_up(src0_nrows_per_thread, 32);- const uint32_t src0_start_row = src0_nrows_per_thread * ith;- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, src0_nrows);+ const uint32_t src0_start_row = start_row + src0_nrows_per_thread * ith;+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, end_row);if (src0_start_row >= src0_end_row) continue;const uint8_t * src0_row = (const uint8_t *) src_w->data + cur_a * src_w->nb[2];@@ -1323,15 +1372,33 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;- const uint32_t src0_nrows = ne01 * ne02 * ne03;+ const uint32_t src0_nrows = ne01;const uint32_t src1_nrows = ne11 * ne12 * ne13;+ uint32_t src0_row_start = 0;+ uint32_t src0_row_end = src0_nrows;++ if (octx->ctx->mdev.count > 1) {+ const bool can_split = htp_tensor_can_row_partition(dst, sizeof(float));+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(src0_nrows, can_split ? 32 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ src0_row_start = range.start;+ src0_row_end = range.start + range.count;+ }++ if (src0_row_start >= src0_row_end) {+ return HTP_STATUS_OK;+ }++ const uint32_t nrows = src0_row_end - src0_row_start;+ mmctx->src0_row_start = src0_row_start;+ mmctx->src0_row_end = src0_row_end;+bool is_repacked = (src0->type == HTP_TYPE_Q4_0 || src0->type == HTP_TYPE_Q4_1 ||src0->type == HTP_TYPE_Q8_0 || src0->type == HTP_TYPE_IQ4_NL ||src0->type == HTP_TYPE_MXFP4);// Compute src0_nrows_per_thread- mmctx->src0_nrows_per_thread = fastdiv(src0_nrows + octx->n_threads - 1, &octx->ctx->n_threads_div);+ mmctx->src0_nrows_per_thread = fastdiv(nrows + octx->n_threads - 1, &octx->n_threads_div);if (is_repacked) {mmctx->src0_nrows_per_thread = hex_round_up(mmctx->src0_nrows_per_thread, 32);} else {@@ -1503,13 +1570,13 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {kparams->kernel_type == HTP_MM_KERNEL_HVX_QUANT_BLOCK) {mmctx->vtcm_src1_size_per_thread = L.src1_bytes;} else {- mmctx->vtcm_src1_size_per_thread = fastdiv(L.src1_bytes, &octx->ctx->n_threads_div);+ mmctx->vtcm_src1_size_per_thread = fastdiv(L.src1_bytes, &octx->n_threads_div);}- mmctx->vtcm_src0_size_per_thread = fastdiv(L.src0_bytes, &octx->ctx->n_threads_div);- mmctx->vtcm_dst_size_per_thread = fastdiv(L.dst_bytes, &octx->ctx->n_threads_div);+ mmctx->vtcm_src0_size_per_thread = fastdiv(L.src0_bytes, &octx->n_threads_div);+ mmctx->vtcm_dst_size_per_thread = fastdiv(L.dst_bytes, &octx->n_threads_div);- size_t vtcm_size = kparams->vtcm_size > 0 ? (size_t)kparams->vtcm_size : L.total_bytes;+ const size_t vtcm_size = L.total_bytes;FARF(HIGH, "matmul-%s : src0-vtcm-size %zu src1-vtcm-size %zu dst-vtcm-size %zu (%zu)\n", mmctx->type,L.src0_bytes, L.src1_bytes, L.dst_bytes, vtcm_size);@@ -1583,13 +1650,21 @@ static void hvx_mm_nx_2d(unsigned int nth, unsigned int ith, void * data) {const uint32_t ne00 = src_w->ne[0];const uint32_t ne01 = src_w->ne[1];- const uint32_t src0_nrows = ne01 * src_w->ne[2] * src_w->ne[3];+ uint32_t start_row = 0;+ uint32_t end_row = ne01;+ if (octx->ctx->mdev.count > 1) {+ const bool can_split = htp_tensor_can_row_partition(dst, sizeof(float));+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(ne01, can_split ? 32 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ start_row = range.start;+ end_row = range.start + range.count;+ }- uint32_t src0_nrows_per_thread = fastdiv(src0_nrows + nth - 1, &octx->ctx->n_threads_div);+ const uint32_t nrows = end_row - start_row;+ uint32_t src0_nrows_per_thread = fastdiv(nrows + nth - 1, &octx->n_threads_div);src0_nrows_per_thread += (src0_nrows_per_thread & 1);- const uint32_t src0_start_row = src0_nrows_per_thread * ith;- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, src0_nrows);+ const uint32_t src0_start_row = start_row + src0_nrows_per_thread * ith;+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, end_row);const uint32_t src0_end_row_x2 = src0_start_row + ((src0_end_row - src0_start_row) & ~1U);if (src0_start_row >= src0_end_row) continue;@@ -2638,10 +2713,6 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_kconst struct htp_tensor * restrict src0 = octx->src[0];const struct htp_tensor * restrict act = octx->src[n_weights];- if (!src0 || !act) {- return HTP_STATUS_INVAL_PARAMS;- }-const int weight_type = (int) src0->type;const int k = (int) act->ne[0];const int k_valid = (int) act->ne[0];@@ -2714,16 +2785,31 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_khmx_init_column_scales(vtcm_scales, Q6_V_vsplat_R(0x3c00)); // scale: 1.0, bias: 0.0 in FP16- FARF(HIGH, "hmx-mm-nx-2d: n_weights %u m %d k %d wtype %d mc %d nc %d vtcm %zu/%zu",- n_weights, m, k, weight_type, m_chunk_n_rows, n_chunk_n_cols, L.total_bytes, vtcm_budget);+ int m_start = 0;+ int m_rows = m;+ if (octx->ctx->mdev.count > 1) {+ const bool can_split = htp_tensor_can_row_partition(octx->dsts[0], sizeof(float));+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition((uint32_t) m, can_split ? 1 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ m_start = (int) range.start;+ m_rows = (int) range.count;+ }++ if (m_rows == 0) {+ return HTP_STATUS_OK;+ }++ FARF(HIGH, "hmx-mm-nx-2d: n_weights %u m %d (%d..%d) k %d wtype %d mc %d nc %d vtcm %zu/%zu",+ n_weights, m, m_start, m_start + m_rows, k, weight_type, m_chunk_n_rows, n_chunk_n_cols, L.total_bytes, vtcm_budget);htp_trace_event_stop(tr, HTP_TRACE_EVT_INIT, 0);+ const size_t mr_end = (size_t)(m_start + m_rows);+if (pipeline) {hmx_matmul_job_t job_slots[2];- for (size_t mr = 0; mr < (size_t) m; mr += m_chunk_n_rows) {- const size_t n_rows = hex_smin(m - mr, m_chunk_n_rows);+ for (size_t mr = (size_t) m_start; mr < mr_end; mr += m_chunk_n_rows) {+ const size_t n_rows = hex_smin(mr_end - mr, m_chunk_n_rows);void *vtcm_weight_bufs[2] = { vtcm_scratch0, vtcm_scratch1 };void *vtcm_output_bufs[2] = { vtcm_output, vtcm_scratch2 };@@ -2822,8 +2908,8 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_k}} else {hmx_matmul_job_t job;- for (size_t mr = 0; mr < (size_t) m; mr += m_chunk_n_rows) {- const size_t n_rows = hex_smin(m - mr, m_chunk_n_rows);+ for (size_t mr = (size_t) m_start; mr < mr_end; mr += m_chunk_n_rows) {+ const size_t n_rows = hex_smin(mr_end - mr, m_chunk_n_rows);struct activation_transfer_params act_params = {.ctx = ctx,@@ -3095,7 +3181,7 @@ static int hmx_mm_f16_f32_batched(struct htp_context *ctx, const hmx_mm_f16_f32_int chunk_dst_cols = params->n - (int)nc;if (chunk_dst_cols > 0) {transfer_output_chunk_threaded(ctx, output, src2_chunk, vtcm_output, (int) n_rows, (int) n_cols,- params->dst_stride, params->src2_stride, chunk_dst_cols, ctx->n_threads);+ params->dst_stride, params->src2_stride, chunk_dst_cols, n_threads);}}}@@ -3216,7 +3302,10 @@ static int hmx_mm_id_2d_f32(struct htp_context *ctx,int weight_type,const struct mmid_row_mapping *matrix_rows,int cur_a,- int mapping_stride) {+ int mapping_stride,+ int m_start,+ int m_end,+ int n_threads) {struct htp_thread_trace * tr = &ctx->trace[0];htp_trace_event_start(tr, HTP_TRACE_EVT_INIT, 0);@@ -3247,7 +3336,6 @@ static int hmx_mm_id_2d_f32(struct htp_context *ctx,const int n_k_tiles = k / HTP_MM_HMX_TILE_N_COLS;const struct fastdiv_values n_k_tiles_div = init_fastdiv_values(n_k_tiles);- const int n_threads = ctx->n_threads;const bool is_quant = (weight_type != HTP_TYPE_F16 && weight_type != HTP_TYPE_F32);const size_t vec_dot_size = k * sizeof(__fp16);@@ -3303,8 +3391,8 @@ static int hmx_mm_id_2d_f32(struct htp_context *ctx,hmx_matmul_job_t job;- for (size_t mr = 0; mr < (size_t) m_padded; mr += m_chunk_n_rows) {- const size_t n_rows = hex_smin(m_padded - mr, m_chunk_n_rows);+ for (size_t mr = (size_t) m_start; mr < (size_t) m_end; mr += m_chunk_n_rows) {+ const size_t n_rows = hex_smin((size_t) m_end - mr, m_chunk_n_rows);const size_t n_row_tiles = hmx_ceil_div(n_rows, HTP_MM_HMX_TILE_N_ROWS);transfer_activation_chunk_gathered_threaded(@@ -3368,31 +3456,48 @@ static int hmx_mm_op_matmul(struct htp_ops_context * octx, const struct htp_mm_kconst int act_stride = (int)(src1->nb[1] / sizeof(float));const int wgt_stride = (int)(src0->nb[1] / sizeof(__fp16));+ int m_start = 0;+ int m_rows = m_total;+ if (octx->ctx->mdev.count > 1) {+ const bool can_split = htp_tensor_can_row_partition(dst, sizeof(float));+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition((uint32_t) m_total, can_split ? 1 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ m_start = (int) range.start;+ m_rows = (int) range.count;+ }++ if (m_rows == 0) {+ return HTP_STATUS_OK;+ }+const float * src2_ptr = NULL;uint32_t src2_stride = 0;size_t src2_nb2 = 0;size_t src2_nb3 = 0;if (src2) {- src2_ptr = (const float *) src2->data;src2_stride = (src2->ne[1] == 1) ? 0 : (uint32_t) (src2->nb[1] / sizeof(float));+ src2_ptr = (const float *) src2->data + m_start * src2_stride;src2_nb2 = (src2->ne[2] == 1) ? 0 : src2->nb[2];src2_nb3 = (src2->ne[3] == 1) ? 0 : src2->nb[3];}+ const int dst_stride = (int)(dst->nb[1] / sizeof(float));+ float * dst_ptr = (float *) dst->data + m_start * dst_stride;+ const float * act_ptr = (const float *) src1->data + m_start * act_stride;+int ret = -1;- const int n_threads = MIN(kparams->n_threads, (int) octx->n_threads);+ const int n_threads = kparams->n_threads;if (kparams->kernel_type == HTP_MM_KERNEL_HMX_F16_BATCHED) {hmx_mm_f16_f32_batched_params_t batch_params = {- .dst = (float *) dst->data,+ .dst = dst_ptr,.src2 = src2_ptr,- .activation = (float *) src1->data,+ .activation = act_ptr,.weight = (const __fp16 *) src0->data,- .m = m_total,+ .m = m_rows,.k = k,.n = n,.act_stride = act_stride,.weight_stride = wgt_stride,- .dst_stride = (int) (dst->nb[1] / sizeof(float)),+ .dst_stride = dst_stride,.src2_stride = src2_stride,.ne02 = ne02,.ne03 = ne03,@@ -3420,9 +3525,9 @@ static int hmx_mm_op_matmul(struct htp_ops_context * octx, const struct htp_mm_kkparams->vtcm_size);} else {ret = hmx_mm_2d_f32(- octx->ctx, (float*) dst->data, src2_ptr, (float*) src1->data, (const uint8_t *) src0->data,- m_total, k, n, act_stride, (int) src0->nb[1], (int) src0->type, (int) src1->ne[0],- (int)(dst->nb[1] / sizeof(float)), src2_stride, (int)dst->ne[0],+ octx->ctx, dst_ptr, src2_ptr, act_ptr, (const uint8_t *) src0->data,+ m_rows, k, n, act_stride, (int) src0->nb[1], (int) src0->type, (int) src1->ne[0],+ dst_stride, src2_stride, (int)dst->ne[0],kparams->m_chunk, kparams->n_chunk, kparams->pipeline, n_threads,kparams->n_act_threads,&kparams->div_n_act_threads,@@ -3441,6 +3546,11 @@ static int hmx_mm_op_matmul(struct htp_ops_context * octx, const struct htp_mm_kint op_matmul(struct htp_ops_context * octx) {const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;+ const int status = htp_mm_init_context(octx, kparams);+ if (status != HTP_STATUS_OK) {+ return status;+ }+if (kparams->n_hmx) {return hmx_mm_op_matmul(octx, kparams);}@@ -3463,6 +3573,16 @@ static int hmx_mm_op_matmul_id(const int32_t cne1 = matrix_row_counts[cur_a];if (cne1 == 0) continue;+ const int m_padded = hex_align_up(cne1, 32);+ int m_start = 0, m_end = m_padded;+ if (octx->ctx->mdev.count > 1) {+ const bool can_split = htp_tensor_mdev_data_aligned(dst) && (uint32_t) cne1 >= octx->ctx->mdev.count;+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition((uint32_t) m_padded, can_split ? 1 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ m_start = (int) range.start;+ m_end = (int) (range.start + range.count);+ }+ if (m_start >= m_end) continue;+int ret = hmx_mm_id_2d_f32(octx->ctx, (float*) dst->data, (float*) src1->data,(const uint8_t *) src0->data + cur_a * nb02,cne1, ne00, ne01,@@ -3471,7 +3591,8 @@ static int hmx_mm_op_matmul_id(nb11, nb12,nb1, nb2,(int) src0->nb[1], (int) src0->type,- matrix_rows, cur_a, mmctx->mapping_stride);+ matrix_rows, cur_a, mmctx->mapping_stride,+ m_start, m_end, (int) octx->n_threads);if (ret != 0) {FARF(ERROR, "HMX matmul failed for expert %u, error %d\n", cur_a, ret);return HTP_STATUS_NO_SUPPORT;@@ -3524,7 +3645,7 @@ static int hvx_mm_matmul_id(htp_mm_hvx_vtcm_layout_build(&L, kparams->kernel_type, src0->type, ne10, src1_nrows, octx->n_threads,0, src0_row_size, src1_row_size, 0, kparams->n_prefetch, true, false);- size_t vtcm_size = kparams->vtcm_size > 0 ? (size_t)kparams->vtcm_size : L.total_bytes;+ const size_t vtcm_size = L.total_bytes;FARF(HIGH, "matmul-id-%s : src0-spad-size %zu src1-spad-size %zu src2-spad-size 0 dst-spad-size %zu (%zu)\n", mmctx->type,L.src0_bytes, L.src1_bytes, L.dst_bytes, vtcm_size);@@ -3554,10 +3675,10 @@ static int hvx_mm_matmul_id(mmctx->vtcm_src0_stride = src0_row_size_padded;mmctx->vtcm_src1_stride = src1_row_size;- mmctx->vtcm_src0_size_per_thread = fastdiv(L.src0_bytes, &octx->ctx->n_threads_div);+ mmctx->vtcm_src0_size_per_thread = fastdiv(L.src0_bytes, &octx->n_threads_div);mmctx->vtcm_src1_size_per_thread = L.src1_bytes;mmctx->vtcm_src2_size_per_thread = 0;- mmctx->vtcm_dst_size_per_thread = fastdiv(L.dst_bytes, &octx->ctx->n_threads_div);+ mmctx->vtcm_dst_size_per_thread = fastdiv(L.dst_bytes, &octx->n_threads_div);mmctx->n_quant_rows_per_thread = (src1_nrows + n_quant_tasks - 1) / n_quant_tasks;mmctx->quant_task_func = quant_task_func;@@ -3587,6 +3708,20 @@ static int hmx_mm_op_matmul_id_nx(const int32_t cne1 = matrix_row_counts[cur_a];if (cne1 == 0) continue;+ const int m_padded = hex_align_up(cne1, 32);+ int m_start = 0, m_end = m_padded;+ if (octx->ctx->mdev.count > 1) {+ bool can_split = (uint32_t) cne1 >= octx->ctx->mdev.count;+ for (uint32_t p = 0; p < n_weights && can_split; ++p) {+ const struct htp_tensor * restrict dst = octx->dsts[p];+ can_split = !dst || htp_tensor_mdev_data_aligned(dst);+ }+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition((uint32_t) m_padded, can_split ? 1 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ m_start = (int) range.start;+ m_end = (int) (range.start + range.count);+ }+ if (m_start >= m_end) continue;+for (uint32_t p = 0; p < n_weights; ++p) {const struct htp_tensor * restrict src_w = octx->src[p];const struct htp_tensor * restrict dst = octx->dsts[p];@@ -3600,7 +3735,8 @@ static int hmx_mm_op_matmul_id_nx(act->nb[1], act->nb[2],dst->nb[1], dst->nb[2],(int) src_w->nb[1], (int) src_w->type,- matrix_rows, cur_a, mmctx->mapping_stride);+ matrix_rows, cur_a, mmctx->mapping_stride,+ m_start, m_end, (int) octx->n_threads);if (ret != 0) {FARF(ERROR, "HMX matmul ID NX failed for expert %u weight %u, error %d\n", cur_a, p, ret);return HTP_STATUS_NO_SUPPORT;@@ -3656,7 +3792,7 @@ static int hvx_mm_matmul_id_nx(htp_mm_hvx_vtcm_layout_build(&L, kparams->kernel_type, src0->type, act->ne[0], src1_nrows, octx->n_threads,0, src0_row_size, src1_row_size, 0, kparams->n_prefetch, true, false);- size_t vtcm_size = kparams->vtcm_size > 0 ? (size_t)kparams->vtcm_size : L.total_bytes;+ const size_t vtcm_size = L.total_bytes;if (octx->ctx->vtcm_size < vtcm_size) {FARF(ERROR, "matmul-id-nx: current VTCM reservation %zu is too small, needed %zu\n",@@ -3678,9 +3814,9 @@ static int hvx_mm_matmul_id_nx(mmctx->vtcm_src0_stride = 0;mmctx->vtcm_src1_stride = src1_row_size;- mmctx->vtcm_src0_size_per_thread = fastdiv(L.src0_bytes, &octx->ctx->n_threads_div);+ mmctx->vtcm_src0_size_per_thread = fastdiv(L.src0_bytes, &octx->n_threads_div);mmctx->vtcm_src1_size_per_thread = L.src1_bytes;- mmctx->vtcm_dst_size_per_thread = fastdiv(L.dst_bytes, &octx->ctx->n_threads_div);+ mmctx->vtcm_dst_size_per_thread = fastdiv(L.dst_bytes, &octx->n_threads_div);mmctx->n_quant_rows_per_thread = (src1_nrows + n_quant_tasks - 1) / n_quant_tasks;mmctx->quant_task_func = quant_task_func;@@ -3769,16 +3905,21 @@ static inline void scan_expert_ids(int op_matmul_id(struct htp_ops_context * octx) {htp_matmul_tensors_preamble;+ const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;+ struct htp_mm_context mmctx_struct = {0};+ struct htp_mm_context * mmctx = &mmctx_struct;++ const int status = htp_mm_init_context(octx, kparams);+ if (status != HTP_STATUS_OK) {+ return status;+ }+struct htp_thread_trace * tr = &octx->ctx->trace[0];htp_trace_event_start(tr, HTP_TRACE_EVT_INIT, 0);- struct htp_mm_context mmctx_struct = {0};- struct htp_mm_context * mmctx = &mmctx_struct;mmctx->octx = octx;mmctx->act = src1;- const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;-const struct htp_tensor * restrict ids = octx->src[2];const size_t src0_row_size = nb01;@@ -3789,9 +3930,6 @@ int op_matmul_id(struct htp_ops_context * octx) {const uint32_t src0_nrows = ne01; // per expertconst uint32_t src1_nrows = ne11 * ne12 * ne13;- mmctx->src0_nrows_per_thread = fastdiv(src0_nrows + octx->n_threads - 1, &octx->ctx->n_threads_div);- mmctx->src0_nrows_per_thread = hex_round_up(mmctx->src0_nrows_per_thread, 32);-// row groupsconst int n_ids = ids->ne[0]; // n_expert_usedconst int n_as = ne02; // n_expert@@ -3843,6 +3981,29 @@ int op_matmul_id(struct htp_ops_context * octx) {if (kparams->n_hmx) {s = hmx_mm_op_matmul_id(octx, mmctx);} else {+ uint32_t src0_row_start = 0;+ uint32_t src0_row_end = src0_nrows;+ if (octx->ctx->mdev.count > 1) {+ const bool can_split = htp_tensor_can_row_partition(dst, sizeof(float));+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(src0_nrows, can_split ? 32 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ src0_row_start = range.start;+ src0_row_end = range.start + range.count;+ }++ if (src0_row_start >= src0_row_end) {+ if (mapping_buf != octx->ctx->ddr_spad_base) {+ free(mapping_buf);+ }+ return HTP_STATUS_OK;+ }++ const uint32_t nrows = src0_row_end - src0_row_start;+ mmctx->src0_row_start = src0_row_start;+ mmctx->src0_row_end = src0_row_end;++ mmctx->src0_nrows_per_thread = fastdiv(nrows + octx->n_threads - 1, &octx->n_threads_div);+ mmctx->src0_nrows_per_thread = hex_round_up(mmctx->src0_nrows_per_thread, 32);+if (hvx_mm_init_vec_dot(mmctx, src0->type) == 0) {s = hvx_mm_matmul_id(octx, mmctx, src1_nrows > 1 ? hvx_mm_id : hvx_mv_id);} else {@@ -3858,29 +4019,31 @@ int op_matmul_id(struct htp_ops_context * octx) {}int op_matmul_id_nx(struct htp_ops_context * octx) {+ const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;+ struct htp_mm_context mmctx_struct = {0};+ struct htp_mm_context * mmctx = &mmctx_struct;++ const int status = htp_mm_init_context(octx, kparams);+ if (status != HTP_STATUS_OK) {+ return status;+ }+struct htp_thread_trace * tr = &octx->ctx->trace[0];htp_trace_event_start(tr, HTP_TRACE_EVT_INIT, 0);- const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;+ mmctx->octx = octx;const uint32_t n_weights = kparams->n_weights;const struct htp_tensor * restrict src0 = octx->src[0];const struct htp_tensor * restrict act = octx->src[n_weights];const struct htp_tensor * restrict ids = octx->src[n_weights + 1];- struct htp_mm_context mmctx_struct = {0};- struct htp_mm_context * mmctx = &mmctx_struct;- mmctx->octx = octx;mmctx->act = act;const size_t src0_row_size = src0->nb[1];const size_t src0_row_size_padded = hex_round_up(src0_row_size, 128);- const uint32_t src0_nrows = src0->ne[1];const uint32_t src1_nrows = act->ne[1] * act->ne[2] * act->ne[3];- mmctx->src0_nrows_per_thread = fastdiv(src0_nrows + octx->n_threads - 1, &octx->ctx->n_threads_div);- mmctx->src0_nrows_per_thread = hex_round_up(mmctx->src0_nrows_per_thread, 32);-const int n_ids = ids->ne[0];const int n_as = src0->ne[2];@@ -3946,6 +4109,12 @@ int op_matmul_id_nx(struct htp_ops_context * octx) {}int op_matmul_nx(struct htp_ops_context * octx) {const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;++ const int status = htp_mm_init_context(octx, kparams);+ if (status != HTP_STATUS_OK) {+ return status;+ }+if (kparams->n_hmx) {return hmx_mm_nx_2d_f32(octx, kparams);}@@ -4012,7 +4181,7 @@ int op_matmul_nx(struct htp_ops_context * octx) {htp_mm_hvx_vtcm_layout_build(&L, kparams->kernel_type, src0->type, act->ne[0], src1_nrows, octx->n_threads,0, src0_row_size, src1_row_size, 0, kparams->n_prefetch, false, true);- size_t vtcm_size = kparams->vtcm_size > 0 ? (size_t)kparams->vtcm_size : L.total_bytes;+ const size_t vtcm_size = L.total_bytes;if (octx->ctx->vtcm_size < vtcm_size) {FARF(ERROR, "matmul-nx: current VTCM reservation %zu is too small, needed %zu\n",@@ -4034,9 +4203,9 @@ int op_matmul_nx(struct htp_ops_context * octx) {mmctx->vtcm_src0_stride = is_repacked ? 0 : src0_row_size_padded;mmctx->vtcm_src1_stride = src1_row_size;- mmctx->vtcm_src0_size_per_thread = fastdiv(L.src0_bytes, &octx->ctx->n_threads_div);+ mmctx->vtcm_src0_size_per_thread = fastdiv(L.src0_bytes, &octx->n_threads_div);mmctx->vtcm_src1_size_per_thread = L.src1_bytes;- mmctx->vtcm_dst_size_per_thread = fastdiv(L.dst_bytes, &octx->ctx->n_threads_div);+ mmctx->vtcm_dst_size_per_thread = fastdiv(L.dst_bytes, &octx->n_threads_div);mmctx->n_quant_rows_per_thread = (src1_nrows + n_quant_tasks - 1) / n_quant_tasks;mmctx->quant_task_func = quant_task_func;diff --git a/ggml/src/ggml-hexagon/htp/pad-ops.c b/ggml/src/ggml-hexagon/htp/pad-ops.cindex aaa72b3159..0222f24dcb 100644--- a/ggml/src/ggml-hexagon/htp/pad-ops.c+++ b/ggml/src/ggml-hexagon/htp/pad-ops.c@@ -12,8 +12,11 @@#define GGML_COMMON_DECL_C#include "ggml-common.h"+#include "hex-common.h"+#include "hex-profile.h"#include "htp-ctx.h"#include "htp-ops.h"+#include "htp-tensor.h"/* Circular wrap: maps any integer x into [0, n) */static inline uint32_t wrap_around(int32_t x, uint32_t n) {@@ -68,6 +71,7 @@ struct htp_pad_context {uint32_t nrows_per_thread;uint32_t total_dst_rows;+ uint32_t row_start;size_t type_size;@@ -78,39 +82,39 @@ struct htp_pad_context {size_t dst_row_size_aligned;};-#define htp_pad_preamble \- const struct htp_tensor * src = octx->src[0]; \- const struct htp_tensor * dst = octx->dst; \- \- const uint32_t ne00 = src->ne[0]; \- const uint32_t nb00 = src->nb[0]; \- \- const uint32_t ne0 = dst->ne[0]; \- const uint32_t ne1 = dst->ne[1]; \- const uint32_t ne2 = dst->ne[2]; \- const uint32_t ne3 = dst->ne[3]; \- \- const uint32_t nb1 = dst->nb[1]; \- const uint32_t nb2 = dst->nb[2]; \- const uint32_t nb3 = dst->nb[3]; \- \- const int32_t lp0 = pctx->lp0, rp0 = pctx->rp0; \- const int32_t lp1 = pctx->lp1, rp1 = pctx->rp1; \- const int32_t lp2 = pctx->lp2, rp2 = pctx->rp2; \- const int32_t lp3 = pctx->lp3, rp3 = pctx->rp3; \- \- const size_t type_size = pctx->type_size; \- \- const uint32_t row_start = pctx->nrows_per_thread * ith; \- const uint32_t row_end = MIN(row_start + pctx->nrows_per_thread, pctx->total_dst_rows);---#define htp_pad_dma_preamble \- const size_t src_row_size = pctx->src_row_size; \- const size_t src_row_size_aligned = pctx->src_row_size_aligned; \- const size_t dst_row_size = pctx->dst_row_size; \- const size_t dst_row_size_aligned = pctx->dst_row_size_aligned; \- \+#define htp_pad_preamble \+ const struct htp_tensor * src = octx->src[0]; \+ const struct htp_tensor * dst = octx->dst; \+ \+ const uint32_t ne00 = src->ne[0]; \+ const uint32_t nb00 = src->nb[0]; \+ \+ const uint32_t ne0 = dst->ne[0]; \+ const uint32_t ne1 = dst->ne[1]; \+ const uint32_t ne2 = dst->ne[2]; \+ const uint32_t ne3 = dst->ne[3]; \+ \+ const uint32_t nb1 = dst->nb[1]; \+ const uint32_t nb2 = dst->nb[2]; \+ const uint32_t nb3 = dst->nb[3]; \+ \+ const int32_t lp0 = pctx->lp0, rp0 = pctx->rp0; \+ const int32_t lp1 = pctx->lp1, rp1 = pctx->rp1; \+ const int32_t lp2 = pctx->lp2, rp2 = pctx->rp2; \+ const int32_t lp3 = pctx->lp3, rp3 = pctx->rp3; \+ \+ const size_t type_size = pctx->type_size; \+ \+ const uint32_t row_start = pctx->row_start + pctx->nrows_per_thread * ith; \+ const uint32_t row_end = MIN(row_start + pctx->nrows_per_thread, pctx->row_start + pctx->total_dst_rows);+++#define htp_pad_dma_preamble \+ const size_t src_row_size = pctx->src_row_size; \+ const size_t src_row_size_aligned = pctx->src_row_size_aligned; \+ const size_t dst_row_size = pctx->dst_row_size; \+ const size_t dst_row_size_aligned = pctx->dst_row_size_aligned; \+ \uint8_t * src_spad_base = octx->src0_spad.data + ith * octx->src0_spad.size_per_thread; \uint8_t * dst_spad_base = octx->dst_spad.data + ith * octx->dst_spad.size_per_thread; \\@@ -125,8 +129,8 @@ static void pad_job_per_thread_hvx(unsigned int nth, unsigned int ith, void * dastruct htp_ops_context * octx = pctx->octx;htp_pad_preamble;- uint64_t t1, t2;- t1 = HAP_perf_get_qtimer_count();+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, row_start);for (uint32_t dst_row = row_start; dst_row < row_end; dst_row++) {uint32_t i1, i2, i3;@@ -165,18 +169,17 @@ static void pad_job_per_thread_hvx(unsigned int nth, unsigned int ith, void * da}}- t2 = HAP_perf_get_qtimer_count();+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, row_start);- FARF(HIGH, "pad-hvx %d/%d: (%ux%ux%ux%u) -> (%ux%ux%ux%u) rows %u:%u usec %u\n",+ FARF(HIGH, "pad-hvx %d/%d: (%ux%ux%ux%u) -> (%ux%ux%ux%u) rows %u:%u\n",ith, nth,src->ne[0], src->ne[1], src->ne[2], src->ne[3],dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3],- row_start, row_end,- (unsigned) HAP_perf_qtimer_count_to_us(t2 - t1));+ row_start, row_end);}// ----------------------------------------------------------------------------// HVX + DMA PAD kernel — aligned, double-buffered+// HVX + DMA PAD kernel - aligned, double-buffered// ---------------------------------------------------------------------------static void pad_job_per_thread_hvx_dma(unsigned int nth, unsigned int ith, void * data) {@@ -185,9 +188,6 @@ static void pad_job_per_thread_hvx_dma(unsigned int nth, unsigned int ith, voidhtp_pad_preamble;htp_pad_dma_preamble;- uint64_t t1, t2;- t1 = HAP_perf_get_qtimer_count();-// -----------------------------------------------------------------------// Priming phase: push 2 pairs of (dummy_dst_DMA, src_DMA) to seed the// double-buffer pipeline before the main loop begins.@@ -222,6 +222,8 @@ static void pad_job_per_thread_hvx_dma(unsigned int nth, unsigned int ith, void// Main loop: pop completed DMAs, compute in VTCM with aligned HVX ops,// push dst DMA and prefetch src for the next+1 row.// -----------------------------------------------------------------------+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+for (uint32_t ir = row_start; ir < row_end; ir++) {uint8_t * dst_spad_cur = (uint8_t *) dma_queue_pop(dma).src;uint8_t * src_spad_cur = (uint8_t *) dma_queue_pop(dma).dst;@@ -236,6 +238,7 @@ static void pad_job_per_thread_hvx_dma(unsigned int nth, unsigned int ith, voidlp2, rp2, ne2,lp3, rp3, ne3);+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);if (!interior) {hvx_splat_f32_a(dst_spad_cur, 0.0f, ne0);} else {@@ -249,6 +252,7 @@ static void pad_job_per_thread_hvx_dma(unsigned int nth, unsigned int ith, voidhvx_copy_f32_ua(dst_interior, src_spad_cur, ne00);}}+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);dma_queue_push_vtcm_to_ddr(dma,dma_make_ptr(dst_ptr, dst_spad_cur),@@ -274,14 +278,11 @@ static void pad_job_per_thread_hvx_dma(unsigned int nth, unsigned int ith, voiddma_queue_flush(dma);- t2 = HAP_perf_get_qtimer_count();-- FARF(HIGH, "pad-hvx-dma %d/%d: (%ux%ux%ux%u) -> (%ux%ux%ux%u) rows %u:%u usec %u\n",+ FARF(HIGH, "pad-hvx-dma %d/%d: (%ux%ux%ux%u) -> (%ux%ux%ux%u) rows %u:%u\n",ith, nth,src->ne[0], src->ne[1], src->ne[2], src->ne[3],dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3],- row_start, row_end,- (unsigned) HAP_perf_qtimer_count_to_us(t2 - t1));+ row_start, row_end);}// ---------------------------------------------------------------------------@@ -293,8 +294,8 @@ static void pad_job_per_thread_hvx_circular(unsigned int nth, unsigned int ith,struct htp_ops_context * octx = pctx->octx;htp_pad_preamble;- uint64_t t1, t2;- t1 = HAP_perf_get_qtimer_count();+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, row_start);for (uint32_t dst_row = row_start; dst_row < row_end; dst_row++) {uint32_t i1, i2, i3;@@ -344,18 +345,17 @@ static void pad_job_per_thread_hvx_circular(unsigned int nth, unsigned int ith,}}- t2 = HAP_perf_get_qtimer_count();+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, row_start);- FARF(HIGH, "pad-hvx-circ %d/%d: (%ux%ux%ux%u) -> (%ux%ux%ux%u) rows %u:%u usec %u\n",+ FARF(HIGH, "pad-hvx-circ %d/%d: (%ux%ux%ux%u) -> (%ux%ux%ux%u) rows %u:%u\n",ith, nth,src->ne[0], src->ne[1], src->ne[2], src->ne[3],dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3],- row_start, row_end,- (unsigned) HAP_perf_qtimer_count_to_us(t2 - t1));+ row_start, row_end);}// ----------------------------------------------------------------------------// HVX + DMA circular PAD kernel — aligned, double-buffered+// HVX + DMA circular PAD kernel - aligned, double-buffered// ---------------------------------------------------------------------------static void pad_job_per_thread_hvx_circular_dma(unsigned int nth, unsigned int ith, void * data) {@@ -364,9 +364,6 @@ static void pad_job_per_thread_hvx_circular_dma(unsigned int nth, unsigned int ihtp_pad_preamble;htp_pad_dma_preamble;- uint64_t t1, t2;- t1 = HAP_perf_get_qtimer_count();-// -----------------------------------------------------------------------// Priming phase: push 2 pairs of (dummy_dst_DMA, src_DMA) to seed the// double-buffer pipeline. Every row is a real src DMA (no null DMAs).@@ -390,6 +387,8 @@ static void pad_job_per_thread_hvx_circular_dma(unsigned int nth, unsigned int i// Main loop: pop completed DMAs, assemble circular row in VTCM with// aligned HVX ops, push dst DMA and prefetch src for the next+1 row.// -----------------------------------------------------------------------+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+for (uint32_t ir = row_start; ir < row_end; ir++) {uint8_t * dst_spad_cur = (uint8_t *) dma_queue_pop(dma).src;uint8_t * src_spad_cur = (uint8_t *) dma_queue_pop(dma).dst;@@ -398,7 +397,7 @@ static void pad_job_per_thread_hvx_circular_dma(unsigned int nth, unsigned int ipad_decompose_row(ir, ne1, ne2, &i1, &i2, &i3);uint8_t * dst_ptr = (uint8_t *) dst->data + i1 * nb1 + i2 * nb2 + i3 * nb3;-+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);if (lp0 > 0) {uint8_t * dst_left = dst_spad_cur;const uint8_t * src_left = src_spad_cur + (size_t)(ne00 - (uint32_t)lp0) * type_size;@@ -430,6 +429,7 @@ static void pad_job_per_thread_hvx_circular_dma(unsigned int nth, unsigned int i}}}+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir);dma_queue_push_vtcm_to_ddr(dma,dma_make_ptr(dst_ptr, dst_spad_cur),@@ -448,14 +448,11 @@ static void pad_job_per_thread_hvx_circular_dma(unsigned int nth, unsigned int idma_queue_flush(dma);- t2 = HAP_perf_get_qtimer_count();-- FARF(HIGH, "pad-hvx-circ-dma %d/%d: (%ux%ux%ux%u) -> (%ux%ux%ux%u) rows %u:%u usec %u\n",+ FARF(HIGH, "pad-hvx-circ-dma %d/%d: (%ux%ux%ux%u) -> (%ux%ux%ux%u) rows %u:%u\n",ith, nth,src->ne[0], src->ne[1], src->ne[2], src->ne[3],dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3],- row_start, row_end,- (unsigned) HAP_perf_qtimer_count_to_us(t2 - t1));+ row_start, row_end);}int op_pad(struct htp_ops_context * octx) {@@ -489,19 +486,33 @@ int op_pad(struct htp_ops_context * octx) {const uint32_t ne00 = src0->ne[0];const uint32_t total_dst_rows = dst->ne[1] * dst->ne[2] * dst->ne[3];- const uint32_t n_threads = MIN(octx->n_threads, total_dst_rows > 0 ? total_dst_rows : 1);+ const size_t dst_row_size = (size_t)ne0 * type_size;++ uint32_t row_start = 0;+ uint32_t nrows = total_dst_rows;++ if (octx->ctx->mdev.count > 1) {+ uint32_t rows_per_chunk = 0;+ htp_tensor_mdev_rows_per_chunk(dst, type_size, (uint32_t) dst_row_size, &rows_per_chunk);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_dst_rows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ nrows = range.count;+ }++ if (nrows == 0) {+ return HTP_STATUS_OK;+ }++ const uint32_t n_threads = octx->n_threads;const size_t src_row_size = (size_t)ne00 * type_size;- const size_t dst_row_size = (size_t)ne0 * type_size;const size_t src_row_size_aligned = hex_round_up(src_row_size, VLEN);const size_t dst_row_size_aligned = hex_round_up(dst_row_size, VLEN);// Total VTCM needed: 2 buffers (ping+pong) for src and dst, per threadconst size_t vtcm_needed = (size_t)n_threads * 2 * (src_row_size_aligned + dst_row_size_aligned);- const int use_dma = (src0->nb[0] == (uint32_t)type_size) &&- (ne00 >= 512) &&- (octx->ctx->vtcm_base != NULL) &&+ const int use_dma = (src0->nb[0] == (uint32_t)type_size) && (ne00 >= 512) &&(octx->ctx->vtcm_size >= vtcm_needed);if (use_dma) {@@ -521,8 +532,9 @@ int op_pad(struct htp_ops_context * octx) {.lp1 = lp1, .rp1 = rp1,.lp2 = lp2, .rp2 = rp2,.lp3 = lp3, .rp3 = rp3,- .nrows_per_thread = (total_dst_rows + n_threads - 1) / n_threads,- .total_dst_rows = total_dst_rows,+ .nrows_per_thread = fastdiv(nrows + n_threads - 1, &octx->n_threads_div),+ .total_dst_rows = nrows,+ .row_start = row_start,.type_size = type_size,.src_row_size = src_row_size,.src_row_size_aligned = src_row_size_aligned,@@ -537,11 +549,10 @@ int op_pad(struct htp_ops_context * octx) {dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3],lp0, rp0, lp1, rp1, lp2, rp2, lp3, rp3);- if (circular && use_dma) { worker_pool_run_func(octx->ctx->worker_pool, pad_job_per_thread_hvx_circular_dma, &pctx, n_threads); }- else if (circular) { worker_pool_run_func(octx->ctx->worker_pool, pad_job_per_thread_hvx_circular, &pctx, n_threads); }- else if (use_dma) { worker_pool_run_func(octx->ctx->worker_pool, pad_job_per_thread_hvx_dma, &pctx, n_threads); }- else { worker_pool_run_func(octx->ctx->worker_pool, pad_job_per_thread_hvx, &pctx, n_threads); }+ if (circular && use_dma) { work_queue_run(octx->ctx->work_queue, pad_job_per_thread_hvx_circular_dma, &pctx, n_threads); }+ else if (circular) { work_queue_run(octx->ctx->work_queue, pad_job_per_thread_hvx_circular, &pctx, n_threads); }+ else if (use_dma) { work_queue_run(octx->ctx->work_queue, pad_job_per_thread_hvx_dma, &pctx, n_threads); }+ else { work_queue_run(octx->ctx->work_queue, pad_job_per_thread_hvx, &pctx, n_threads); }return HTP_STATUS_OK;}-diff --git a/ggml/src/ggml-hexagon/htp/repeat-ops.c b/ggml/src/ggml-hexagon/htp/repeat-ops.cindex a6f2f0ed5f..530279d650 100644--- a/ggml/src/ggml-hexagon/htp/repeat-ops.c+++ b/ggml/src/ggml-hexagon/htp/repeat-ops.c@@ -12,8 +12,10 @@#define GGML_COMMON_DECL_C#include "ggml-common.h"#include "htp-ctx.h"+#include "hex-common.h"+#include "hex-profile.h"#include "htp-ops.h"-#include "htp-ops.h"+#include "htp-tensor.h"struct htp_repeat_context {struct htp_ops_context * octx;@@ -25,6 +27,7 @@ struct htp_repeat_context {uint32_t nrows_per_thread;uint32_t total_dst_rows; // ne1 * ne2 * ne3+ uint32_t row_start;size_t type_size;};@@ -62,11 +65,11 @@ static void repeat_job_per_thread(unsigned int nth, unsigned int ith, void * datconst size_t row_bytes = ne00 * rctx->type_size;- const uint32_t row_start = rctx->nrows_per_thread * ith;- const uint32_t row_end = MIN(row_start + rctx->nrows_per_thread, rctx->total_dst_rows);+ const uint32_t row_start = rctx->row_start + rctx->nrows_per_thread * ith;+ const uint32_t row_end = MIN(row_start + rctx->nrows_per_thread, rctx->row_start + rctx->total_dst_rows);- uint64_t t1, t2;- t1 = HAP_perf_get_qtimer_count();+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, row_start);for (uint32_t dst_row = row_start; dst_row < row_end; dst_row++) {// Decompose flat dst row index into (i1, i2, i3)@@ -89,12 +92,12 @@ static void repeat_job_per_thread(unsigned int nth, unsigned int ith, void * dat}}- t2 = HAP_perf_get_qtimer_count();+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, row_start);- FARF(HIGH, "repeat %d/%d: (%ux%ux%ux%u) -> (%ux%ux%ux%u) rows %u:%u usec %u\n",+ FARF(HIGH, "repeat %d/%d: (%ux%ux%ux%u) -> (%ux%ux%ux%u) rows %u:%u\n",ith, nth, src->ne[0], src->ne[1], src->ne[2], src->ne[3],dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3],- row_start, row_end, (unsigned) HAP_perf_qtimer_count_to_us(t2 - t1));+ row_start, row_end);}int op_repeat(struct htp_ops_context * octx) {@@ -119,21 +122,39 @@ int op_repeat(struct htp_ops_context * octx) {return HTP_STATUS_NO_SUPPORT;}- const uint32_t total_dst_rows = dst->ne[1] * dst->ne[2] * dst->ne[3];- const uint32_t n_threads = MIN(octx->n_threads, total_dst_rows);-if (octx->flags & HTP_OPFLAGS_SKIP_COMPUTE) {return HTP_STATUS_OK;}+ const uint32_t total_dst_rows = dst->ne[1] * dst->ne[2] * dst->ne[3];+ const size_t dst_row_size = dst->ne[0] * type_size;++ uint32_t row_start = 0;+ uint32_t nrows = total_dst_rows;++ if (octx->ctx->mdev.count > 1) {+ uint32_t rows_per_chunk = 0;+ htp_tensor_mdev_rows_per_chunk(dst, type_size, (uint32_t) dst_row_size, &rows_per_chunk);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_dst_rows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ nrows = range.count;+ }++ if (nrows == 0) {+ return HTP_STATUS_OK;+ }++ const uint32_t n_threads = octx->n_threads;+struct htp_repeat_context rctx = {.octx = octx,.nr0 = dst->ne[0] / src0->ne[0],.nr1 = dst->ne[1] / src0->ne[1],.nr2 = dst->ne[2] / src0->ne[2],.nr3 = dst->ne[3] / src0->ne[3],- .nrows_per_thread = (total_dst_rows + n_threads - 1) / n_threads,- .total_dst_rows = total_dst_rows,+ .nrows_per_thread = fastdiv(nrows + n_threads - 1, &octx->n_threads_div),+ .total_dst_rows = nrows,+ .row_start = row_start,.type_size = type_size,};@@ -142,7 +163,7 @@ int op_repeat(struct htp_ops_context * octx) {dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3],rctx.nr0, rctx.nr1, rctx.nr2, rctx.nr3);- worker_pool_run_func(octx->ctx->worker_pool, repeat_job_per_thread, &rctx, n_threads);+ work_queue_run(octx->ctx->work_queue, repeat_job_per_thread, &rctx, n_threads);return HTP_STATUS_OK;}diff --git a/ggml/src/ggml-hexagon/htp/rope-ops.c b/ggml/src/ggml-hexagon/htp/rope-ops.cindex 0a4b31ccb1..c36976ed03 100644--- a/ggml/src/ggml-hexagon/htp/rope-ops.c+++ b/ggml/src/ggml-hexagon/htp/rope-ops.c@@ -80,6 +80,8 @@ struct htp_rope_context {size_t dst_row_stride;size_t src0_row_size_aligned;uint32_t src0_nrows;+ uint32_t row_start;+ uint32_t nrows;struct fastdiv_values div_ne2_ne1;struct fastdiv_values div_ne1;@@ -539,11 +541,11 @@ static void rope_job_f32(unsigned int nth, unsigned int ith, void * data) {htp_rope_preamble;- const uint32_t src0_nrows = rctx->src0_nrows;+ const uint32_t src0_nrows = rctx->nrows;const uint32_t src0_nrows_per_thread = rctx->src0_nrows_per_thread;- const uint32_t src0_start_row = src0_nrows_per_thread * ith;- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, src0_nrows);+ const uint32_t src0_start_row = rctx->row_start + src0_nrows_per_thread * ith;+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, rctx->row_start + src0_nrows);// no work for this threadif (src0_start_row >= src0_end_row) {@@ -706,9 +708,32 @@ static int execute_op_rope_f32(struct htp_ops_context * octx) {}const struct htp_rope_kernel_params * kparams = (const struct htp_rope_kernel_params *) octx->kernel_params;- assert(kparams->n_threads > 0);+ if (!htp_ops_context_set_n_threads(octx, kparams->n_threads)) {+ return HTP_STATUS_INVAL_PARAMS;+ }assert(octx->ctx->vtcm_size >= kparams->vtcm_size);+ const uint32_t total_rows = src0->ne[1] * src0->ne[2] * src0->ne[3];+ const size_t dst_data_row_size = dst->ne[0] * sizeof(float);++ uint32_t row_start = 0;+ uint32_t nrows = total_rows;++ if (octx->ctx->mdev.count > 1) {+ uint32_t rows_per_chunk = 0;+ htp_tensor_mdev_rows_per_chunk(dst, sizeof(float), (uint32_t) dst_data_row_size, &rows_per_chunk);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(+ total_rows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ nrows = range.count;+ }++ if (nrows == 0) {+ return HTP_STATUS_OK;+ }++ const uint32_t n_threads = octx->n_threads;+const uint32_t ne0 = dst->ne[0];const size_t src0_row_size = src0->ne[0] * sizeof(float);const size_t src0_row_stride = src0->nb[1];@@ -752,15 +777,17 @@ static int execute_op_rope_f32(struct htp_ops_context * octx) {rctx.dst_row_stride = dst_row_stride;rctx.src0_row_size_aligned = kparams->src0_row_size_aligned;- rctx.src0_nrows = kparams->src0_nrows;- rctx.src0_nrows_per_thread = kparams->src0_nrows_per_thread;+ rctx.src0_nrows = nrows;+ rctx.nrows = nrows;+ rctx.row_start = row_start;+ rctx.src0_nrows_per_thread = fastdiv(nrows + n_threads - 1, &octx->n_threads_div);rctx.div_ne2_ne1 = kparams->div_ne2_ne1;rctx.div_ne1 = kparams->div_ne1;FARF(HIGH, "rope-f32 n-rows %u n-dims %d ne0 %u ext-factor %.6f theta-scale %.6f attn-factor %.6f\n", rctx.src0_nrows, rctx.n_dims, ne0,rctx.ext_factor, rctx.theta_scale, rctx.attn_factor);- work_queue_run(octx->ctx->work_queue, rope_job_f32, &rctx, kparams->n_threads);+ work_queue_run(octx->ctx->work_queue, rope_job_f32, &rctx, n_threads);return err;}diff --git a/ggml/src/ggml-hexagon/htp/set-rows-ops.c b/ggml/src/ggml-hexagon/htp/set-rows-ops.cindex 340a497f7a..fbd5162a7c 100644--- a/ggml/src/ggml-hexagon/htp/set-rows-ops.c+++ b/ggml/src/ggml-hexagon/htp/set-rows-ops.c@@ -18,6 +18,7 @@#define GGML_COMMON_DECL_C#include "ggml-common.h"+#include "hex-common.h"#include "htp-ctx.h"#include "htp-ops.h"#include "htp-tensor.h"@@ -58,6 +59,9 @@ struct set_rows_context {const struct htp_set_rows_kernel_params * kparams;struct htp_set_rows_vtcm_layout vtcm_layout;uint8_t * vtcm_base;+ uint32_t task_start;+ uint32_t tasks;+ uint32_t tasks_per_thread;};#define SET_ROWS_THREAD_DMA_FN(TYPE_NAME, IDX_TYPE, COMPUTE_EXPR) \@@ -67,12 +71,12 @@ static void set_rows_thread_dma_##TYPE_NAME##_##IDX_TYPE(unsigned int nth, unsigconst struct htp_set_rows_kernel_params * kparams = srctx->kparams; \set_rows_preamble; \struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \- const uint32_t dr = kparams->tasks_per_thread; \- const uint32_t ir0 = dr * ith; \- if (ir0 >= kparams->total_tasks) { \+ const uint32_t dr = srctx->tasks_per_thread; \+ const uint32_t ir0 = srctx->task_start + dr * ith; \+ if (ir0 >= srctx->task_start + srctx->tasks) { \return; \} \- const uint32_t ir1 = MIN(ir0 + dr, kparams->total_tasks); \+ const uint32_t ir1 = MIN(ir0 + dr, srctx->task_start + srctx->tasks); \dma_queue * dma_queue = octx->ctx->dma[ith]; \const struct htp_set_rows_vtcm_layout * vtcm_layout = &srctx->vtcm_layout; \uint8_t * vtcm_src0 = srctx->vtcm_base + vtcm_layout->off_src0 + ith * vtcm_layout->src0_bytes_per_thread; \@@ -192,18 +196,44 @@ int op_set_rows(struct htp_ops_context * octx) {return HTP_STATUS_NO_SUPPORT;}- if (octx->src[1]->type != HTP_TYPE_I32 && octx->src[1]->type != HTP_TYPE_I64) {- return HTP_STATUS_NO_SUPPORT;+ if (octx->flags & HTP_OPFLAGS_SKIP_COMPUTE) {+ return HTP_STATUS_OK;+ }++ const struct htp_tensor * dst = octx->dst;+ const uint32_t total_tasks = kparams->total_tasks;++ uint32_t task_start = 0;+ uint32_t tasks = total_tasks;++ if (octx->ctx->mdev.count > 1) {+ const bool can_split = htp_tensor_mdev_data_aligned(dst) && (dst->nb[1] & (HTP_TENSOR_MDEV_LINE_SIZE - 1)) == 0 && !htp_tensor_is_permuted(dst);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_tasks, can_split ? 1 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ task_start = range.start;+ tasks = range.count;}+ if (tasks == 0) {+ return HTP_STATUS_OK;+ }++ if (!htp_ops_context_set_n_threads(octx, (uint32_t) kparams->n_threads)) {+ return HTP_STATUS_INVAL_PARAMS;+ }++ const uint32_t n_threads = octx->n_threads;+// l2fetch the src1 (indices) tensor in the main threadhex_l2fetch_block((const void *)octx->src[1]->data, octx->src[1]->ne[3] * octx->src[1]->nb[3]);struct set_rows_context srctx;srctx.octx = octx;srctx.kparams = kparams;+ srctx.task_start = task_start;+ srctx.tasks = tasks;+ srctx.tasks_per_thread = fastdiv(tasks + n_threads - 1, &octx->n_threads_div);- htp_set_rows_vtcm_layout_build(&srctx.vtcm_layout, octx->dst->type, ne00, kparams->n_threads);+ htp_set_rows_vtcm_layout_build(&srctx.vtcm_layout, octx->dst->type, ne00, n_threads);srctx.vtcm_base = (uint8_t *)octx->ctx->vtcm_base;work_queue_func_t q_func = NULL;@@ -216,15 +246,15 @@ int op_set_rows(struct htp_ops_context * octx) {default: return HTP_STATUS_NO_SUPPORT;}- FARF(HIGH, "set-rows: (%ux%ux%ux%u) x (%ux%ux%ux%u) -> (%ux%ux%ux%u) : src0-vtcm-size %zu dst-vtcm-size %zu n_threads %d\n",+ FARF(HIGH, "set-rows: (%ux%ux%ux%u) x (%ux%ux%ux%u) -> (%ux%ux%ux%u) : src0-vtcm-size %zu dst-vtcm-size %zu n-threads %d\n",octx->src[0]->ne[0], octx->src[0]->ne[1], octx->src[0]->ne[2], octx->src[0]->ne[3],octx->src[1]->ne[0], octx->src[1]->ne[1], octx->src[1]->ne[2], octx->src[1]->ne[3],octx->dst->ne[0], octx->dst->ne[1], octx->dst->ne[2], octx->dst->ne[3],- srctx.vtcm_layout.src0_bytes_per_thread * kparams->n_threads,- srctx.vtcm_layout.dst_bytes_per_thread * kparams->n_threads,- kparams->n_threads);+ srctx.vtcm_layout.src0_bytes_per_thread * n_threads,+ srctx.vtcm_layout.dst_bytes_per_thread * n_threads,+ n_threads);- work_queue_run(octx->ctx->work_queue, q_func, &srctx, kparams->n_threads);+ work_queue_run(octx->ctx->work_queue, q_func, &srctx, n_threads);return HTP_STATUS_OK;}diff --git a/ggml/src/ggml-hexagon/htp/softmax-ops.c b/ggml/src/ggml-hexagon/htp/softmax-ops.cindex d78bcc0eb2..2497ec7632 100644--- a/ggml/src/ggml-hexagon/htp/softmax-ops.c+++ b/ggml/src/ggml-hexagon/htp/softmax-ops.c@@ -14,9 +14,11 @@#define GGML_COMMON_DECL_C#include "ggml-common.h"+#include "hex-common.h"+#include "hex-profile.h"#include "htp-ctx.h"#include "htp-ops.h"-#include "htp-ops.h"+#include "htp-tensor.h"#define htp_softmax_preamble3 \const uint32_t ne00 = src0->ne[0]; \@@ -69,6 +71,8 @@ struct htp_softmax_context {struct fastdiv_values fastdiv_ne13; // For mask broadcastinguint32_t src0_nrows_per_thread;+ uint32_t row_start;+ uint32_t nrows;};static void apply_mask(float * restrict wp0,@@ -223,19 +227,17 @@ static void softmax_job_f32(unsigned int nth, unsigned int ith, void * data) {htp_softmax_preamble3;- const uint32_t src0_nrows = ne01 * ne02 * ne03; // src0 rows+ const uint32_t src0_nrows = smctx->nrows;const uint32_t src0_nrows_per_thread = smctx->src0_nrows_per_thread;- const uint32_t src0_start_row = src0_nrows_per_thread * ith;- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, src0_nrows);+ const uint32_t src0_start_row = smctx->row_start + src0_nrows_per_thread * ith;+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, smctx->row_start + src0_nrows);// no work for this threadif (src0_start_row >= src0_end_row) {return;}- uint64_t qt = HAP_perf_get_qtimer_count();-int is_aligned = 1;int opt_path = 0;@@ -262,6 +264,9 @@ static void softmax_job_f32(unsigned int nth, unsigned int ith, void * data) {uint32_t prev_i2 = (uint32_t)-1;float slope = 1.0f;+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, src0_start_row);+for (uint32_t r = src0_start_row; r < src0_end_row; ++r) {uint32_t i1 = fastmodulo(r, ne01, &smctx->fastdiv_ne01);uint32_t r_div_ne01 = fastdiv(r, &smctx->fastdiv_ne01);@@ -323,10 +328,11 @@ static void softmax_job_f32(unsigned int nth, unsigned int ith, void * data) {}}- qt = HAP_perf_qtimer_count_to_us(HAP_perf_get_qtimer_count() - qt);- FARF(HIGH, "softmax-f32 %d/%d: %ux%ux%ux%u (%u:%u) x %ux%ux%ux%u -> %ux%ux%ux%u : opt %u f16 %u usec %u\n", ith, nth,+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, src0_start_row);++ FARF(HIGH, "softmax-f32 %d/%d: %ux%ux%ux%u (%u:%u) x %ux%ux%ux%u -> %ux%ux%ux%u : opt %u f16 %u\n", ith, nth,ne00, ne01, ne02, ne03, src0_start_row, src0_end_row, ne10, ne11, ne12, ne13,- ne0, ne1, ne2, ne3, opt_path, smctx->use_f16, (unsigned) qt);+ ne0, ne1, ne2, ne3, opt_path, smctx->use_f16);}static int execute_op_softmax_f32(struct htp_ops_context * octx) {@@ -342,13 +348,32 @@ static int execute_op_softmax_f32(struct htp_ops_context * octx) {init_softmax_ctx(&smctx, octx);const uint32_t src0_nrows = src0->ne[1] * src0->ne[2] * src0->ne[3];- const uint32_t n_threads = MIN(octx->n_threads, src0_nrows);+ const size_t elem_size = sizeof(float);+ const size_t dst_row_size = dst->nb[1];++ uint32_t row_start = 0;+ uint32_t nrows = src0_nrows;++ if (octx->ctx->mdev.count > 1) {+ uint32_t rows_per_chunk = 0;+ htp_tensor_mdev_rows_per_chunk(dst, (uint32_t) elem_size, (uint32_t) dst_row_size, &rows_per_chunk);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(src0_nrows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ nrows = range.count;+ }- smctx.src0_nrows_per_thread = (src0_nrows + n_threads - 1) / n_threads;+ if (nrows == 0) {+ return HTP_STATUS_OK;+ }++ const uint32_t n_threads = octx->n_threads;++ smctx.src0_nrows_per_thread = fastdiv(nrows + n_threads - 1, &octx->n_threads_div);+ smctx.row_start = row_start;+ smctx.nrows = nrows;const size_t src0_row_size = src0->nb[1];const size_t src1_row_size = src0_row_size;- const size_t dst_row_size = dst->nb[1];// VTCM scratchpads for all tensors// 4 rows per thread, padded to HVX vector size@@ -383,9 +408,7 @@ static int execute_op_softmax_f32(struct htp_ops_context * octx) {octx->src1_spad.data = octx->src0_spad.data + octx->src0_spad.size; octx->src1_spad.src = NULL;octx->dst_spad.data = octx->src1_spad.data + octx->src1_spad.size; octx->dst_spad.src = NULL;- if (octx->flags & HTP_OPFLAGS_SKIP_COMPUTE) return err;-- worker_pool_run_func(octx->ctx->worker_pool, softmax_job_f32, &smctx, n_threads);+ work_queue_run(octx->ctx->work_queue, softmax_job_f32, &smctx, n_threads);return err;}diff --git a/ggml/src/ggml-hexagon/htp/solve-tri-ops.c b/ggml/src/ggml-hexagon/htp/solve-tri-ops.cindex ae8e1a5049..847a78712d 100644--- a/ggml/src/ggml-hexagon/htp/solve-tri-ops.c+++ b/ggml/src/ggml-hexagon/htp/solve-tri-ops.c@@ -1,13 +1,16 @@#pragma clang diagnostic ignored "-Wunused-but-set-variable"#include <HAP_farf.h>-#include <HAP_perf.h>#include <string.h>+#include "hex-common.h"+#include "hex-profile.h"+#define GGML_COMMON_DECL_C#include "ggml-common.h"#include "htp-ctx.h"#include "htp-ops.h"+#include "htp-tensor.h"#include "hvx-types.h"#include "hvx-utils.h"@@ -15,6 +18,7 @@ struct htp_solve_tri_context {struct htp_ops_context * octx;uint32_t jobs_per_thread;uint32_t total_jobs;+ uint32_t job_start;uint32_t k_chunks;uint32_t col_block;};@@ -89,11 +93,11 @@ static void solve_tri_batch_thread_f32(unsigned int nth, unsigned int ith, voidconst uint32_t col_block = VLEN_FP32;const uint32_t k_full = (k / col_block) * col_block;- const uint32_t start_batch = sctx->jobs_per_thread * ith;- const uint32_t end_batch = MIN(start_batch + sctx->jobs_per_thread, sctx->total_jobs);+ const uint32_t start_batch = sctx->job_start + sctx->jobs_per_thread * ith;+ const uint32_t end_batch = MIN(start_batch + sctx->jobs_per_thread, sctx->job_start + sctx->total_jobs);- uint64_t t1, t2;- t1 = HAP_perf_get_qtimer_count();+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) start_batch);for (uint32_t batch = start_batch; batch < end_batch; ++batch) {const uint32_t i03 = batch / ne02;@@ -127,11 +131,10 @@ static void solve_tri_batch_thread_f32(unsigned int nth, unsigned int ith, void}}- t2 = HAP_perf_get_qtimer_count();+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) end_batch);- FARF(HIGH, "solve-tri-batch %d/%d: A=(%ux%u) B=(%ux%u) batch %u:%u usec %u\n",- ith, nth, n, n, k, n, start_batch, end_batch,- (unsigned) HAP_perf_qtimer_count_to_us(t2 - t1));+ FARF(HIGH, "solve-tri-batch %d/%d: A=(%ux%u) B=(%ux%u) batch %u:%u\n",+ ith, nth, n, n, k, n, start_batch, end_batch);}// Chunk-level thread: each job is one (batch, col_chunk) pair.@@ -148,11 +151,11 @@ static void solve_tri_chunk_thread_f32(unsigned int nth, unsigned int ith, voidconst uint32_t ne02 = src0->ne[2];- const uint32_t start_job = sctx->jobs_per_thread * ith;- const uint32_t end_job = MIN(start_job + sctx->jobs_per_thread, sctx->total_jobs);+ const uint32_t start_job = sctx->job_start + sctx->jobs_per_thread * ith;+ const uint32_t end_job = MIN(start_job + sctx->jobs_per_thread, sctx->job_start + sctx->total_jobs);- uint64_t t1, t2;- t1 = HAP_perf_get_qtimer_count();+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) start_job);for (uint32_t job = start_job; job < end_job; ++job) {const uint32_t batch = job / sctx->k_chunks;@@ -161,16 +164,14 @@ static void solve_tri_chunk_thread_f32(unsigned int nth, unsigned int ith, voidconst uint32_t i03 = batch / ne02;const uint32_t i02 = batch - i03 * ne02;- const uint32_t col0 = chunk * sctx->col_block;- const uint32_t coln = MIN(sctx->col_block, k - col0);-const float * A_batch =(const float *) ((const uint8_t *) (uintptr_t) src0->data + i02 * src0->nb[2] + i03 * src0->nb[3]);const float * B_batch =(const float *) ((const uint8_t *) (uintptr_t) src1->data + i02 * src1->nb[2] + i03 * src1->nb[3]);float * X_batch = (float *) ((uint8_t *) (uintptr_t) dst->data + i02 * dst->nb[2] + i03 * dst->nb[3]);- const bool use_hvx = (coln >= 8);+ const uint32_t col0 = chunk * sctx->col_block;+ const uint32_t coln = MIN(sctx->col_block, k - col0);for (uint32_t row = 0; row < n; ++row) {const float diag = A_batch[row * n + row];@@ -179,7 +180,7 @@ static void solve_tri_chunk_thread_f32(unsigned int nth, unsigned int ith, voidconst float * A_row = A_batch + row * n;const float * B_row = B_batch + row * k;- if (use_hvx) {+ if (coln >= 8) {solve_tri_row_hvx(A_row, B_row, X_batch, row, k, col0, coln, inv_diag);} else {solve_tri_row_scalar(A_row, B_row, X_batch, row, k, col0, coln, inv_diag);@@ -187,11 +188,10 @@ static void solve_tri_chunk_thread_f32(unsigned int nth, unsigned int ith, void}}- t2 = HAP_perf_get_qtimer_count();+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) end_job);- FARF(HIGH, "solve-tri-chunk %d/%d: A=(%ux%u) B=(%ux%u) job %u:%u usec %u\n",- ith, nth, n, n, k, n, start_job, end_job,- (unsigned) HAP_perf_qtimer_count_to_us(t2 - t1));+ FARF(HIGH, "solve-tri-chunk %d/%d: A=(%ux%u) B=(%ux%u) jobs %u:%u\n",+ ith, nth, n, n, k, n, start_job, end_job);}int op_solve_tri(struct htp_ops_context * octx) {@@ -235,32 +235,64 @@ int op_solve_tri(struct htp_ops_context * octx) {dst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3], batched);if (batched) {+ uint32_t job_start = 0;+ uint32_t njobs = total_batches;++ if (octx->ctx->mdev.count > 1) {+ const uint32_t batch_size = dst->nb[2];+ const uint32_t batches_per_chunk = (batch_size > 0) ? (HEX_L2_LINE_SIZE / hex_gcd_u32(batch_size, HEX_L2_LINE_SIZE)) : 1;+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_batches, htp_tensor_mdev_data_aligned(dst) ? batches_per_chunk : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ job_start = range.start;+ njobs = range.count;+ }++ if (njobs == 0) {+ return HTP_STATUS_OK;+ }+// Batch-level parallelism- const uint32_t n_threads = MIN((uint32_t) octx->n_threads, total_batches);+ const uint32_t n_threads = octx->n_threads;struct htp_solve_tri_context sctx = {.octx = octx,- .jobs_per_thread = (total_batches + n_threads - 1) / n_threads,- .total_jobs = total_batches,+ .jobs_per_thread = fastdiv(njobs + n_threads - 1, &octx->n_threads_div),+ .total_jobs = njobs,+ .job_start = job_start,.k_chunks = k_chunks,.col_block = col_block,};- worker_pool_run_func(octx->ctx->worker_pool, solve_tri_batch_thread_f32, &sctx, n_threads);+ work_queue_run(octx->ctx->work_queue, solve_tri_batch_thread_f32, &sctx, n_threads);} else {// Chunk-level parallelismconst uint32_t total_jobs = total_batches * k_chunks;- const uint32_t n_threads = MIN((uint32_t) octx->n_threads, MAX(total_jobs, 1));++ uint32_t job_start = 0;+ uint32_t njobs = total_jobs;++ if (octx->ctx->mdev.count > 1) {+ const bool can_split = htp_tensor_mdev_data_aligned(dst) && ((dst->nb[1] & (HTP_TENSOR_MDEV_LINE_SIZE - 1)) == 0);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_jobs, can_split ? 1 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ job_start = range.start;+ njobs = range.count;+ }++ if (njobs == 0) {+ return HTP_STATUS_OK;+ }++ const uint32_t n_threads = octx->n_threads;struct htp_solve_tri_context sctx = {.octx = octx,- .jobs_per_thread = (total_jobs + n_threads - 1) / n_threads,- .total_jobs = total_jobs,+ .jobs_per_thread = fastdiv(njobs + n_threads - 1, &octx->n_threads_div),+ .total_jobs = njobs,+ .job_start = job_start,.k_chunks = k_chunks,.col_block = col_block,};- worker_pool_run_func(octx->ctx->worker_pool, solve_tri_chunk_thread_f32, &sctx, n_threads);+ work_queue_run(octx->ctx->work_queue, solve_tri_chunk_thread_f32, &sctx, n_threads);}return HTP_STATUS_OK;diff --git a/ggml/src/ggml-hexagon/htp/ssm-conv.c b/ggml/src/ggml-hexagon/htp/ssm-conv.cindex a48bc9ed86..bef1425368 100644--- a/ggml/src/ggml-hexagon/htp/ssm-conv.c+++ b/ggml/src/ggml-hexagon/htp/ssm-conv.c@@ -4,7 +4,6 @@#include <HAP_farf.h>#include <HAP_mem.h>-#include <HAP_perf.h>#include <HAP_ps.h>#include <hexagon_protos.h>#include <hexagon_types.h>@@ -16,8 +15,9 @@#include "ggml-common.h"#include "htp-ctx.h"#include "hex-dma.h"+#include "hex-profile.h"#include "htp-ops.h"-#include "htp-ops.h"+#include "htp-tensor.h"#include "hvx-utils.h"#define htp_ssm_conv_tensors_preamble \@@ -63,6 +63,8 @@ struct htp_ssm_conv_context {uint32_t nrows_per_thread;uint32_t d_inner_tile;uint64_t t_start;+ uint32_t row_start;+ uint32_t nrows;};#define htp_ssm_conv_preamble \@@ -75,9 +77,6 @@ struct htp_ssm_conv_context {static void ssm_conv_thread_f32_f32(unsigned int nth, unsigned int ith, void *data) {htp_ssm_conv_preamble;- uint64_t t1, t2;- t1 = HAP_perf_get_qtimer_count();-const uint32_t d_conv = src1->ne[0];const uint32_t d_inner = src0->ne[1];const uint32_t n_t = dst->ne[1];@@ -95,14 +94,17 @@ static void ssm_conv_thread_f32_f32(unsigned int nth, unsigned int ith, void *da// Calculate row range for this threadconst uint32_t d_inner_per_thread = scctx->nrows_per_thread;- const uint32_t d_inner_start = d_inner_per_thread * ith;- const uint32_t d_inner_end = MIN(d_inner_start + d_inner_per_thread, d_inner);+ const uint32_t d_inner_start = scctx->row_start + d_inner_per_thread * ith;+ const uint32_t d_inner_end = MIN(d_inner_start + d_inner_per_thread, scctx->row_start + scctx->nrows);// No work for this threadif (d_inner_start >= d_inner_end) {return;}+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) d_inner_start);+for (uint32_t i3 = 0; i3 < n_s; ++i3) {for (uint32_t i2 = 0; i2 < n_t; ++i2) {for (uint32_t i1 = d_inner_start; i1 < d_inner_end; ++i1) {@@ -121,12 +123,12 @@ static void ssm_conv_thread_f32_f32(unsigned int nth, unsigned int ith, void *da}}- t2 = HAP_perf_get_qtimer_count();+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) d_inner_end);- FARF(HIGH, "ssm-conv-f32 %d/%d: %ux%ux%ux%u (%u:%u) * %ux%ux%ux%u -> %ux%ux%ux%u usec %u\n",+ FARF(HIGH, "ssm-conv-f32 %d/%d: %ux%ux%ux%u (%u:%u) * %ux%ux%ux%u -> %ux%ux%ux%u\n",ith, nth, src0->ne[0], src0->ne[1], src0->ne[2], src0->ne[3], d_inner_start, d_inner_end,src1->ne[0], src1->ne[1], src1->ne[2], src1->ne[3], dst->ne[0], dst->ne[1],- dst->ne[2], dst->ne[3], (unsigned) HAP_perf_qtimer_count_to_us(t2 - t1));+ dst->ne[2], dst->ne[3]);}@@ -257,9 +259,6 @@ static inline void transpose_src0_block(const float * src0_block,static void ssm_conv_thread_f32_f32_hvx(unsigned int nth, unsigned int ith, void *data) {htp_ssm_conv_preamble;- uint64_t t1, t2;- t1 = HAP_perf_get_qtimer_count();-const uint32_t d_conv = src1->ne[0];const uint32_t d_inner = src0->ne[1];const uint32_t n_t = dst->ne[1];@@ -273,13 +272,16 @@ static void ssm_conv_thread_f32_f32_hvx(unsigned int nth, unsigned int ith, voidconst uint32_t dst_stride_seq = dst->nb[2] / sizeof(float);const uint32_t dr = scctx->nrows_per_thread;- const uint32_t ir0 = dr * ith;- const uint32_t ir1 = MIN(ir0 + dr, d_inner);+ const uint32_t ir0 = scctx->row_start + dr * ith;+ const uint32_t ir1 = MIN(ir0 + dr, scctx->row_start + scctx->nrows);if (ir0 >= ir1) {return;}+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir0);+const uint32_t d_inner_per_thread = ir1 - ir0;const uint32_t d_inner_stride = scctx->nrows_per_thread;const uint32_t d_inner_tile = scctx->d_inner_tile;@@ -319,97 +321,118 @@ static void ssm_conv_thread_f32_f32_hvx(unsigned int nth, unsigned int ith, voidHVX_Vector w = *(const HVX_Vector *) (src1_T + j * d_inner_stride + tile_off + cb);acc = Q6_Vqf32_vadd_Vqf32Vqf32(acc, Q6_Vqf32_vmpy_VsfVsf(x, w));}- HVX_Vector res = Q6_Vsf_equals_Vqf32(acc);- float * dst_ptr = dst_data + i3 * dst_stride_seq + t * dst_stride_token + (ir0 + tile_off + cb);+ HVX_Vector y = Q6_Vsf_equals_Vqf32(acc);++ float * dst_ptr = dst_data + (ir0 + tile_off + cb) + t * dst_stride_token + i3 * dst_stride_seq;if (cb_n == C_TILE) {- *(HVX_UVector *) dst_ptr = res;+ *(HVX_UVector *) dst_ptr = y;} else {- hvx_vec_store_u(dst_ptr, cb_n * sizeof(float), res);+ hvx_vec_store_u(dst_ptr, cb_n * sizeof(float), y);}}}}}- t2 = HAP_perf_get_qtimer_count();+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) ir1);- FARF(HIGH, "ssm-conv-f32-hvx %d/%d: %ux%ux%ux%u (%u:%u) tile=%u * %ux%ux%ux%u -> %ux%ux%ux%u usec %u\n",- ith, nth, src0->ne[0], src0->ne[1], src0->ne[2], src0->ne[3], ir0, ir1, d_inner_tile,+ FARF(HIGH, "ssm-conv-f32-hvx %d/%d: %ux%ux%ux%u (%u:%u) * %ux%ux%ux%u -> %ux%ux%ux%u\n",+ ith, nth, src0->ne[0], src0->ne[1], src0->ne[2], src0->ne[3], ir0, ir1,src1->ne[0], src1->ne[1], src1->ne[2], src1->ne[3], dst->ne[0], dst->ne[1],- dst->ne[2], dst->ne[3], (unsigned) HAP_perf_qtimer_count_to_us(t2 - t1));+ dst->ne[2], dst->ne[3]);}int op_ssm_conv_f32(struct htp_ops_context * octx) {- htp_ssm_conv_tensors_preamble;+ const struct htp_tensor * src0 = octx->src[0];+ const struct htp_tensor * src1 = octx->src[1];+ const struct htp_tensor * dst = octx->dst;if (src0->type != HTP_TYPE_F32 || src1->type != HTP_TYPE_F32 || dst->type != HTP_TYPE_F32) {- FARF(ERROR, "ssm_conv: only (F32 x F32 -> F32) OPs supported");return HTP_STATUS_NO_SUPPORT;}- struct htp_ssm_conv_context scctx = { 0 };- scctx.octx = octx;-const uint32_t d_conv = src1->ne[0];const uint32_t d_inner = src0->ne[1];const uint32_t n_t = dst->ne[1]; // tokens per sequenceconst uint32_t n_s = dst->ne[2]; // number of sequences in the batch- const uint32_t n_threads = MIN(octx->n_threads, d_inner);+ if (octx->flags & HTP_OPFLAGS_SKIP_COMPUTE) {+ return HTP_STATUS_OK;+ }- if (!(octx->flags & HTP_OPFLAGS_SKIP_COMPUTE)) {- uint32_t use_hvx = 0;- if (d_inner >= VLEN_FP32 && n_t >= VLEN_FP32) {- use_hvx = 1;- }+ uint32_t row_start = 0;+ uint32_t nrows = d_inner;++ if (octx->ctx->mdev.count > 1) {+ const uint32_t elems_per_chunk = VLEN_FP32;+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(d_inner, htp_tensor_mdev_data_aligned(dst) ? elems_per_chunk : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ nrows = range.count;+ }++ if (nrows == 0) {+ return HTP_STATUS_OK;+ }++ const uint32_t n_threads = octx->n_threads;++ struct htp_ssm_conv_context scctx = { 0 };+ scctx.octx = octx;+ scctx.row_start = row_start;+ scctx.nrows = nrows;++ uint32_t use_hvx = 0;+ if (nrows >= VLEN_FP32 && n_t >= VLEN_FP32) {+ use_hvx = 1;+ }++ const uint32_t raw_rpt = fastdiv(nrows + n_threads - 1, &octx->n_threads_div);+ scctx.nrows_per_thread = hex_round_up(raw_rpt, VLEN_FP32);- scctx.nrows_per_thread = hex_round_up((d_inner + n_threads - 1) / n_threads, VLEN_FP32);+ const uint32_t d_inner_per_thread = scctx.nrows_per_thread;+ const uint32_t ncs = src0->ne[0];- const uint32_t d_inner_per_thread = scctx.nrows_per_thread;- const uint32_t ncs = src0->ne[0];+ const uint32_t src1_T_size = hex_round_up(d_conv * d_inner_per_thread * sizeof(float), 256);+ const uint32_t src0_T_max = HTP_SSM_CONV_VTCM_BUDGET > src1_T_size ? HTP_SSM_CONV_VTCM_BUDGET - src1_T_size : 0;- const uint32_t src1_T_size = hex_round_up(d_conv * d_inner_per_thread * sizeof(float), 256);- const uint32_t src0_T_max = HTP_SSM_CONV_VTCM_BUDGET > src1_T_size ? HTP_SSM_CONV_VTCM_BUDGET - src1_T_size : 0;+ uint32_t d_inner_tile = (src0_T_max / sizeof(float)) / ncs;+ d_inner_tile -= (d_inner_tile % VLEN_FP32);+ if (d_inner_tile == 0) {+ FARF(HIGH, "ssm_conv-f32: inner tile rounds to 0 (ncs=%u), falling back to scalar\n", ncs);+ use_hvx = 0;+ } else {+ scctx.d_inner_tile = d_inner_tile;- uint32_t d_inner_tile = (src0_T_max / sizeof(float)) / ncs;- d_inner_tile -= (d_inner_tile % VLEN_FP32);- if (d_inner_tile == 0) {- FARF(HIGH, "ssm_conv-f32: inner tile rounds to 0 (ncs=%u), falling back to scalar\n", ncs);+ octx->src0_spad.size_per_thread = hex_round_up(d_inner_tile * ncs * sizeof(float), 256);+ octx->src1_spad.size_per_thread = src1_T_size;+ octx->dst_spad.size_per_thread = 0;++ octx->src0_spad.size = octx->src0_spad.size_per_thread * n_threads;+ octx->src1_spad.size = octx->src1_spad.size_per_thread * n_threads;+ octx->dst_spad.size = 0;++ octx->src0_spad.data = octx->ctx->vtcm_base;+ octx->src1_spad.data = octx->src0_spad.data + octx->src0_spad.size;+ octx->src0_spad.src = NULL;+ octx->src1_spad.src = NULL;++ const size_t total_spad = octx->src0_spad.size + octx->src1_spad.size;+ if (total_spad > octx->ctx->vtcm_size) {+ FARF(HIGH, "ssm_conv-f32: scratchpad %zu exceeds VTCM %zu, falling back to scalar\n",+ total_spad, octx->ctx->vtcm_size);use_hvx = 0;- } else {- scctx.d_inner_tile = d_inner_tile;-- octx->src0_spad.size_per_thread = hex_round_up(d_inner_tile * ncs * sizeof(float), 256);- octx->src1_spad.size_per_thread = src1_T_size;- octx->dst_spad.size_per_thread = 0;-- octx->src0_spad.size = octx->src0_spad.size_per_thread * n_threads;- octx->src1_spad.size = octx->src1_spad.size_per_thread * n_threads;- octx->dst_spad.size = 0;-- octx->src0_spad.data = octx->ctx->vtcm_base;- octx->src1_spad.data = octx->src0_spad.data + octx->src0_spad.size;- octx->src0_spad.src = NULL;- octx->src1_spad.src = NULL;-- const size_t total_spad = octx->src0_spad.size + octx->src1_spad.size;- if (total_spad > octx->ctx->vtcm_size) {- FARF(HIGH, "ssm_conv-f32: scratchpad %zu exceeds VTCM %zu, falling back to scalar\n",- total_spad, octx->ctx->vtcm_size);- use_hvx = 0;- }}+ }- FARF(HIGH, "ssm-conv-f32: (%ux%ux%ux%u) x (%ux%ux%ux%u) -> (%ux%ux%ux%u) : use_hvx %d\n", src0->ne[0],- src0->ne[1], src0->ne[2], src0->ne[3], src1->ne[0], src1->ne[1], src1->ne[2], src1->ne[3], dst->ne[0],- dst->ne[1], dst->ne[2], dst->ne[3], use_hvx);+ FARF(HIGH, "ssm-conv-f32: (%ux%ux%ux%u) x (%ux%ux%ux%u) -> (%ux%ux%ux%u) : use_hvx %d\n", src0->ne[0],+ src0->ne[1], src0->ne[2], src0->ne[3], src1->ne[0], src1->ne[1], src1->ne[2], src1->ne[3], dst->ne[0],+ dst->ne[1], dst->ne[2], dst->ne[3], use_hvx);- if (use_hvx) {- worker_pool_run_func(octx->ctx->worker_pool, ssm_conv_thread_f32_f32_hvx, &scctx, n_threads);- } else {- worker_pool_run_func(octx->ctx->worker_pool, ssm_conv_thread_f32_f32, &scctx, n_threads);- }+ if (use_hvx) {+ work_queue_run(octx->ctx->work_queue, ssm_conv_thread_f32_f32_hvx, &scctx, n_threads);+ } else {+ work_queue_run(octx->ctx->work_queue, ssm_conv_thread_f32_f32, &scctx, n_threads);}return HTP_STATUS_OK;diff --git a/ggml/src/ggml-hexagon/htp/sum-rows-ops.c b/ggml/src/ggml-hexagon/htp/sum-rows-ops.cindex 874c41ab2a..faf716b4bc 100644--- a/ggml/src/ggml-hexagon/htp/sum-rows-ops.c+++ b/ggml/src/ggml-hexagon/htp/sum-rows-ops.c@@ -13,35 +13,38 @@#define GGML_COMMON_DECL_C#include "ggml-common.h"+#include "hex-common.h"+#include "hex-profile.h"#include "htp-ctx.h"#include "htp-ops.h"-#include "htp-ops.h"+#include "htp-tensor.h"#define sum_rows_preamble \const struct htp_tensor *src0 = octx->src[0]; \const struct htp_tensor *dst = octx->dst; \\- const uint32_t ne00 = src0->ne[0]; \- const uint32_t ne01 = src0->ne[1]; \- const uint32_t ne02 = src0->ne[2]; \- const uint32_t ne03 = src0->ne[3]; \- \- const uint32_t nb00 = src0->nb[0]; \- const uint32_t nb01 = src0->nb[1]; \- const uint32_t nb02 = src0->nb[2]; \- const uint32_t nb03 = src0->nb[3]; \- \- const uint32_t ne0 = dst->ne[0]; \- const uint32_t ne1 = dst->ne[1]; \- const uint32_t ne2 = dst->ne[2]; \- const uint32_t ne3 = dst->ne[3]; \- \- const uint32_t nb0 = dst->nb[0]; \- const uint32_t nb1 = dst->nb[1]; \- const uint32_t nb2 = dst->nb[2]; \- const uint32_t nb3 = dst->nb[3]; \+ const uint32_t ne00 = src0->ne[0]; \+ const uint32_t ne01 = src0->ne[1]; \+ const uint32_t ne02 = src0->ne[2]; \+ const uint32_t ne03 = src0->ne[3]; \+ \+ const uint32_t nb00 = src0->nb[0]; \+ const uint32_t nb01 = src0->nb[1]; \+ const uint32_t nb02 = src0->nb[2]; \+ const uint32_t nb03 = src0->nb[3]; \+ \+ const uint32_t ne0 = dst->ne[0]; \+ const uint32_t ne1 = dst->ne[1]; \+ const uint32_t ne2 = dst->ne[2]; \+ const uint32_t ne3 = dst->ne[3]; \+ \+ const uint32_t nb0 = dst->nb[0]; \+ const uint32_t nb1 = dst->nb[1]; \+ const uint32_t nb2 = dst->nb[2]; \+ const uint32_t nb3 = dst->nb[3]; \struct sum_rows_context {+ struct htp_ops_context * octx;const uint8_t * src_data;uint8_t * dst_data;uint32_t ne00;@@ -76,6 +79,9 @@ static void sum_rows_thread_f32(unsigned int nth, unsigned int ith, void *data)// Calculate actual number of rows for this threadconst uint32_t n_rows = end_row - start_row;+ struct htp_thread_trace * tr = &smctx->octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) start_row);+for (uint32_t ir = 0; ir < n_rows; ir++) {const float * restrict src_local = src_th + (ir * (src_stride / sizeof(float)));@@ -89,6 +95,8 @@ static void sum_rows_thread_f32(unsigned int nth, unsigned int ith, void *data)dst_th[ir] = hvx_reduce_sum_f32((const uint8_t *) src_local, ne00);}}++ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) start_row);}int op_sum_rows(struct htp_ops_context * octx) {@@ -102,9 +110,26 @@ int op_sum_rows(struct htp_ops_context * octx) {return HTP_STATUS_OK;}- const uint32_t src0_nrows = ne01 * ne02 * ne03;- const uint32_t n_threads = MIN(octx->n_threads, src0_nrows);- const uint32_t rows_per_thread = (src0_nrows + n_threads - 1) / n_threads;+ const uint32_t src0_nrows = ne01 * ne02 * ne03;+ const size_t dst_data_row_size = dst->ne[0] * sizeof(float);++ uint32_t row_start = 0;+ uint32_t nrows = src0_nrows;++ if (octx->ctx->mdev.count > 1) {+ uint32_t rows_per_chunk = 0;+ htp_tensor_mdev_rows_per_chunk(dst, sizeof(float), (uint32_t) dst_data_row_size, &rows_per_chunk);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(src0_nrows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ nrows = range.count;+ }++ if (nrows == 0) {+ return HTP_STATUS_OK;+ }++ const uint32_t n_threads = octx->n_threads;+ const uint32_t rows_per_thread = fastdiv(nrows + n_threads - 1, &octx->n_threads_div);bool opt_path = false;if ((0 == hex_is_aligned((void *) src0->data, VLEN)) && !(nb01 & (VLEN - 1))) {@@ -112,17 +137,18 @@ int op_sum_rows(struct htp_ops_context * octx) {}struct sum_rows_context smctx = {- .src_data = (const uint8_t *) src0->data,- .dst_data = (uint8_t *) dst->data,+ .octx = octx,+ .src_data = (const uint8_t *) src0->data + row_start * nb01,+ .dst_data = (uint8_t *) dst->data + row_start * nb1,.ne00 = ne00,.src_stride = nb01,.dst_stride = nb1,.rows_per_thread = rows_per_thread,- .total_rows = src0_nrows,+ .total_rows = nrows,.opt_path = opt_path,};- worker_pool_run_func(octx->ctx->worker_pool, sum_rows_thread_f32, &smctx, n_threads);+ work_queue_run(octx->ctx->work_queue, sum_rows_thread_f32, &smctx, n_threads);return HTP_STATUS_OK;}diff --git a/ggml/src/ggml-hexagon/htp/unary-ops.c b/ggml/src/ggml-hexagon/htp/unary-ops.cindex 7850ab27e0..cb82bfa3c2 100644--- a/ggml/src/ggml-hexagon/htp/unary-ops.c+++ b/ggml/src/ggml-hexagon/htp/unary-ops.c@@ -46,6 +46,7 @@ struct htp_unary_context {uint32_t block;uint32_t src0_nrows;uint32_t src0_nrows_per_thread;+ uint32_t row_start;uint32_t nc;uint32_t col_tile; // tiled modebool broadcast_weight;@@ -496,7 +497,7 @@ static void tri_f32(const float * restrict src,}if (boundary > ne0) boundary = ne0;- // Full HVX vectors — each starts at a 128-byte aligned offset+ // Full HVX vectors - each starts at a 128-byte aligned offsetfor (uint32_t i = 0; i < nvec; i++) {const uint32_t vec_start = i * VLEN_FP32;const uint32_t vec_end = vec_start + VLEN_FP32;@@ -563,7 +564,7 @@ static void softplus_f32(const float * restrict src,for (uint32_t i = 0; i < ne0; i++) {float x = src_f[i];- // For x > 20: softplus(x) ≈ x (avoids exp overflow)+ // For x > 20: softplus(x) ~ x (avoids exp overflow)dst_f[i] = (x > 20.0f) ? x : logf(1.0f + expf(x));}}@@ -661,8 +662,8 @@ static void unary_task_##SUFFIX##_##NAME(unsigned int nth, unsigned int ith, voiconst size_t dst_row_size_aligned = uctx->dst_row_size_aligned; \\const uint32_t src0_nrows = uctx->src0_nrows; \- const uint32_t src0_start_row = src0_nrows_per_thread * ith; \- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, src0_nrows); \+ const uint32_t src0_start_row = uctx->row_start + src0_nrows_per_thread * ith; \+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, uctx->row_start + src0_nrows); \\if (src0_start_row >= src0_end_row) { \return; \@@ -833,124 +834,126 @@ DEFINE_UNARY_TASK_IMPL(unary_abs, _Float16, f16, false, false, abs_f16(src0_vtcmDEFINE_UNARY_TASK_IMPL(unary_log, _Float16, f16, false, false, log_f16(src0_vtcm, dst_vtcm, block_size, uctx))// Apply a pointwise unary op to one column tile that is already in VTCM.-#define DEFINE_UNARY_TILED_TASK(NAME, IS_TRI, CORE_TILE_EXPR) \-static void unary_task_f32_tiled_##NAME(unsigned int nth, unsigned int ith, void * data) { \- const struct htp_unary_context * uctx = (const struct htp_unary_context *) data; \- struct htp_ops_context * octx = uctx->octx; \- const struct htp_tensor * src = octx->src[0]; \- const struct htp_tensor * dst = octx->dst; \- struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \- \- htp_unary_preamble; \- \- int32_t * op_params = octx->op_params; \- const uint32_t col_tile = uctx->col_tile; \- \- const uint32_t src0_nrows = uctx->src0_nrows; \- const uint32_t src0_start_row = uctx->src0_nrows_per_thread * ith; \- const uint32_t src0_end_row = MIN(src0_start_row + uctx->src0_nrows_per_thread, src0_nrows); \- \- if (src0_start_row >= src0_end_row) { \- return; \- } \- \- const uint8_t * restrict data_src = uctx->data_src0; \- uint8_t * restrict data_dst = uctx->data_dst; \- \- uint8_t * src0_vtcm_data = uctx->vtcm_src0 + (ith * uctx->vtcm_src0_size_per_thread); \- uint8_t * dst_vtcm_data = uctx->vtcm_dst + (ith * uctx->vtcm_dst_size_per_thread); \- \- const size_t src0_half = uctx->src0_vtcm_half_size; \- const size_t dst_half = uctx->dst_vtcm_half_size; \- \- dma_queue * dmaq = octx->ctx->dma[ith]; \- \- const struct fastdiv_values * div_ne01 = &uctx->kparams->div_ne01; \- const struct fastdiv_values * div_ne02 = &uctx->kparams->div_ne02; \- const struct fastdiv_values * div_ne012 = &uctx->kparams->div_ne012; \- const struct fastdiv_values * div_tpr = &uctx->kparams->div_tpr; \- \- const uint32_t tiles_per_row = (ne0 + col_tile - 1) / col_tile; \- const int32_t tri_ttype = (IS_TRI) ? op_params[0] : 0; \- \- const bool src0_contig = (nb02 == (size_t)ne01 * nb01) && \- (nb03 == (size_t)ne02 * nb02); \- const bool dst_contig = (nb2 == (size_t)ne1 * nb1) && \- (nb3 == (size_t)ne2 * nb2); \- \- const uint32_t total_tiles = (src0_end_row - src0_start_row) * tiles_per_row; \- \- for (uint32_t t = 0, vtcm_idx = 0; t < total_tiles && vtcm_idx < 2; t++, vtcm_idx++) { \- const uint32_t row = src0_start_row + t / tiles_per_row; \- const uint32_t col = (t % tiles_per_row) * col_tile; \- const uint32_t tw = MIN(col_tile, ne0 - col); \- const size_t tb = (size_t) tw * sizeof(float); \- const size_t soff = (src0_contig ? (row * nb01) : \- unary_row_offset(row, ne01, ne02, div_ne01, div_ne02, div_ne012, nb01, nb02, nb03)) +\- (size_t) col * sizeof(float); \- \- dma_queue_push(dmaq, dma_make_ptr(data_dst, dst_vtcm_data + (vtcm_idx * dst_half)), 0, 0, 0, 0); \- dma_queue_push(dmaq, dma_make_ptr(src0_vtcm_data + (vtcm_idx * src0_half), data_src + soff), tb, tb, tb, 1);\- } \- \- uint32_t row = src0_start_row; \- uint32_t col = 0; \- uint32_t tile_in_row = 0; \- uint32_t i01 = fastmodulo(row, ne01, div_ne01); \- \- uint32_t prow = src0_start_row + fastdiv(2, div_tpr); \- uint32_t pcol = fastmodulo(2, tiles_per_row, div_tpr) * col_tile; \- uint32_t ptile_in_row = fastmodulo(2, tiles_per_row, div_tpr); \- \- for (uint32_t t = 0; t < total_tiles; t++) { \- uint8_t * dst_vtcm = (uint8_t *) dma_queue_pop(dmaq).src; \- uint8_t * src_vtcm = (uint8_t *) dma_queue_pop(dmaq).dst; \- \- const uint32_t tw = MIN(col_tile, ne0 - col); \- \- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, t); \- CORE_TILE_EXPR; \- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, t); \- \- const size_t doff = (dst_contig ? (row * nb1) : \- unary_row_offset(row, ne1, ne2, div_ne01, div_ne02, div_ne012, nb1, nb2, nb3)) + \- (size_t) col * sizeof(float); \- const size_t tb = (size_t) tw * sizeof(float); \- dma_queue_push(dmaq, dma_make_ptr(data_dst + doff, dst_vtcm), tb, tb, tb, 1); \- \- const uint32_t pt = t + 2; \- if (pt < total_tiles) { \- const uint32_t ptw = MIN(col_tile, ne0 - pcol); \- const size_t ptb = (size_t) ptw * sizeof(float); \- const size_t psoff = (src0_contig ? (prow * nb01) : \- unary_row_offset(prow, ne01, ne02, div_ne01, div_ne02, div_ne012, nb01, nb02, \- nb03)) + \- (size_t) pcol * sizeof(float); \- dma_queue_push(dmaq, dma_make_ptr(src_vtcm, data_src + psoff), ptb, ptb, ptb, 1); \- } \- \- tile_in_row++; \- col += col_tile; \- if (tile_in_row == tiles_per_row) { \- tile_in_row = 0; \- col = 0; \- row++; \- i01++; \- if (i01 == ne01) { \- i01 = 0; \- } \- } \- \- ptile_in_row++; \- pcol += col_tile; \- if (ptile_in_row == tiles_per_row) { \- ptile_in_row = 0; \- pcol = 0; \- prow++; \- } \- } \- \- dma_queue_flush(dmaq); \+#define DEFINE_UNARY_TILED_TASK(NAME, IS_TRI, CORE_TILE_EXPR) \+static void unary_task_f32_tiled_##NAME(unsigned int nth, unsigned int ith, void * data) { \+ const struct htp_unary_context * uctx = (const struct htp_unary_context *) data; \+ struct htp_ops_context * octx = uctx->octx; \+ const struct htp_tensor * src = octx->src[0]; \+ const struct htp_tensor * dst = octx->dst; \+ struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \+ \+ htp_unary_preamble; \+ \+ uint32_t src0_nrows_per_thread = uctx->src0_nrows_per_thread; \+ \+ int32_t * op_params = octx->op_params; \+ const uint32_t col_tile = uctx->col_tile; \+ \+ const uint32_t src0_nrows = uctx->src0_nrows; \+ const uint32_t src0_start_row = uctx->row_start + src0_nrows_per_thread * ith; \+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, uctx->row_start + src0_nrows); \+ \+ if (src0_start_row >= src0_end_row) { \+ return; \+ } \+ \+ const uint8_t * restrict data_src = uctx->data_src0; \+ uint8_t * restrict data_dst = uctx->data_dst; \+ \+ uint8_t * src0_vtcm_data = uctx->vtcm_src0 + (ith * uctx->vtcm_src0_size_per_thread); \+ uint8_t * dst_vtcm_data = uctx->vtcm_dst + (ith * uctx->vtcm_dst_size_per_thread); \+ \+ const size_t src0_half = uctx->src0_vtcm_half_size; \+ const size_t dst_half = uctx->dst_vtcm_half_size; \+ \+ dma_queue * dmaq = octx->ctx->dma[ith]; \+ \+ const struct fastdiv_values * div_ne01 = &uctx->kparams->div_ne01; \+ const struct fastdiv_values * div_ne02 = &uctx->kparams->div_ne02; \+ const struct fastdiv_values * div_ne012 = &uctx->kparams->div_ne012; \+ const struct fastdiv_values * div_tpr = &uctx->kparams->div_tpr; \+ \+ const uint32_t tiles_per_row = (ne0 + col_tile - 1) / col_tile; \+ const int32_t tri_ttype = (IS_TRI) ? op_params[0] : 0; \+ \+ const bool src0_contig = (nb02 == (size_t)ne01 * nb01) && \+ (nb03 == (size_t)ne02 * nb02); \+ const bool dst_contig = (nb2 == (size_t)ne1 * nb1) && \+ (nb3 == (size_t)ne2 * nb2); \+ \+ const uint32_t total_tiles = (src0_end_row - src0_start_row) * tiles_per_row; \+ \+ for (uint32_t t = 0, vtcm_idx = 0; t < total_tiles && vtcm_idx < 2; t++, vtcm_idx++) { \+ const uint32_t row = src0_start_row + t / tiles_per_row; \+ const uint32_t col = (t % tiles_per_row) * col_tile; \+ const uint32_t tw = MIN(col_tile, ne0 - col); \+ const size_t tb = (size_t) tw * sizeof(float); \+ const size_t soff = (src0_contig ? (row * nb01) : \+ unary_row_offset(row, ne01, ne02, div_ne01, div_ne02, div_ne012, nb01, nb02, nb03)) + \+ (size_t) col * sizeof(float); \+ \+ dma_queue_push(dmaq, dma_make_ptr(data_dst, dst_vtcm_data + (vtcm_idx * dst_half)), 0, 0, 0, 0); \+ dma_queue_push(dmaq, dma_make_ptr(src0_vtcm_data + (vtcm_idx * src0_half), data_src + soff), tb, tb, tb, 1); \+ } \+ \+ uint32_t row = src0_start_row; \+ uint32_t col = 0; \+ uint32_t tile_in_row = 0; \+ uint32_t i01 = fastmodulo(row, ne01, div_ne01); \+ \+ uint32_t prow = src0_start_row + fastdiv(2, div_tpr); \+ uint32_t pcol = fastmodulo(2, tiles_per_row, div_tpr) * col_tile; \+ uint32_t ptile_in_row = fastmodulo(2, tiles_per_row, div_tpr); \+ \+ for (uint32_t t = 0; t < total_tiles; t++) { \+ uint8_t * dst_vtcm = (uint8_t *) dma_queue_pop(dmaq).src; \+ uint8_t * src_vtcm = (uint8_t *) dma_queue_pop(dmaq).dst; \+ \+ const uint32_t tw = MIN(col_tile, ne0 - col); \+ \+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, t); \+ CORE_TILE_EXPR; \+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, t); \+ \+ const size_t doff = (dst_contig ? (row * nb1) : \+ unary_row_offset(row, ne1, ne2, div_ne01, div_ne02, div_ne012, nb1, nb2, nb3)) + \+ (size_t) col * sizeof(float); \+ const size_t tb = (size_t) tw * sizeof(float); \+ dma_queue_push(dmaq, dma_make_ptr(data_dst + doff, dst_vtcm), tb, tb, tb, 1); \+ \+ const uint32_t pt = t + 2; \+ if (pt < total_tiles) { \+ const uint32_t ptw = MIN(col_tile, ne0 - pcol); \+ const size_t ptb = (size_t) ptw * sizeof(float); \+ const size_t psoff = (src0_contig ? (prow * nb01) : \+ unary_row_offset(prow, ne01, ne02, div_ne01, div_ne02, div_ne012, nb01, nb02, \+ nb03)) + \+ (size_t) pcol * sizeof(float); \+ dma_queue_push(dmaq, dma_make_ptr(src_vtcm, data_src + psoff), ptb, ptb, ptb, 1); \+ } \+ \+ tile_in_row++; \+ col += col_tile; \+ if (tile_in_row == tiles_per_row) { \+ tile_in_row = 0; \+ col = 0; \+ row++; \+ i01++; \+ if (i01 == ne01) { \+ i01 = 0; \+ } \+ } \+ \+ ptile_in_row++; \+ pcol += col_tile; \+ if (ptile_in_row == tiles_per_row) { \+ ptile_in_row = 0; \+ pcol = 0; \+ prow++; \+ } \+ } \+ \+ dma_queue_flush(dmaq); \}static inline void tile_scale_f32(uint8_t * dst_vtcm, const uint8_t * src_vtcm, uint32_t tw, const int32_t * op_params) {@@ -1146,14 +1149,32 @@ static int execute_op_unary(struct htp_ops_context * octx) {const struct htp_unary_kernel_params * kparams = (const struct htp_unary_kernel_params *) octx->kernel_params;- const uint32_t src0_nrows = src0->ne[1] * src0->ne[2] * src0->ne[3];- const uint32_t n_threads = kparams->n_threads;+ if (!htp_ops_context_set_n_threads(octx, kparams->n_threads)) {+ return HTP_STATUS_INVAL_PARAMS;+ }+ const uint32_t src0_nrows = src0->ne[1] * src0->ne[2] * src0->ne[3];const size_t elem_size = is_f16 ? sizeof(_Float16) : sizeof(float);-const size_t src0_data_row_size = src0->ne[0] * elem_size;const size_t dst_data_row_size = dst->ne[0] * elem_size;+ uint32_t row_start = 0;+ uint32_t nrows = src0_nrows;++ if (octx->ctx->mdev.count > 1) {+ uint32_t rows_per_chunk = 0;+ htp_tensor_mdev_rows_per_chunk(dst, (uint32_t) elem_size, (uint32_t) dst_data_row_size, &rows_per_chunk);+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(src0_nrows, rows_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ row_start = range.start;+ nrows = range.count;+ }++ if (nrows == 0) {+ return HTP_STATUS_OK;+ }++ const uint32_t n_threads = octx->n_threads;+const size_t src0_row_size_aligned = kparams->src0_row_size_aligned;const size_t dst_row_size_aligned = kparams->dst_row_size_aligned;@@ -1191,8 +1212,9 @@ static int execute_op_unary(struct htp_ops_context * octx) {struct htp_unary_context uctx = {.octx = octx,.kparams = kparams,- .src0_nrows_per_thread = (src0_nrows + n_threads - 1) / n_threads,- .src0_nrows = src0_nrows,+ .src0_nrows_per_thread = fastdiv(nrows + n_threads - 1, &octx->n_threads_div),+ .src0_nrows = nrows,+ .row_start = row_start,.data_src0 = (const uint8_t *)src0->data,.data_src1 = (octx->op == HTP_OP_RMS_NORM_MUL) ? (const uint8_t *)src1->data : NULL,@@ -1287,7 +1309,7 @@ static int execute_op_unary(struct htp_ops_context * octx) {}if (task_func) {- worker_pool_run_func(octx->ctx->worker_pool, task_func, &uctx, n_threads);+ work_queue_run(octx->ctx->work_queue, task_func, &uctx, n_threads);} else {FARF(ERROR, "execute_op_unary: task function is NULL for op %d\n", octx->op);err = HTP_STATUS_NO_SUPPORT;diff --git a/scripts/snapdragon/ggml-hexagon-align-macros.py b/scripts/snapdragon/ggml-hexagon-align-macros.pynew file mode 100755index 0000000000..b64db3e654--- /dev/null+++ b/scripts/snapdragon/ggml-hexagon-align-macros.py@@ -0,0 +1,296 @@+#!/usr/bin/env python3+"""+align-macros.py - Inspect and align trailing backslashes in multiline C/C++ macros.++Usage:+ align-macros.py [paths...] # Check and report misaligned macros+ align-macros.py --diff [paths...] # Show unified diff of fixes+ align-macros.py --fix [paths...] # Fix misaligned macros in-place+ align-macros.py --fix --mode majority ... # Align to the dominant column+ align-macros.py --fix --pad 2 ... # Align to (max_content_len + pad)++Safety rules:+ - Macros that are ALREADY aligned are NEVER touched (unless --all is given).+ - Whitespace after trailing backslashes is flagged and cleaned.+"""++import argparse+import difflib+import logging+import os+import re+import sys+from collections import Counter+from typing import List, Optional, Tuple, NamedTuple++logger = logging.getLogger("ggml-hexagon-align-macros")+++class MacroLine(NamedTuple):+ line_num: int # 1-indexed+ raw: str # Original line including newline+ content: str # Line content before trailing backslash (stripped of trailing whitespace)+ bs_col: Optional[int] # 1-indexed column of backslash, or None if last line has no backslash+ trailing_ws: bool # True if whitespace existed after the backslash+++class MacroDef(NamedTuple):+ name: str+ filepath: str+ start_line: int+ end_line: int+ lines: List[MacroLine]+++def parse_macros(filepath: str) -> List[MacroDef]:+ """Extract all multiline macros from a C/C++ source file."""+ try:+ with open(filepath, "r", encoding="utf-8", errors="replace") as f:+ lines = f.readlines()+ except Exception as e:+ logger.error(f"Error reading {filepath}: {e}")+ return []++ macros: List[MacroDef] = []+ i = 0+ n = len(lines)++ while i < n:+ line = lines[i]+ m = re.match(r"^\s*#\s*define\s+([A-Za-z_][A-Za-z0-9_]*)", line)+ if m:+ macro_name = m.group(1)+ macro_start = i + 1+ macro_lines: List[MacroLine] = []+ cur = i++ while cur < n:+ l_raw = lines[cur]+ l_rstrip = l_raw.rstrip("\r\n")++ # Check if line has a trailing backslash+ # Note: handle possible accidental spaces after backslash+ match_bs = re.search(r"\\([ \t]*)$", l_rstrip)+ if match_bs:+ has_trailing_ws = len(match_bs.group(1)) > 0+ bs_index = match_bs.start()+ content = l_rstrip[:bs_index].rstrip()+ # 1-indexed column of the backslash+ bs_col = bs_index + 1+ macro_lines.append(MacroLine(+ line_num=cur + 1,+ raw=l_raw,+ content=content,+ bs_col=bs_col,+ trailing_ws=has_trailing_ws+ ))+ cur += 1+ else:+ # Line does not end with backslash+ if cur == i:+ # Single-line macro, not multiline+ break+ else:+ # Final line of a multiline macro+ macro_lines.append(MacroLine(+ line_num=cur + 1,+ raw=l_raw,+ content=l_rstrip.rstrip(),+ bs_col=None,+ trailing_ws=False+ ))+ break++ # Only record if it is a multiline macro (has at least one continuation line)+ continuation_lines = [ml for ml in macro_lines if ml.bs_col is not None]+ if continuation_lines:+ macro_end = macro_lines[-1].line_num+ macros.append(MacroDef(+ name=macro_name,+ filepath=filepath,+ start_line=macro_start,+ end_line=macro_end,+ lines=macro_lines+ ))+ i = cur+ i += 1++ return macros+++def is_macro_aligned(macro: MacroDef) -> bool:+ """A macro is aligned if all continuation lines have backslashes at the same column."""+ bs_cols = [ml.bs_col for ml in macro.lines if ml.bs_col is not None]+ if not bs_cols:+ return True+ has_trailing_ws = any(ml.trailing_ws for ml in macro.lines)+ return len(set(bs_cols)) == 1 and not has_trailing_ws+++def compute_target_column(macro: MacroDef, mode: str, pad: int, target_col: Optional[int]) -> int:+ """Determine the column where backslashes should be aligned."""+ max_content_len = max(len(ml.content) for ml in macro.lines)+ min_needed = max_content_len + pad++ if target_col is not None:+ return max(target_col, min_needed)++ bs_cols = [ml.bs_col for ml in macro.lines if ml.bs_col is not None]+ if not bs_cols:+ return min_needed++ if mode == "min":+ return min_needed+ elif mode == "max":+ return max(max(bs_cols), min_needed)+ elif mode == "majority":+ counts = Counter(bs_cols)+ # Sort by frequency descending, then by column descending+ majority_col = sorted(counts.items(), key=lambda x: (-x[1], -x[0]))[0][0]+ return max(majority_col, min_needed)+ else:+ return min_needed+++def realign_macro_lines(macro: MacroDef, target_col: int) -> List[str]:+ """Format macro lines with backslashes aligned at target_col."""+ new_lines: List[str] = []+ for ml in macro.lines:+ nl = "\r\n" if ml.raw.endswith("\r\n") else "\n"+ if ml.bs_col is None:+ # Last line without backslash+ new_lines.append(ml.raw)+ else:+ if not ml.content:+ spaces = " " * (target_col - 1)+ new_lines.append(f"{spaces}\\{nl}")+ else:+ spaces_needed = max(1, target_col - len(ml.content) - 1)+ new_lines.append(f"{ml.content}{' ' * spaces_needed}\\{nl}")+ return new_lines+++def process_file(filepath: str, args: argparse.Namespace) -> Tuple[int, int, Optional[str]]:+ macros = parse_macros(filepath)+ if not macros:+ return 0, 0, None++ with open(filepath, "r", encoding="utf-8", errors="replace") as f:+ file_lines = f.readlines()++ misaligned_count = 0+ modified = False+ new_file_lines = list(file_lines)++ for macro in macros:+ aligned = is_macro_aligned(macro)+ if not aligned or args.all:+ if not aligned:+ misaligned_count += 1++ bs_cols = [ml.bs_col for ml in macro.lines if ml.bs_col is not None]+ max_content = max(len(ml.content) for ml in macro.lines)+ col_counts = Counter(bs_cols)++ if not args.quiet:+ logger.info(f"{filepath}:{macro.start_line}-{macro.end_line} [{macro.name}]")+ logger.info(f" Max content width: {max_content}, Min needed column (+{args.pad}): {max_content + args.pad}")+ logger.info(f" Current backslash columns: {dict(sorted(col_counts.items()))}")+ trailing_ws_lines = [ml.line_num for ml in macro.lines if ml.trailing_ws]+ if trailing_ws_lines:+ logger.warning(f" Warning: Trailing whitespace after backslash on line(s): {trailing_ws_lines}")++ target_col = compute_target_column(macro, args.mode, args.pad, args.target_col)+ if not args.quiet:+ logger.info(f" -> Target alignment column: {target_col}")++ realigned = realign_macro_lines(macro, target_col)++ start_idx = macro.start_line - 1+ end_idx = start_idx + len(macro.lines)+ if new_file_lines[start_idx:end_idx] != realigned:+ new_file_lines[start_idx:end_idx] = realigned+ modified = True++ diff_text = None+ if modified:+ diff = difflib.unified_diff(+ file_lines,+ new_file_lines,+ fromfile=f"a/{filepath}",+ tofile=f"b/{filepath}",+ lineterm=""+ )+ diff_text = "\n".join(diff)++ if args.fix:+ with open(filepath, "w", encoding="utf-8") as f:+ f.writelines(new_file_lines)+ if not args.quiet:+ logger.info(f" [FIXED] Updated {filepath}")++ return len(macros), misaligned_count, diff_text+++def find_source_files(paths: List[str]) -> List[str]:+ extensions = {".c", ".cpp", ".cc", ".cxx", ".h", ".hpp", ".inl"}+ result: List[str] = []+ for p in paths:+ if os.path.isfile(p):+ result.append(p)+ elif os.path.isdir(p):+ for root, _, files in os.walk(p):+ for file in sorted(files):+ _, ext = os.path.splitext(file)+ if ext.lower() in extensions:+ result.append(os.path.join(root, file))+ return sorted(result)+++def main():+ logging.basicConfig(level=logging.INFO, format="%(message)s")+ parser = argparse.ArgumentParser(+ description="Inspect and align backslashes in multiline C/C++ macros."+ )+ parser.add_argument("paths", nargs="*", default=["."], help="Files or directories to scan (default: current dir)")+ parser.add_argument("--fix", action="store_true", help="Fix misaligned macros in-place")+ parser.add_argument("--diff", action="store_true", help="Display unified diff of suggested fixes")+ parser.add_argument("--check", action="store_true", help="Exit with code 1 if misaligned macros exist")+ parser.add_argument("--mode", choices=["min", "max", "majority"], default="min",+ help="Alignment mode: 'min' (max_len + pad), 'max' (max existing col), 'majority' (dominant col)")+ parser.add_argument("--pad", type=int, default=2, help="Spaces between longest line and backslash (default: 2)")+ parser.add_argument("--target-col", type=int, default=None, help="Force alignment to an exact column")+ parser.add_argument("--all", action="store_true", help="Realign all macros even if already aligned (default: only misaligned)")+ parser.add_argument("-q", "--quiet", action="store_true", help="Only output errors and diffs/summary")++ args = parser.parse_args()++ files = find_source_files(args.paths)+ if not files:+ logger.error("No C/C++ source files found.")+ sys.exit(0)++ total_macros = 0+ total_misaligned = 0+ diffs: List[str] = []++ for filepath in files:+ num_macros, num_misaligned, diff_text = process_file(filepath, args)+ total_macros += num_macros+ total_misaligned += num_misaligned+ if diff_text:+ diffs.append(diff_text)++ if args.diff and diffs:+ logger.info("\n--- Proposed Changes ---\n")+ for d in diffs:+ logger.info(d)++ logger.info(f"\nSummary: scanned {len(files)} files, {total_macros} multiline macros, {total_misaligned} misaligned.")++ if args.check and total_misaligned > 0:+ sys.exit(1)+++if __name__ == "__main__":+ main()diff --git a/scripts/snapdragon/run.py b/scripts/snapdragon/run.pyindex 81eecd2e0c..dc71d4a321 100755--- a/scripts/snapdragon/run.py+++ b/scripts/snapdragon/run.py@@ -14,6 +14,42 @@ import logginglogger = logging.getLogger("run")+MANAGED_ENV_NAMES = (+ "GGML_HEXAGON_DEVICES",+ "GGML_HEXAGON_VERBOSE",+ "GGML_HEXAGON_PROFILE",+ "GGML_HEXAGON_NHVX",+ "GGML_HEXAGON_NHMX",+ "GGML_HEXAGON_HOSTBUF",+ "GGML_HEXAGON_OPBATCH",+ "GGML_HEXAGON_OPQUEUE",+ "GGML_HEXAGON_OPPOLL",+ "GGML_HEXAGON_OPFILTER",+ "GGML_HEXAGON_OPFUSION",+ "GGML_HEXAGON_VMEM",+ "GGML_HEXAGON_MBUF",+ "GGML_HEXAGON_MM_SELECT",+ "GGML_HEXAGON_FA_SELECT",+ "GGML_HEXAGON_AR_SELECT",+ "GGML_HEXAGON_ETM",+ "GGML_HEXAGON_ARCH",+ "GGML_HEXAGON_OPTRACE",+ "GGML_OPENCL_PLATFORM",+ "GGML_OPENCL_DEVICE",+ "GGML_OPENCL_OPFILTER",+ "GGML_OPENCL_KERNEL_CACHE_DIR",+ "GGML_OPENCL_KERNEL_CACHE_DEBUG",+ "GGML_OPENCL_FA_TUNE",+ "GGML_OPENCL_DISABLE_FUSION",+ "GGML_OPENCL_ADRENO_XMEM_GEMM",+ "GGML_OPENCL_ADRENO_USE_LARGE_BUFFER",+ "GGML_SCHED_DEBUG",+ "MTMD_BACKEND_DEVICE",+ "D",+ "DEVICE",+)++def parse_target(target_str):if not target_str:return None, None@@ -38,6 +74,57 @@ def shlex_join(args_list):return " ".join(pipes.quote(x) for x in args_list)+def split_device_list(devices):+ parts = []+ curr = []+ bracket_depth = 0++ for ch in devices:+ if ch == '[':+ bracket_depth += 1+ curr.append(ch)+ elif ch == ']':+ if bracket_depth > 0:+ bracket_depth -= 1+ curr.append(ch)+ elif ch == ',' and bracket_depth == 0:+ part = "".join(curr).strip()+ if part:+ parts.append(part)+ curr = []+ else:+ curr.append(ch)++ part = "".join(curr).strip()+ if part:+ parts.append(part)++ return parts+++def device_arg_from_devices(devices):+ if devices.isdigit():+ n = int(devices)+ return ",".join(f"HTP{i}" for i in range(n))++ names = []+ for part in split_device_list(devices):+ if "[" in part:+ part = part.split("[", 1)[0].strip()+ if part:+ names.append(part)++ return ",".join(names)+++def normalize_cmd_device_args(cmd_args):+ for i, arg in enumerate(cmd_args):+ if arg == "--device" and i + 1 < len(cmd_args):+ cmd_args[i + 1] = device_arg_from_devices(cmd_args[i + 1])+ elif arg.startswith("--device="):+ cmd_args[i] = "--device=" + device_arg_from_devices(arg.split("=", 1)[1])++def main():logging.basicConfig(level=logging.INFO, format='%(message)s')# Split arguments at '--'@@ -142,8 +229,6 @@ def main():def set_env(env_name, opt_val):if opt_val is not None:env_vars[env_name] = str(opt_val)- elif env_name in os.environ:- env_vars[env_name] = os.environ[env_name]# Resolve and filter devices (HTP vs OpenCL)device_in_cmd = None@@ -166,7 +251,7 @@ def main():hex_devices = devices_valcl_device = ""else:- parts = [p.strip() for p in devices_val.split(",")]+ parts = split_device_list(devices_val)# Any device containing "htp" is Hexagon, rest is OpenCLhex_parts = [p for p in parts if "htp" in p.lower()]cl_parts = [@@ -181,15 +266,13 @@ def main():# Set Hexagon devicesif hex_devices:env_vars["GGML_HEXAGON_DEVICES"] = hex_devices- elif "GGML_HEXAGON_DEVICES" in os.environ:- env_vars["GGML_HEXAGON_DEVICES"] = os.environ["GGML_HEXAGON_DEVICES"]++ normalize_cmd_device_args(cmd_args)# Set OpenCL device (unless overridden by --cl-device)final_cl_device = args.cl_device if args.cl_device is not None else cl_deviceif final_cl_device:env_vars["GGML_OPENCL_DEVICE"] = final_cl_device- elif "GGML_OPENCL_DEVICE" in os.environ:- env_vars["GGML_OPENCL_DEVICE"] = os.environ["GGML_OPENCL_DEVICE"]# Map shared & backend-specific parameters with correct overrides@@ -206,8 +289,6 @@ def main():if args.cl_fa_tune or args.profile is not None:env_vars["GGML_OPENCL_FA_TUNE"] = "1"- elif "GGML_OPENCL_FA_TUNE" in os.environ:- env_vars["GGML_OPENCL_FA_TUNE"] = os.environ["GGML_OPENCL_FA_TUNE"]# Other Hexagon environment variablesset_env("GGML_HEXAGON_NHVX", args.hex_nhvx)@@ -235,18 +316,12 @@ def main():if args.cl_disable_fusion:env_vars["GGML_OPENCL_DISABLE_FUSION"] = "1"- elif "GGML_OPENCL_DISABLE_FUSION" in os.environ:- env_vars["GGML_OPENCL_DISABLE_FUSION"] = os.environ["GGML_OPENCL_DISABLE_FUSION"]if args.cl_adreno_xmem:env_vars["GGML_OPENCL_ADRENO_XMEM_GEMM"] = "1"- elif "GGML_OPENCL_ADRENO_XMEM_GEMM" in os.environ:- env_vars["GGML_OPENCL_ADRENO_XMEM_GEMM"] = os.environ["GGML_OPENCL_ADRENO_XMEM_GEMM"]if args.cl_adreno_large_buffer:env_vars["GGML_OPENCL_ADRENO_USE_LARGE_BUFFER"] = "1"- elif "GGML_OPENCL_ADRENO_USE_LARGE_BUFFER" in os.environ:- env_vars["GGML_OPENCL_ADRENO_USE_LARGE_BUFFER"] = os.environ["GGML_OPENCL_ADRENO_USE_LARGE_BUFFER"]if args.sched_debug:env_vars["GGML_SCHED_DEBUG"] = "2"@@ -288,15 +363,7 @@ def main():has_b = any(arg == "-b" for arg in cmd_args)if not has_b:if args.devices:- if args.devices.isdigit():- n = int(args.devices)- device_val = ",".join(f"HTP{i}" for i in range(n))- else:- device_val = args.devices- elif "D" in os.environ:- device_val = os.environ["D"]- elif "DEVICE" in os.environ:- device_val = os.environ["DEVICE"]+ device_val = device_arg_from_devices(args.devices)else:device_val = "HTP0"if device_val:@@ -305,17 +372,10 @@ def main():has_device = any(arg.startswith("--device") for arg in cmd_args)if not has_device:if args.devices:- if args.devices.isdigit():- n = int(args.devices)- device_val = ",".join(f"HTP{i}" for i in range(n))- else:- device_val = args.devices- elif "D" in os.environ:- device_val = os.environ["D"]- elif "DEVICE" in os.environ:- device_val = os.environ["DEVICE"]+ device_val = device_arg_from_devices(args.devices)else:device_val = "HTP0"+if device_val:cmd_args += ["--device", device_val]@@ -415,6 +475,8 @@ def main():else:local_env["LD_LIBRARY_PATH"] = lib_dir + os.path.pathsep + local_env.get("LD_LIBRARY_PATH", "")+ for k in MANAGED_ENV_NAMES:+ local_env.pop(k, None)for k, v in env_vars.items():local_env[k] = v