LLM inference in C/C++
C++ ggml
hexagon: tensor repacking rework and support for tiled and flat quantized and fp weights (#30635)
Max Krasnyanskyb797b9ca816b
diff --git a/docs/backend/snapdragon/developer.md b/docs/backend/snapdragon/developer.mdindex 3608250dc7..e088390e81 100644--- a/docs/backend/snapdragon/developer.md+++ b/docs/backend/snapdragon/developer.md@@ -134,6 +134,18 @@ Writing high-performance operators for Hexagon requires following specific guide- **Distinction**: `octx->src[N]` pointers *can* be NULL by design and must be checked when optional.Examples include attention masks, optional bias or weights in fused kernels, and frequency factors.+### `supports_op` Independence from Buffer and Repack State++- `supports_op` (and op-specific checkers such as `supported_mul_mat()`, `supported_get_rows()`) runs during GGML graph scheduling **before** buffers are allocated and before tensor data is populated.+- **Never check buffer usage or repacking state in `supports_op`**:+ - `t->buffer` may be NULL or uninitialized.+ - `t->data` may be NULL.+ - `buffer->usage` is unknown at scheduling time.+ - `is_tiled` or `extra->flags & GGML_HEXAGON_TENSOR_TILED` is not set yet.+- Operator support decisions must depend strictly on static tensor properties: operand types (`t->type`), shapes (`t->ne`), strides (`t->nb`), and operation parameters.+- Repacking (converting weights to tiled layout) is an internal optimization applied during `set_tensor()` or rt-repack. Non-weight tensors or test harness tensors with the same type remain in flat layout.+- Therefore, any supported type (such as BF16, F16, quantized types) must be supported both when repacked into tiles and when stored flat in memory.+### Multiline Macro Formatting- Keep trailing backslashes in multiline `#define` macros cleanly aligned to a consistent column.@@ -190,7 +202,7 @@ sessions.Common partitioning logic is factored into reusable inline helpers in[`htp-tensor.h`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h):-1. [`htp_tensor_mdev_rows_per_chunk`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L71):+1. [`htp_tensor_mdev_rows_per_chunk`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L104):Determines the minimum number of rows per chunk so that the chunk byte size is a multiple of 128 bytes:```@@ -201,7 +213,7 @@ Common partitioning logic is factored into reusable inline helpers inReturns `false` if the tensor cannot be safely row-partitioned (such as unaligned base pointer, permuted layout,or non-128-byte aligned outer strides).-2. [`htp_tensor_mdev_partition`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L98):+2. [`htp_tensor_mdev_partition`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L131):Calculates the per-device work range `struct htp_tensor_mdev_range { uint32_t start; uint32_t count; }` given`total_units`, `units_per_chunk`, `mdev_idx`, `mdev_count`, and the precomputed `mdev_count_div`.Handles chunk distribution across devices, assigns remainder units to the last device, and automatically triggers@@ -240,11 +252,11 @@ if (nrows == 0) {For flat element-wise operations that write directly to DDR:- Partition total linear elements N = ne0 * ne1 * ne2 * ne3 in 128-byte cache line chunks (`elems_per_line = (elem_size == 4) ? 32 : 64`).- Requires strict 1D contiguity:- [`htp_tensor_is_contiguous(dst, elem_size)`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L32)+ [`htp_tensor_is_contiguous(dst, elem_size)`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L65)and 128-byte aligned destination pointer- [`htp_tensor_mdev_data_aligned(dst)`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L51).+ [`htp_tensor_mdev_data_aligned(dst)`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L84).- If contiguous and aligned, pass `elems_per_line` to- [`htp_tensor_mdev_partition`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L98);+ [`htp_tensor_mdev_partition`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h#L131);otherwise pass 0 to trigger Device 0 fallback.### Single-Device Fallback (Device 0)@@ -312,7 +324,7 @@ Multi-device execution synchronizes worker sessions through atomic fence slots a(Output Slice 0) (Output Slice 1)| |Tensor Cache Flush Tensor Cache Flush- (htp_tensor_flush_all) (htp_tensor_flush_all)+ (htp_dcache_flush_dirty_tensors) (htp_dcache_flush_dirty_tensors)| |Post-Op/Batch Barrier ---------------------- Post-Op/Batch Barrier(htp_mdev_group_barrier) (htp_mdev_group_barrier)@@ -350,8 +362,8 @@ Multi-device execution synchronizes worker sessions through atomic fence slots a### Tensor Cache Flush and Pipeline Completion- In the kernel, ensure all pushed DMA operations have been popped in strict FIFO order to drain the queue.-- Use [`htp_tensor_flush_all()`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h) to flush specific dirty tensors back to DDR:- - [`htp_tensor_flush_all()`](../../../ggml/src/ggml-hexagon/htp/htp-tensor.h) flushes modified tensor address ranges, or the+- Use [`htp_dcache_flush_dirty_tensors()`](../../../ggml/src/ggml-hexagon/htp/htp-dcache.h) to flush specific dirty tensors back to DDR:+ - [`htp_dcache_flush_dirty_tensors()`](../../../ggml/src/ggml-hexagon/htp/htp-dcache.h) flushes modified tensor address ranges, or thefull D-cache when their total size exceeds the flush threshold, ensuring peer devices and the host CPU observe consistentdata in DDR.- Never signal completion before all DMA transfers are drained and dirty tensor flushes have completed.diff --git a/ggml/src/ggml-hexagon/CMakeLists.txt b/ggml/src/ggml-hexagon/CMakeLists.txtindex 20c720263b..277a4c863a 100644--- a/ggml/src/ggml-hexagon/CMakeLists.txt+++ b/ggml/src/ggml-hexagon/CMakeLists.txt@@ -50,6 +50,8 @@ endif()set(TARGET_NAME ggml-hexagon)ggml_add_backend_library(${TARGET_NAME}ggml-hexagon.cpp+ htp-repack.cpp+ htp-repack.hhtp-drv.cpphtp-drv.hlibdl.hdiff --git a/ggml/src/ggml-hexagon/ggml-hexagon.cpp b/ggml/src/ggml-hexagon/ggml-hexagon.cppindex 08c15ff786..54c4741224 100644--- a/ggml/src/ggml-hexagon/ggml-hexagon.cpp+++ b/ggml/src/ggml-hexagon/ggml-hexagon.cpp@@ -77,6 +77,7 @@#include "htp/im2col-ops.h"#include "htp_iface.h"#include "htp-drv.h"+#include "htp-repack.h"using intvec = std::vector<int>;using uintvec = std::vector<unsigned int>;@@ -164,7 +165,8 @@ static int opt_opbatch = 1280; // max number of ops in a batchstatic int opt_opqueue = 8; // max number of pending batchesstatic int opt_optrace = 0; // trace buffer size per thread (0 means default)static int opt_oppoll = 0; // polling for batch completions-static int opt_opfusion = 1; // enable/disable op fusion+static int opt_opfusion = 1; // enable/disable op fusion+static int opt_rt_repack = 1; // runtime repacking: 0 = disabled, 1 = quantized only (default), 2 = alwaysenum ggml_hexagon_fusion_flags {GGML_HEXAGON_FUSE_ALLREDUCE_ADD = (1 << 1), // 2@@ -309,42 +311,12 @@ static void ggml_hexagon_dump_trace_events(const std::string & sess_name, const}enum ggml_hexagon_tensor_flags {- GGML_HEXAGON_TENSOR_REPACK = (1 << 0),+ GGML_HEXAGON_TENSOR_TILED = (1 << 0),GGML_HEXAGON_TENSOR_WEIGHT = (1 << 1),GGML_HEXAGON_TENSOR_FENCE = (1 << 2),GGML_HEXAGON_TENSOR_FUSEABLE = (1 << 3),};-static inline bool ggml_hexagon_is_repack_type(enum ggml_type type) {- return type == GGML_TYPE_Q4_0 || type == GGML_TYPE_Q4_1 ||- type == GGML_TYPE_Q8_0 || type == GGML_TYPE_IQ4_NL ||- type == GGML_TYPE_MXFP4 || type == GGML_TYPE_Q6_K ||- type == GGML_TYPE_Q4_K || type == GGML_TYPE_Q5_K ||- type == GGML_TYPE_Q3_K || type == GGML_TYPE_Q2_K;-}--// Size of one repacked row in the DSP tiled layout. The Q3_K..Q6_K tiles store uncompressed scales/mins,-// so they are larger than the ggml blocks. For the other repack types (Q2_K included) the tile has the same size.-static inline size_t ggml_hexagon_tiled_row_size(enum ggml_type type, int64_t ne0) {- if (type == GGML_TYPE_Q6_K) {- return (size_t) (ne0 / 32) * (HTP_MM_WEIGHT_TILE_SIZE_Q6_K / 32);- }- if (type == GGML_TYPE_Q4_K) {- return (size_t) (ne0 / 32) * (HTP_MM_WEIGHT_TILE_SIZE_Q4_1 / 32);- }- if (type == GGML_TYPE_Q5_K) {- return (size_t) (ne0 / 32) * (HTP_MM_WEIGHT_TILE_SIZE_Q5_K / 32);- }- if (type == GGML_TYPE_Q3_K) {- return (size_t) (ne0 / 32) * (HTP_MM_WEIGHT_TILE_SIZE_Q3_K / 32);- }- return ggml_row_size(type, ne0);-}--static inline bool ggml_hexagon_is_hmx_weight_type(enum ggml_type type) {- return type == GGML_TYPE_F16 || type == GGML_TYPE_F32 || ggml_hexagon_is_repack_type(type);-}-struct ggml_hexagon_session;static void ggml_hexagon_precompute_matmul_params(@@ -519,6 +491,32 @@ static inline bool ggml_hexagon_tensor_is_fuseable(const struct ggml_tensor * t)return (extra->flags & GGML_HEXAGON_TENSOR_FUSEABLE) != 0;}+static inline bool ggml_hexagon_can_rt_repack(const struct ggml_tensor * t) {+ if (opt_rt_repack == 0 || !htp_is_tiled_weight_tensor(t)) {+ return false;+ }+ if (t->buffer && ggml_backend_buft_is_host(t->buffer->buft)) {+ return false;+ }+ return opt_rt_repack > 1 || ggml_is_quantized(t->type);+}++static inline bool ggml_hexagon_tensor_is_tiled(const struct ggml_tensor * t) {+ const auto * extra = (const ggml_hexagon_tensor_extra *) t->extra;+ if (extra && (extra->flags & GGML_HEXAGON_TENSOR_TILED)) {+ return true;+ }+ return ggml_hexagon_can_rt_repack(t);+}++static inline bool ggml_hexagon_needs_repack(const struct ggml_tensor * t) {+ const auto * extra = (const ggml_hexagon_tensor_extra *) t->extra;+ if (extra && (extra->flags & GGML_HEXAGON_TENSOR_TILED)) {+ return false;+ }+ return ggml_hexagon_can_rt_repack(t);+}+static inline bool ggml_hexagon_tensors_overlap(const struct ggml_tensor * a, const struct ggml_tensor * b) {const uintptr_t a0 = (uintptr_t) a->data;const uintptr_t b0 = (uintptr_t) b->data;@@ -600,8 +598,6 @@ struct ggml_hexagon_session {uint64_t cached_uid = 0;std::vector<htp_opnode> cached_nodes;- mutable std::unordered_set<const ggml_tensor *> needs_repack;-ggml_hexagon_mdev_group mdev;ggml_backend_dev_t dev = nullptr;ggml_backend_hexagon_device_context * dev_ctx = nullptr;@@ -850,1617 +846,54 @@ struct ggml_hexagon_fence_buffer : public ggml_hexagon_shared_buffer {if (ptr) {memset(ptr, 0, (size_t) n_slots * GGML_HEXAGON_FENCE_SLOT_SIZE);}- return ptr;- }-- void free_slot(void * ptr, uint32_t n_slots = 1) {- if (!ptr) return;- uint32_t slot = ((uint8_t *) ptr - base()) / GGML_HEXAGON_FENCE_SLOT_SIZE;- for (uint32_t i = 0; i < n_slots; i++) {- free_slots.push_back(slot + i);- }- }-};--inline uint8_t * ggml_hexagon_session::alloc_fence(uint32_t n_slots) {- uint8_t * ptr = fence_buf->alloc_slot(n_slots);- GGML_ASSERT(ptr);- return ptr;-}--inline void ggml_hexagon_session::free_fence(void * ptr, uint32_t n_slots) {- if (fence_buf) {- fence_buf->free_slot(ptr, n_slots);- }-}--static void ggml_backend_hexagon_buffer_free_buffer(ggml_backend_buffer_t buffer) {- auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(buffer->context);- sbuf->sess->unclone_buffer(sbuf);- delete sbuf;-}--static void * ggml_backend_hexagon_buffer_get_base(ggml_backend_buffer_t buffer) {- auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(buffer->context);- return sbuf->base();-}--static enum ggml_status ggml_backend_hexagon_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) {- auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(buffer->context);- auto sess = sbuf->sess;-- HEX_VERBOSE("ggml-hex: %s init-tensor %s : base %p data %p nbytes %zu\n", sess->c_name(),- tensor->name, (void *) sbuf->base(), tensor->data, ggml_nbytes(tensor));-- auto extra = new ggml_hexagon_tensor_extra();- sbuf->tensor_extra.push_back(extra);-- tensor->extra = extra;- if (ggml_hexagon_is_repack_type(tensor->type)) {- if (sess->needs_repack.count(tensor)) {- extra->flags |= GGML_HEXAGON_TENSOR_REPACK;- sess->needs_repack.erase(tensor);- }- }-- return GGML_STATUS_SUCCESS;-}--// ** Repack helpers for tiled quantized weights--static void unpack_q4_0_quants(uint8_t * qs, const block_q4_0 * x, unsigned int bi) {- static const int qk = QK4_0;-- for (unsigned int i = 0; i < qk / 2; ++i) {- const int x0 = (x->qs[i] & 0x0F);- const int x1 = (x->qs[i] >> 4);- qs[bi * qk + i + 0] = x0;- qs[bi * qk + i + qk / 2] = x1;- }-}--static void pack_q4_0_quants(block_q4_0 * x, const uint8_t * qs, unsigned int bi) {- static const int qk = QK4_0;-- for (unsigned int i = 0; i < qk / 2; ++i) {- const uint8_t x0 = qs[bi * qk + i + 0];- const uint8_t x1 = qs[bi * qk + i + qk / 2];- x->qs[i] = x0 | (x1 << 4);- }-}--static void unpack_q4_1_quants(uint8_t * qs, const block_q4_1 * x, unsigned int bi) {- static const int qk = QK4_1;-- for (unsigned int i = 0; i < qk / 2; ++i) {- const int x0 = (x->qs[i] & 0x0F);- const int x1 = (x->qs[i] >> 4);- qs[bi * qk + i + 0] = x0;- qs[bi * qk + i + qk / 2] = x1;- }-}--static void pack_q4_1_quants(block_q4_1 * x, const uint8_t * qs, unsigned int bi) {- static const int qk = QK4_1;-- for (unsigned int i = 0; i < qk / 2; ++i) {- const uint8_t x0 = qs[bi * qk + i + 0];- const uint8_t x1 = qs[bi * qk + i + qk / 2];- x->qs[i] = x0 | (x1 << 4);- }-}--static void unpack_mxfp4_quants(uint8_t * qs, const block_mxfp4 * x, unsigned int bi) {- static const int qk = QK_MXFP4;-- for (unsigned int i = 0; i < qk / 2; ++i) {- const int x0 = (x->qs[i] & 0x0F);- const int x1 = (x->qs[i] >> 4);- qs[bi * qk + i + 0] = x0;- qs[bi * qk + i + qk / 2] = x1;- }-}--static void pack_mxfp4_quants(block_mxfp4 * x, const uint8_t * qs, unsigned int bi) {- static const int qk = QK_MXFP4;-- for (unsigned int i = 0; i < qk / 2; ++i) {- const uint8_t x0 = qs[bi * qk + i + 0];- const uint8_t x1 = qs[bi * qk + i + qk / 2];- x->qs[i] = x0 | (x1 << 4);- }-}--// repack q4_0 data into q4_0_tiled tensor-static void repack_q4_0_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {- const block_q4_0 * src_matrix = (const block_q4_0 *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- int n_col_tiles = ne1_padded / 32;- int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q4_0;- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;-- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);- int64_t start_slice = offset / slice_size;- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;- if (end_slice > ne2 * ne3) {- end_slice = ne2 * ne3;- }-- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {- const block_q4_0 * src_slice = src_matrix + (slice_idx - start_slice) * (ne1 * (ne0 / 32));- uint8_t * matrix_dst = (uint8_t *) t->data + slice_idx * matrix_size;-- for (int ct = 0; ct < n_col_tiles; ct++) {- for (int kt = 0; kt < n_k_tiles; kt++) {- uint8_t * tile_dst = matrix_dst + (ct * n_k_tiles + kt) * tile_size;-- uint8_t tile_quants[32][32];- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- if (r < ne1 && kt < ne0 / 32) {- unpack_q4_0_quants(tile_quants[row], &src_slice[r * (ne0 / 32) + kt], 0);- } else {- memset(tile_quants[row], 8, 32);- }- }-- for (int cp = 0; cp < 16; cp++) {- for (int row = 0; row < 32; row++) {- tile_dst[cp * 32 + row] = (tile_quants[row][2 * cp + 1] << 4) | tile_quants[row][2 * cp];- }- }-- ggml_half * scale_dst = (ggml_half *)(tile_dst + 512);- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- scale_dst[row] = (r < ne1 && kt < ne0 / 32) ? src_slice[r * (ne0 / 32) + kt].d : 0;- }- }- }- }-}--// repack q4_0_tiled tensor into q4_0 data-static void repack_tiled_q4_0(void * data, const ggml_tensor * t, size_t offset, size_t size) {- block_q4_0 * dst_matrix = (block_q4_0 *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- int n_col_tiles = ne1_padded / 32;- int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q4_0;- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;-- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);- size_t row_size_bytes = ggml_row_size(t->type, ne0);- int64_t start_slice = offset / slice_size;- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;- if (end_slice > ne2 * ne3) {- end_slice = ne2 * ne3;- }-- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {- size_t cur_start_byte = (std::max)(offset, (size_t) slice_idx * slice_size);- size_t cur_end_byte = (std::min)(offset + size, (size_t) (slice_idx + 1) * slice_size);- size_t slice_offset_start = cur_start_byte - (size_t) slice_idx * slice_size;- size_t slice_offset_end = cur_end_byte - (size_t) slice_idx * slice_size;-- int64_t start_row = slice_offset_start / row_size_bytes;- int64_t end_row = (slice_offset_end + row_size_bytes - 1) / row_size_bytes;- end_row = (std::min)(end_row, ne1);-- int start_ct = start_row / 32;- int end_ct = (end_row + 31) / 32;- end_ct = (std::min)(end_ct, n_col_tiles);-- block_q4_0 * dst_slice = dst_matrix + (cur_start_byte - offset) / sizeof(block_q4_0);- const uint8_t * matrix_src = (const uint8_t *) t->data + slice_idx * matrix_size;-- for (int ct = start_ct; ct < end_ct; ct++) {- for (int kt = 0; kt < n_k_tiles; kt++) {- const uint8_t * tile_src = matrix_src + (ct * n_k_tiles + kt) * tile_size;-- uint8_t tile_quants[32][32];- for (int cp = 0; cp < 16; cp++) {- for (int row = 0; row < 32; row++) {- uint8_t val = tile_src[cp * 32 + row];- tile_quants[row][2 * cp + 0] = val & 0x0F;- tile_quants[row][2 * cp + 1] = val >> 4;- }- }-- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- if (r >= start_row && r < end_row && kt < ne0 / 32) {- pack_q4_0_quants(&dst_slice[(r - start_row) * (ne0 / 32) + kt], tile_quants[row], 0);- }- }-- const ggml_half * scale_src = (const ggml_half *)(tile_src + 512);- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- if (r >= start_row && r < end_row && kt < ne0 / 32) {- dst_slice[(r - start_row) * (ne0 / 32) + kt].d = scale_src[row];- }- }- }- }- }-}--// repack q4_1 data into q4_1_tiled tensor-static void repack_q4_1_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {- const block_q4_1 * src_matrix = (const block_q4_1 *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- int n_col_tiles = ne1_padded / 32;- int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q4_1;- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;-- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);- int64_t start_slice = offset / slice_size;- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;- if (end_slice > ne2 * ne3) {- end_slice = ne2 * ne3;- }-- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {- const block_q4_1 * src_slice = src_matrix + (slice_idx - start_slice) * (ne1 * (ne0 / 32));- uint8_t * matrix_dst = (uint8_t *) t->data + slice_idx * matrix_size;-- for (int ct = 0; ct < n_col_tiles; ct++) {- for (int kt = 0; kt < n_k_tiles; kt++) {- uint8_t * tile_dst = matrix_dst + (ct * n_k_tiles + kt) * tile_size;-- uint8_t tile_quants[32][32];- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- if (r < ne1 && kt < ne0 / 32) {- unpack_q4_1_quants(tile_quants[row], &src_slice[r * (ne0 / 32) + kt], 0);- } else {- memset(tile_quants[row], 0, 32);- }- }-- for (int cp = 0; cp < 16; cp++) {- for (int row = 0; row < 32; row++) {- tile_dst[cp * 32 + row] = (tile_quants[row][2 * cp + 1] << 4) | tile_quants[row][2 * cp];- }- }-- ggml_half * scale_dst = (ggml_half *)(tile_dst + 512);- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- if (r < ne1 && kt < ne0 / 32) {- scale_dst[2 * row + 0] = src_slice[r * (ne0 / 32) + kt].d;- scale_dst[2 * row + 1] = src_slice[r * (ne0 / 32) + kt].m;- } else {- scale_dst[2 * row + 0] = 0;- scale_dst[2 * row + 1] = 0;- }- }- }- }- }-}--// repack q4_1_tiled tensor into q4_1 data-static void repack_tiled_q4_1(void * data, const ggml_tensor * t, size_t offset, size_t size) {- block_q4_1 * dst_matrix = (block_q4_1 *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- int n_col_tiles = ne1_padded / 32;- int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q4_1;- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;-- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);- size_t row_size_bytes = ggml_row_size(t->type, ne0);- int64_t start_slice = offset / slice_size;- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;- if (end_slice > ne2 * ne3) {- end_slice = ne2 * ne3;- }-- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {- size_t cur_start_byte = (std::max)(offset, (size_t) slice_idx * slice_size);- size_t cur_end_byte = (std::min)(offset + size, (size_t) (slice_idx + 1) * slice_size);- size_t slice_offset_start = cur_start_byte - (size_t) slice_idx * slice_size;- size_t slice_offset_end = cur_end_byte - (size_t) slice_idx * slice_size;-- int64_t start_row = slice_offset_start / row_size_bytes;- int64_t end_row = (slice_offset_end + row_size_bytes - 1) / row_size_bytes;- end_row = (std::min)(end_row, ne1);-- int start_ct = start_row / 32;- int end_ct = (end_row + 31) / 32;- end_ct = (std::min)(end_ct, n_col_tiles);-- block_q4_1 * dst_slice = dst_matrix + (cur_start_byte - offset) / sizeof(block_q4_1);- const uint8_t * matrix_src = (const uint8_t *) t->data + slice_idx * matrix_size;-- for (int ct = start_ct; ct < end_ct; ct++) {- for (int kt = 0; kt < n_k_tiles; kt++) {- const uint8_t * tile_src = matrix_src + (ct * n_k_tiles + kt) * tile_size;-- uint8_t tile_quants[32][32];- for (int cp = 0; cp < 16; cp++) {- for (int row = 0; row < 32; row++) {- uint8_t val = tile_src[cp * 32 + row];- tile_quants[row][2 * cp + 0] = val & 0x0F;- tile_quants[row][2 * cp + 1] = val >> 4;- }- }-- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- if (r >= start_row && r < end_row && kt < ne0 / 32) {- pack_q4_1_quants(&dst_slice[(r - start_row) * (ne0 / 32) + kt], tile_quants[row], 0);- }- }-- const ggml_half * scale_src = (const ggml_half *)(tile_src + 512);- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- if (r >= start_row && r < end_row && kt < ne0 / 32) {- dst_slice[(r - start_row) * (ne0 / 32) + kt].d = scale_src[2 * row];- dst_slice[(r - start_row) * (ne0 / 32) + kt].m = scale_src[2 * row + 1];- }- }- }- }- }-}--// repack q8_0 data into q8_0_tiled tensor-static void repack_q8_0_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {- const block_q8_0 * src_matrix = (const block_q8_0 *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- int n_col_tiles = ne1_padded / 32;- int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q8_0;- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;-- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);- int64_t start_slice = offset / slice_size;- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;- if (end_slice > ne2 * ne3) {- end_slice = ne2 * ne3;- }-- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {- const block_q8_0 * src_slice = src_matrix + (slice_idx - start_slice) * (ne1 * (ne0 / 32));- uint8_t * matrix_dst = (uint8_t *) t->data + slice_idx * matrix_size;-- for (int ct = 0; ct < n_col_tiles; ct++) {- for (int kt = 0; kt < n_k_tiles; kt++) {- uint8_t * tile_dst = matrix_dst + (ct * n_k_tiles + kt) * tile_size;-- for (int cp = 0; cp < 16; cp++) {- int col0 = cp * 2;- int col1 = col0 + 1;- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- const block_q8_0 * b = (r < ne1 && kt < ne0 / 32) ? &src_slice[r * (ne0 / 32) + kt] : NULL;- tile_dst[cp * 64 + 2 * row + 0] = b ? b->qs[col0] : 0;- tile_dst[cp * 64 + 2 * row + 1] = b ? b->qs[col1] : 0;- }- }-- ggml_half * scale_dst = (ggml_half *)(tile_dst + 1024);- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- scale_dst[row] = (r < ne1 && kt < ne0 / 32) ? src_slice[r * (ne0 / 32) + kt].d : 0;- }- }- }- }-}--// repack q8_0_tiled tensor into q8_0 data-static void repack_tiled_q8_0(void * data, const ggml_tensor * t, size_t offset, size_t size) {- block_q8_0 * dst_matrix = (block_q8_0 *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- int n_col_tiles = ne1_padded / 32;- int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q8_0;- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;-- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);- size_t row_size_bytes = ggml_row_size(t->type, ne0);- int64_t start_slice = offset / slice_size;- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;- if (end_slice > ne2 * ne3) {- end_slice = ne2 * ne3;- }-- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {- size_t cur_start_byte = (std::max)(offset, (size_t) slice_idx * slice_size);- size_t cur_end_byte = (std::min)(offset + size, (size_t) (slice_idx + 1) * slice_size);- size_t slice_offset_start = cur_start_byte - (size_t) slice_idx * slice_size;- size_t slice_offset_end = cur_end_byte - (size_t) slice_idx * slice_size;-- int64_t start_row = slice_offset_start / row_size_bytes;- int64_t end_row = (slice_offset_end + row_size_bytes - 1) / row_size_bytes;- end_row = (std::min)(end_row, ne1);-- int start_ct = start_row / 32;- int end_ct = (end_row + 31) / 32;- end_ct = (std::min)(end_ct, n_col_tiles);-- block_q8_0 * dst_slice = dst_matrix + (cur_start_byte - offset) / sizeof(block_q8_0);- const uint8_t * matrix_src = (const uint8_t *) t->data + slice_idx * matrix_size;-- for (int ct = start_ct; ct < end_ct; ct++) {- for (int kt = 0; kt < n_k_tiles; kt++) {- const uint8_t * tile_src = matrix_src + (ct * n_k_tiles + kt) * tile_size;-- for (int cp = 0; cp < 16; cp++) {- int col0 = cp * 2;- int col1 = col0 + 1;- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- if (r >= start_row && r < end_row && kt < ne0 / 32) {- block_q8_0 & b = dst_slice[(r - start_row) * (ne0 / 32) + kt];- b.qs[col0] = tile_src[cp * 64 + 2 * row + 0];- b.qs[col1] = tile_src[cp * 64 + 2 * row + 1];- }- }- }-- const ggml_half * scale_src = (const ggml_half *)(tile_src + 1024);- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- if (r >= start_row && r < end_row && kt < ne0 / 32) {- dst_slice[(r - start_row) * (ne0 / 32) + kt].d = scale_src[row];- }- }- }- }- }-}--// repack mxfp4 data into mxfp4_tiled tensor-static void repack_mxfp4_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {- const block_mxfp4 * src_matrix = (const block_mxfp4 *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- int n_col_tiles = ne1_padded / 32;- int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_MXFP4;- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;-- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);- int64_t start_slice = offset / slice_size;- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;- if (end_slice > ne2 * ne3) {- end_slice = ne2 * ne3;- }-- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {- const block_mxfp4 * src_slice = src_matrix + (slice_idx - start_slice) * (ne1 * (ne0 / 32));- uint8_t * matrix_dst = (uint8_t *) t->data + slice_idx * matrix_size;-- for (int ct = 0; ct < n_col_tiles; ct++) {- for (int kt = 0; kt < n_k_tiles; kt++) {- uint8_t * tile_dst = matrix_dst + (ct * n_k_tiles + kt) * tile_size;-- uint8_t tile_quants[32][32];- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- if (r < ne1 && kt < ne0 / 32) {- unpack_mxfp4_quants(tile_quants[row], &src_slice[r * (ne0 / 32) + kt], 0);- } else {- memset(tile_quants[row], 0, 32);- }- }-- for (int cp = 0; cp < 16; cp++) {- for (int row = 0; row < 32; row++) {- tile_dst[cp * 32 + row] = (tile_quants[row][2 * cp + 1] << 4) | tile_quants[row][2 * cp];- }- }-- uint8_t * scale_dst = tile_dst + 512;- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- scale_dst[row] = (r < ne1 && kt < ne0 / 32) ? src_slice[r * (ne0 / 32) + kt].e : 0;- }- }- }- }-}--// repack mxfp4_tiled tensor into mxfp4 data-static void repack_tiled_mxfp4(void * data, const ggml_tensor * t, size_t offset, size_t size) {- block_mxfp4 * dst_matrix = (block_mxfp4 *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- int n_col_tiles = ne1_padded / 32;- int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_MXFP4;- const size_t matrix_size = n_col_tiles * n_k_tiles * tile_size;-- size_t slice_size = ne1 * ggml_row_size(t->type, ne0);- size_t row_size_bytes = ggml_row_size(t->type, ne0);- int64_t start_slice = offset / slice_size;- int64_t end_slice = (offset + size + slice_size - 1) / slice_size;- if (end_slice > ne2 * ne3) {- end_slice = ne2 * ne3;- }-- for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {- size_t cur_start_byte = (std::max)(offset, (size_t) slice_idx * slice_size);- size_t cur_end_byte = (std::min)(offset + size, (size_t) (slice_idx + 1) * slice_size);- size_t slice_offset_start = cur_start_byte - (size_t) slice_idx * slice_size;- size_t slice_offset_end = cur_end_byte - (size_t) slice_idx * slice_size;-- int64_t start_row = slice_offset_start / row_size_bytes;- int64_t end_row = (slice_offset_end + row_size_bytes - 1) / row_size_bytes;- end_row = (std::min)(end_row, ne1);-- int start_ct = start_row / 32;- int end_ct = (end_row + 31) / 32;- end_ct = (std::min)(end_ct, n_col_tiles);-- block_mxfp4 * dst_slice = dst_matrix + (cur_start_byte - offset) / sizeof(block_mxfp4);- const uint8_t * matrix_src = (const uint8_t *) t->data + slice_idx * matrix_size;-- for (int ct = start_ct; ct < end_ct; ct++) {- for (int kt = 0; kt < n_k_tiles; kt++) {- const uint8_t * tile_src = matrix_src + (ct * n_k_tiles + kt) * tile_size;-- uint8_t tile_quants[32][32];- for (int cp = 0; cp < 16; cp++) {- for (int row = 0; row < 32; row++) {- uint8_t val = tile_src[cp * 32 + row];- tile_quants[row][2 * cp + 0] = val & 0x0F;- tile_quants[row][2 * cp + 1] = val >> 4;- }- }-- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- if (r >= start_row && r < end_row && kt < ne0 / 32) {- pack_mxfp4_quants(&dst_slice[(r - start_row) * (ne0 / 32) + kt], tile_quants[row], 0);- }- }-- const uint8_t * scale_src = tile_src + 512;- for (int row = 0; row < 32; row++) {- int64_t r = ct * 32 + row;- if (r >= start_row && r < end_row && kt < ne0 / 32) {- dst_slice[(r - start_row) * (ne0 / 32) + kt].e = scale_src[row];- }- }- }- }- }-}--// unsigned 6-bit value (0..63) of element e of a Q6_K block, same bit layout as dequantize_row_q6_K-static inline uint8_t q6_K_get_quant(const block_q6_K * b, int e) {- const int c = e / 128;- const int w = e % 128;- const int g = w / 32;- const int l = w % 32;- const uint8_t * ql = b->ql + c * 64;- const uint8_t * qh = b->qh + c * 32;- uint8_t lo, hi;- switch (g) {- case 0: lo = ql[l] & 0xF; hi = (qh[l] >> 0) & 3; break;- case 1: lo = ql[l + 32] & 0xF; hi = (qh[l] >> 2) & 3; break;- case 2: lo = ql[l] >> 4; hi = (qh[l] >> 4) & 3; break;- default: lo = ql[l + 32] >> 4; hi = (qh[l] >> 6) & 3; break;- }- return (uint8_t) (lo | (hi << 4));-}--// tile layout: see HTP_MM_WEIGHT_TILE_SIZE_Q6_K in htp/matmul-ops.h-static void repack_q6_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {- GGML_ASSERT(offset == 0);-- const block_q6_K * src_matrix = (const block_q6_K *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- GGML_ASSERT(ne0 % QK_K == 0);-- const int n_col_tiles = ne1_padded / 32;- const int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q6_K;- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;-- const int64_t sb_per_row = ne0 / QK_K;-- for (int i3 = 0; i3 < ne3; i3++) {- for (int i2 = 0; i2 < ne2; i2++) {- const block_q6_K * src_slice = src_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);- uint8_t * matrix_dst = (uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;-- memset(matrix_dst, 0, matrix_size); // padding rows and the OR-ed nibbles below need zeroed tiles-- for (int64_t r = 0; r < ne1; r++) {- const int ct = (int) (r / 32);- const int row = (int) (r % 32);- const block_q6_K * src_row = src_slice + r * sb_per_row;-- for (int kt = 0; kt < n_k_tiles; kt++) {- const int kt_local = kt % 8; // k-tile within the super-block- const block_q6_K * b = &src_row[kt / 8];- const float d = GGML_FP16_TO_FP32(b->d);-- uint8_t * tile = matrix_dst + ((size_t) ct * n_k_tiles + kt) * tile_size;- uint8_t * lo_pl = tile;- uint8_t * hi_pl = tile + 512;- ggml_half * sc_pl = (ggml_half *) (tile + 768);-- for (int lk = 0; lk < 32; lk++) {- const uint8_t q6 = q6_K_get_quant(b, kt_local * 32 + lk);- const int g = lk >> 2;- const int pos = row * 4 + (lk & 3);- lo_pl[(g >> 1) * 128 + pos] |= (uint8_t) ((q6 & 0xF) << ((g & 1) * 4));- hi_pl[(g >> 2) * 128 + pos] |= (uint8_t) ((q6 >> 4) << ((g & 3) * 2));- }- for (int sub = 0; sub < 2; sub++) {- sc_pl[sub * 32 + row] = GGML_FP32_TO_FP16(d * (float) b->scales[kt_local * 2 + sub]);- }- }- }- }- }-- GGML_UNUSED(size);-}--// Reverse of repack_q6_K_tiled. Unpacks quants losslessly and normalizes sub-block scales. Read-back only.-static void repack_tiled_q6_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {- GGML_ASSERT(offset == 0);-- block_q6_K * dst_matrix = (block_q6_K *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- GGML_ASSERT(ne0 % QK_K == 0);-- const int n_col_tiles = ne1_padded / 32;- const int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q6_K;- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;-- const int64_t sb_per_row = ne0 / QK_K;-- for (int i3 = 0; i3 < ne3; i3++) {- for (int i2 = 0; i2 < ne2; i2++) {- block_q6_K * dst_slice = dst_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);- const uint8_t * matrix_src = (const uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;-- for (int64_t r = 0; r < ne1; r++) {- const int ct = (int) (r / 32);- const int row = (int) (r % 32);- block_q6_K * dst_row = dst_slice + r * sb_per_row;-- for (int64_t sb = 0; sb < sb_per_row; sb++) {- block_q6_K * b = &dst_row[sb];- memset(b, 0, sizeof(block_q6_K));-- float sub_scales[16];- for (int kt_local = 0; kt_local < 8; kt_local++) {- const int kt = sb * 8 + kt_local;- const uint8_t * tile = matrix_src + ((size_t) ct * n_k_tiles + kt) * tile_size;- const uint8_t * lo_pl = tile;- const uint8_t * hi_pl = tile + 512;- const ggml_half * sc_pl = (const ggml_half *) (tile + 768);-- const int c = kt_local / 4;- const int g = kt_local % 4;- uint8_t * ql = b->ql + c * 64;- uint8_t * qh = b->qh + c * 32;-- for (int lk = 0; lk < 32; lk++) {- const int g_tile = lk >> 2;- const int pos = row * 4 + (lk & 3);- const uint8_t lo = (lo_pl[(g_tile >> 1) * 128 + pos] >> ((g_tile & 1) * 4)) & 0xF;- const uint8_t hi = (hi_pl[(g_tile >> 2) * 128 + pos] >> ((g_tile & 3) * 2)) & 3;-- switch (g) {- case 0:- ql[lk] |= lo;- qh[lk] |= (hi << 0);- break;- case 1:- ql[lk + 32] |= lo;- qh[lk] |= (hi << 2);- break;- case 2:- ql[lk] |= (lo << 4);- qh[lk] |= (hi << 4);- break;- default:- ql[lk + 32] |= (lo << 4);- qh[lk] |= (hi << 6);- break;- }- }-- for (int sub = 0; sub < 2; sub++) {- sub_scales[kt_local * 2 + sub] = GGML_FP16_TO_FP32(sc_pl[sub * 32 + row]);- }- }-- float max_abs_scale = 0.0f;- for (int s = 0; s < 16; s++) {- float abs_scale = fabsf(sub_scales[s]);- if (abs_scale > max_abs_scale) {- max_abs_scale = abs_scale;- }- }-- if (max_abs_scale == 0.0f) {- b->d = GGML_FP32_TO_FP16(0.0f);- memset(b->scales, 0, sizeof(b->scales));- } else {- float d_flt = max_abs_scale / 127.0f;- b->d = GGML_FP32_TO_FP16(d_flt);- float d_actual = GGML_FP16_TO_FP32(b->d);- float inv_d = (d_actual != 0.0f) ? (1.0f / d_actual) : 0.0f;- for (int s = 0; s < 16; s++) {- int sc = (int) roundf(sub_scales[s] * inv_d);- b->scales[s] = (int8_t) (std::max)(-128, (std::min)(127, sc));- }- }- }- }- }- }-- GGML_UNUSED(size);-}--// low 2 bits (0..3) of element e of a Q2_K or Q3_K block, same bit layout as dequantize_row_q2_K / q3_K-static inline uint8_t q2_3_K_get_low2(const uint8_t * qs, int e) {- const int c = e / 128;- const int j = (e % 128) / 32;- const int l = e % 32;- return (qs[c * 32 + l] >> (2 * j)) & 3;-}--// hmask bit of element e of a Q3_K block, same bit layout as dequantize_row_q3_K-static inline bool q3_K_get_hbit(const block_q3_K * b, int e) {- const int c = e / 128;- const int j = (e % 128) / 32;- const int l = e % 32;- return (b->hmask[l] >> (c * 4 + j)) & 1;-}--// signed 6-bit scale j (-32..31) of a Q3_K block, same packing as quantize_row_q3_K_ref-static inline int q3_K_get_scale(const uint8_t * scales, int j) {- const int lo = (j < 8) ? (scales[j] & 0xF) : (scales[j - 8] >> 4);- const int hi = (scales[8 + j % 4] >> (2 * (j / 4))) & 3;- return (lo | (hi << 4)) - 32;-}--// read-back: find fp16 d and l[j] in [lmin, lmax] with fp16(d * l[j]) == prod[j] for all j, false if none-static bool hexagon_recover_k_scales(const ggml_half * prod, int n, int lmin, int lmax, ggml_half * d_out, int * l_out) {- int jmax = 0;- for (int j = 1; j < n; j++) {- if (fabsf(GGML_FP16_TO_FP32(prod[j])) > fabsf(GGML_FP16_TO_FP32(prod[jmax]))) {- jmax = j;- }- }- const float pmax = GGML_FP16_TO_FP32(prod[jmax]);- if (pmax == 0.0f) {- *d_out = GGML_FP32_TO_FP16(0.0f);- for (int j = 0; j < n; j++) {- l_out[j] = 0;- }- return true;- }-- // the quantizers put the largest scale at or near the range end, so try large |l| first- const int lext = (std::max)(-lmin, lmax);- for (int a = lext; a >= 1; a--) {- for (int sign : { 1, -1 }) {- const int lj = sign * a;- if (lj < lmin || lj > lmax) {- continue;- }- const ggml_half d0 = GGML_FP32_TO_FP16(pmax / (float) lj);- for (int ulp : { 0, -1, 1 }) {- ggml_half d = d0;- uint16_t bits;- memcpy(&bits, &d, sizeof(bits));- bits = (uint16_t) (bits + ulp);- memcpy(&d, &bits, sizeof(bits));-- const float df = GGML_FP16_TO_FP32(d);- if (!std::isfinite(df) || df == 0.0f) {- continue;- }- bool ok = true;- for (int j = 0; j < n && ok; j++) {- const int l = (int) roundf(GGML_FP16_TO_FP32(prod[j]) / df);- const ggml_half p = GGML_FP32_TO_FP16(df * (float) l);- ok = l >= lmin && l <= lmax && memcmp(&p, &prod[j], sizeof(p)) == 0;- l_out[j] = l;- }- if (ok) {- *d_out = d;- return true;- }- }- }- }- return false;-}--// tile layout: see HTP_MM_WEIGHT_TILE_SIZE_Q3_K in htp/matmul-ops.h-static void repack_q3_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {- GGML_ASSERT(offset == 0);-- const block_q3_K * src_matrix = (const block_q3_K *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- GGML_ASSERT(ne0 % QK_K == 0);-- const int n_col_tiles = ne1_padded / 32;- const int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q3_K;- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;-- const int64_t sb_per_row = ne0 / QK_K;-- for (int i3 = 0; i3 < ne3; i3++) {- for (int i2 = 0; i2 < ne2; i2++) {- const block_q3_K * src_slice = src_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);- uint8_t * matrix_dst = (uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;-- memset(matrix_dst, 0, matrix_size); // padding rows and the OR-ed bits below need zeroed tiles-- for (int64_t r = 0; r < ne1; r++) {- const int ct = (int) (r / 32);- const int row = (int) (r % 32);- const block_q3_K * src_row = src_slice + r * sb_per_row;-- for (int kt = 0; kt < n_k_tiles; kt++) {- const int kt_local = kt % 8; // k-tile within the super-block- const block_q3_K * b = &src_row[kt / 8];- const float d = GGML_FP16_TO_FP32(b->d);-- uint8_t * tile = matrix_dst + ((size_t) ct * n_k_tiles + kt) * tile_size;- uint8_t * lo_pl = tile;- uint8_t * neg_pl = tile + 256;- ggml_half * sc_pl = (ggml_half *) (tile + 384);-- for (int lk = 0; lk < 32; lk++) {- const int e = kt_local * 32 + lk;- const int g = lk >> 2;- const int pos = row * 4 + (lk & 3);- lo_pl[(g >> 2) * 128 + pos] |= (uint8_t) (q2_3_K_get_low2(b->qs, e) << ((g & 3) * 2));- if (!q3_K_get_hbit(b, e)) {- neg_pl[pos] |= (uint8_t) (1 << g);- }- }- for (int sub = 0; sub < 2; sub++) {- sc_pl[sub * 32 + row] = GGML_FP32_TO_FP16(d * (float) q3_K_get_scale(b->scales, kt_local * 2 + sub));- }- }- }- }- }-- GGML_UNUSED(size);-}--// Reverse of repack_q3_K_tiled. Unpacks quants losslessly and normalizes sub-block scales. Read-back only.-static void repack_tiled_q3_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {- GGML_ASSERT(offset == 0);-- block_q3_K * dst_matrix = (block_q3_K *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- GGML_ASSERT(ne0 % QK_K == 0);-- const int n_col_tiles = ne1_padded / 32;- const int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q3_K;- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;-- const int64_t sb_per_row = ne0 / QK_K;-- for (int i3 = 0; i3 < ne3; i3++) {- for (int i2 = 0; i2 < ne2; i2++) {- block_q3_K * dst_slice = dst_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);- const uint8_t * matrix_src = (const uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;-- for (int64_t r = 0; r < ne1; r++) {- const int ct = (int) (r / 32);- const int row = (int) (r % 32);- block_q3_K * dst_row = dst_slice + r * sb_per_row;-- for (int64_t sb = 0; sb < sb_per_row; sb++) {- block_q3_K * b = &dst_row[sb];- memset(b, 0, sizeof(block_q3_K));-- ggml_half sub_scales[16];- for (int kt_local = 0; kt_local < 8; kt_local++) {- const int kt = sb * 8 + kt_local;- const uint8_t * tile = matrix_src + ((size_t) ct * n_k_tiles + kt) * tile_size;- const uint8_t * lo_pl = tile;- const uint8_t * neg_pl = tile + 256;- const ggml_half * sc_pl = (const ggml_half *) (tile + 384);-- for (int lk = 0; lk < 32; lk++) {- const int e = kt_local * 32 + lk;- const int g = lk >> 2;- const int pos = row * 4 + (lk & 3);- const uint8_t lo = (lo_pl[(g >> 2) * 128 + pos] >> ((g & 3) * 2)) & 3;-- const int c = e / 128;- const int j = (e % 128) / 32;- const int l = e % 32;- b->qs[c * 32 + l] |= (uint8_t) (lo << (2 * j));- if (!((neg_pl[pos] >> g) & 1)) {- b->hmask[l] |= (uint8_t) (1 << (c * 4 + j));- }- }-- for (int sub = 0; sub < 2; sub++) {- sub_scales[kt_local * 2 + sub] = sc_pl[sub * 32 + row];- }- }-- int ls[16];- if (!hexagon_recover_k_scales(sub_scales, 16, -32, 31, &b->d, ls)) {- // no exact match: same scale choice as quantize_row_q3_K_ref- float max_scale = 0.0f;- for (int s = 0; s < 16; s++) {- if (fabsf(GGML_FP16_TO_FP32(sub_scales[s])) > fabsf(max_scale)) {- max_scale = GGML_FP16_TO_FP32(sub_scales[s]);- }- }- b->d = GGML_FP32_TO_FP16(-max_scale / 32.0f);- const float d_actual = GGML_FP16_TO_FP32(b->d);- const float inv_d = (d_actual != 0.0f) ? (1.0f / d_actual) : 0.0f;- for (int s = 0; s < 16; s++) {- ls[s] = (std::max)(-32, (std::min)(31, (int) roundf(GGML_FP16_TO_FP32(sub_scales[s]) * inv_d)));- }- }-- for (int s = 0; s < 16; s++) {- const int l = ls[s] + 32;- if (s < 8) {- b->scales[s] = l & 0xF;- } else {- b->scales[s - 8] |= (uint8_t) ((l & 0xF) << 4);- }- b->scales[s % 4 + 8] |= (uint8_t) ((l >> 4) << (2 * (s / 4)));- }- }- }- }- }-- GGML_UNUSED(size);-}--// tile layout: see HTP_MM_WEIGHT_TILE_SIZE_Q2_K in htp/matmul-ops.h-static void repack_q2_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {- GGML_ASSERT(offset == 0);-- const block_q2_K * src_matrix = (const block_q2_K *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- GGML_ASSERT(ne0 % QK_K == 0);-- const int n_col_tiles = ne1_padded / 32;- const int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q2_K;- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;-- const int64_t sb_per_row = ne0 / QK_K;-- for (int i3 = 0; i3 < ne3; i3++) {- for (int i2 = 0; i2 < ne2; i2++) {- const block_q2_K * src_slice = src_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);- uint8_t * matrix_dst = (uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;-- memset(matrix_dst, 0, matrix_size); // padding rows and the OR-ed bits below need zeroed tiles-- for (int64_t r = 0; r < ne1; r++) {- const int ct = (int) (r / 32);- const int row = (int) (r % 32);- const block_q2_K * src_row = src_slice + r * sb_per_row;-- for (int kt = 0; kt < n_k_tiles; kt++) {- const int kt_local = kt % 8; // k-tile within the super-block- const block_q2_K * b = &src_row[kt / 8];-- uint8_t * tile = matrix_dst + ((size_t) ct * n_k_tiles + kt) * tile_size;- uint8_t * lo_pl = tile;- uint8_t * sc_pl = tile + 256;- ggml_half * dd_pl = (ggml_half *) (tile + 320);-- for (int lk = 0; lk < 32; lk++) {- const int g = lk >> 2;- const int pos = row * 4 + (lk & 3);- lo_pl[(g >> 2) * 128 + pos] |= (uint8_t) (q2_3_K_get_low2(b->qs, kt_local * 32 + lk) << ((g & 3) * 2));- }- for (int sub = 0; sub < 2; sub++) {- sc_pl[sub * 32 + row] = b->scales[kt_local * 2 + sub];- }- if (row / 8 == kt_local % 4) {- dd_pl[row % 8] = kt_local < 4 ? b->d : (ggml_half) (b->dmin ^ 0x8000); // -dmin- }- }- }- }- }-- GGML_UNUSED(size);-}--// Reverse of repack_q2_K_tiled. Lossless. Read-back only.-static void repack_tiled_q2_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {- GGML_ASSERT(offset == 0);-- block_q2_K * dst_matrix = (block_q2_K *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- GGML_ASSERT(ne0 % QK_K == 0);-- const int n_col_tiles = ne1_padded / 32;- const int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q2_K;- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;-- const int64_t sb_per_row = ne0 / QK_K;-- for (int i3 = 0; i3 < ne3; i3++) {- for (int i2 = 0; i2 < ne2; i2++) {- block_q2_K * dst_slice = dst_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);- const uint8_t * matrix_src = (const uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;-- for (int64_t r = 0; r < ne1; r++) {- const int ct = (int) (r / 32);- const int row = (int) (r % 32);- block_q2_K * dst_row = dst_slice + r * sb_per_row;-- for (int64_t sb = 0; sb < sb_per_row; sb++) {- block_q2_K * b = &dst_row[sb];- memset(b, 0, sizeof(block_q2_K));-- for (int kt_local = 0; kt_local < 8; kt_local++) {- const int kt = sb * 8 + kt_local;- const uint8_t * tile = matrix_src + ((size_t) ct * n_k_tiles + kt) * tile_size;- const uint8_t * lo_pl = tile;- const uint8_t * sc_pl = tile + 256;- const ggml_half * dd_pl = (const ggml_half *) (tile + 320);-- for (int lk = 0; lk < 32; lk++) {- const int e = kt_local * 32 + lk;- const int g = lk >> 2;- const int pos = row * 4 + (lk & 3);- const uint8_t lo = (lo_pl[(g >> 2) * 128 + pos] >> ((g & 3) * 2)) & 3;- b->qs[(e / 128) * 32 + e % 32] |= (uint8_t) (lo << (2 * ((e % 128) / 32)));- }-- for (int sub = 0; sub < 2; sub++) {- b->scales[kt_local * 2 + sub] = sc_pl[sub * 32 + row];- }- if (row / 8 == kt_local % 4) {- if (kt_local < 4) {- b->d = dd_pl[row % 8];- } else {- b->dmin = (ggml_half) (dd_pl[row % 8] ^ 0x8000);- }- }- }- }- }- }- }-- GGML_UNUSED(size);-}--static inline void get_scale_min_k4(int j, const uint8_t * q, uint8_t * d, uint8_t * m) {- if (j < 4) {- *d = q[j] & 63;- *m = q[j + 4] & 63;- } else {- *d = (q[j + 4] & 0xF) | ((q[j - 4] >> 6) << 4);- *m = (q[j + 4] >> 4) | ((q[j - 0] >> 6) << 4);- }-}--// tile layout: see HTP_MM_WEIGHT_TILE_SIZE_Q4_1 in htp/matmul-ops.h-static void repack_q4_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {- GGML_ASSERT(offset == 0);-- const block_q4_K * src_matrix = (const block_q4_K *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- GGML_ASSERT(ne0 % QK_K == 0);-- const int n_col_tiles = ne1_padded / 32;- const int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q4_1;- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;-- const int64_t sb_per_row = ne0 / QK_K;-- for (int i3 = 0; i3 < ne3; i3++) {- for (int i2 = 0; i2 < ne2; i2++) {- const block_q4_K * src_slice = src_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);- uint8_t * matrix_dst = (uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;-- memset(matrix_dst, 0, matrix_size);-- for (int64_t r = 0; r < ne1; r++) {- const int ct = (int) (r / 32);- const int row = (int) (r % 32);- const block_q4_K * src_row = src_slice + r * sb_per_row;-- for (int kt = 0; kt < n_k_tiles; kt++) {- const int kt_local = kt % 8;- const block_q4_K * b = &src_row[kt / 8];- const float d = GGML_FP16_TO_FP32(b->d);- const float dmin = GGML_FP16_TO_FP32(b->dmin);-- uint8_t * tile_dst = matrix_dst + ((size_t) ct * n_k_tiles + kt) * tile_size;-- uint8_t sc, m;- get_scale_min_k4(kt_local, b->scales, &sc, &m);-- const float D = d * (float) sc;- const float M = -dmin * (float) m;-- const uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;- const int shift = (kt_local & 1) ? 4 : 0;-- for (int cp = 0; cp < 16; cp++) {- const uint8_t q0 = (qs_sub[2 * cp + 0] >> shift) & 0x0F;- const uint8_t q1 = (qs_sub[2 * cp + 1] >> shift) & 0x0F;- tile_dst[cp * 32 + row] = (uint8_t) ((q1 << 4) | q0);- }-- ggml_half * scale_dst = (ggml_half *) (tile_dst + 512);- scale_dst[2 * row + 0] = GGML_FP32_TO_FP16(D);- scale_dst[2 * row + 1] = GGML_FP32_TO_FP16(M);- }- }- }- }-- GGML_UNUSED(size);-}--// Reverse of repack_q4_K_tiled. Unpacks quants and normalizes scales/mins. Read-back only.-static void repack_tiled_q4_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {- GGML_ASSERT(offset == 0);-- block_q4_K * dst_matrix = (block_q4_K *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- GGML_ASSERT(ne0 % QK_K == 0);-- const int n_col_tiles = ne1_padded / 32;- const int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q4_1;- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;-- const int64_t sb_per_row = ne0 / QK_K;-- for (int i3 = 0; i3 < ne3; i3++) {- for (int i2 = 0; i2 < ne2; i2++) {- block_q4_K * dst_slice = dst_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);- const uint8_t * matrix_src = (const uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;-- for (int64_t r = 0; r < ne1; r++) {- const int ct = (int) (r / 32);- const int row = (int) (r % 32);- block_q4_K * dst_row = dst_slice + r * sb_per_row;-- for (int64_t sb = 0; sb < sb_per_row; sb++) {- block_q4_K * b = &dst_row[sb];- memset(b, 0, sizeof(block_q4_K));-- float sub_scales[8];- float sub_mins[8];-- for (int kt_local = 0; kt_local < 8; kt_local++) {- const int kt = sb * 8 + kt_local;- const uint8_t * tile_src = matrix_src + ((size_t) ct * n_k_tiles + kt) * tile_size;- const ggml_half * scale_src = (const ggml_half *) (tile_src + 512);-- uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;- const int shift = (kt_local & 1) ? 4 : 0;-- for (int cp = 0; cp < 16; cp++) {- const uint8_t val = tile_src[cp * 32 + row];- const uint8_t q0 = val & 0x0F;- const uint8_t q1 = val >> 4;- qs_sub[2 * cp + 0] |= (uint8_t) (q0 << shift);- qs_sub[2 * cp + 1] |= (uint8_t) (q1 << shift);- }-- const float D = GGML_FP16_TO_FP32(scale_src[2 * row + 0]);- const float M = GGML_FP16_TO_FP32(scale_src[2 * row + 1]);- sub_scales[kt_local] = (D > 0.0f) ? D : 0.0f;- sub_mins[kt_local] = (-M > 0.0f) ? -M : 0.0f;- }-- float max_scale = 0.0f;- float max_min = 0.0f;- for (int j = 0; j < 8; j++) {- if (sub_scales[j] > max_scale) max_scale = sub_scales[j];- if (sub_mins[j] > max_min) max_min = sub_mins[j];- }-- float inv_scale = 0.0f;- if (max_scale > 0.0f) {- b->d = GGML_FP32_TO_FP16(max_scale / 63.0f);- const float d_actual = GGML_FP16_TO_FP32(b->d);- inv_scale = (d_actual > 0.0f) ? (1.0f / d_actual) : 0.0f;- } else {- b->d = GGML_FP32_TO_FP16(0.0f);- }-- float inv_min = 0.0f;- if (max_min > 0.0f) {- b->dmin = GGML_FP32_TO_FP16(max_min / 63.0f);- const float dmin_actual = GGML_FP16_TO_FP32(b->dmin);- inv_min = (dmin_actual > 0.0f) ? (1.0f / dmin_actual) : 0.0f;- } else {- b->dmin = GGML_FP32_TO_FP16(0.0f);- }-- for (int j = 0; j < 8; j++) {- uint8_t ls = (uint8_t) roundf(inv_scale * sub_scales[j]);- uint8_t lm = (uint8_t) roundf(inv_min * sub_mins[j]);- ls = (std::min)((uint8_t) 63, ls);- lm = (std::min)((uint8_t) 63, lm);- if (j < 4) {- b->scales[j] = ls;- b->scales[j + 4] = lm;- } else {- b->scales[j + 4] = (ls & 0xF) | ((lm & 0xF) << 4);- b->scales[j - 4] |= ((ls >> 4) << 6);- b->scales[j - 0] |= ((lm >> 4) << 6);- }- }- }- }- }- }-- GGML_UNUSED(size);-}--// tile layout: see HTP_MM_WEIGHT_TILE_SIZE_Q5_K in htp/matmul-ops.h-static void repack_q5_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {- GGML_ASSERT(offset == 0);-- const block_q5_K * src_matrix = (const block_q5_K *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- GGML_ASSERT(ne0 % QK_K == 0);-- const int n_col_tiles = ne1_padded / 32;- const int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q5_K;- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;-- const int64_t sb_per_row = ne0 / QK_K;-- for (int i3 = 0; i3 < ne3; i3++) {- for (int i2 = 0; i2 < ne2; i2++) {- const block_q5_K * src_slice = src_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);- uint8_t * matrix_dst = (uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;-- memset(matrix_dst, 0, matrix_size);-- for (int64_t r = 0; r < ne1; r++) {- const int ct = (int) (r / 32);- const int row = (int) (r % 32);- const block_q5_K * src_row = src_slice + r * sb_per_row;-- for (int kt = 0; kt < n_k_tiles; kt++) {- const int kt_local = kt % 8;- const block_q5_K * b = &src_row[kt / 8];- const float d = GGML_FP16_TO_FP32(b->d);- const float dmin = GGML_FP16_TO_FP32(b->dmin);-- uint8_t * tile_dst = matrix_dst + ((size_t) ct * n_k_tiles + kt) * tile_size;- uint8_t * plane = tile_dst + 640;-- uint8_t sc, m;- get_scale_min_k4(kt_local, b->scales, &sc, &m);-- const float D = d * (float) sc;- const float M = -dmin * (float) m;-- const uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;- const int shift = (kt_local & 1) ? 4 : 0;- const uint8_t hbit = (uint8_t) (1 << kt_local);-- for (int cp = 0; cp < 16; cp++) {- const uint8_t q0 = (qs_sub[2 * cp + 0] >> shift) & 0x0F;- const uint8_t q1 = (qs_sub[2 * cp + 1] >> shift) & 0x0F;- tile_dst[cp * 32 + row] = (uint8_t) ((q1 << 4) | q0);-- const int i = cp / 4;- const int lane = (cp % 4) * 32 + row;- if (b->qh[2 * cp + 0] & hbit) {- plane[lane] |= (uint8_t) (1 << (2 * i));- }- if (b->qh[2 * cp + 1] & hbit) {- plane[lane] |= (uint8_t) (1 << (2 * i + 1));- }- }-- ggml_half * scale_dst = (ggml_half *) (tile_dst + 512);- scale_dst[2 * row + 0] = GGML_FP32_TO_FP16(D);- scale_dst[2 * row + 1] = GGML_FP32_TO_FP16(M);- }- }- }- }-- GGML_UNUSED(size);-}--// Reverse of repack_q5_K_tiled. Unpacks quants losslessly and normalizes scales/mins. Read-back only.-static void repack_tiled_q5_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {- GGML_ASSERT(offset == 0);-- block_q5_K * dst_matrix = (block_q5_K *) data;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- int64_t ne2 = t->ne[2];- int64_t ne3 = t->ne[3];- int64_t ne0_padded = hex_round_up(ne0, 32);- int64_t ne1_padded = hex_round_up(ne1, 32);-- GGML_ASSERT(ne0 % QK_K == 0);-- const int n_col_tiles = ne1_padded / 32;- const int n_k_tiles = ne0_padded / 32;- const size_t tile_size = HTP_MM_WEIGHT_TILE_SIZE_Q5_K;- const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;-- const int64_t sb_per_row = ne0 / QK_K;-- for (int i3 = 0; i3 < ne3; i3++) {- for (int i2 = 0; i2 < ne2; i2++) {- block_q5_K * dst_slice = dst_matrix + (i3 * ne2 + i2) * (ne1 * sb_per_row);- const uint8_t * matrix_src = (const uint8_t *) t->data + (i3 * ne2 + i2) * matrix_size;-- for (int64_t r = 0; r < ne1; r++) {- const int ct = (int) (r / 32);- const int row = (int) (r % 32);- block_q5_K * dst_row = dst_slice + r * sb_per_row;-- for (int64_t sb = 0; sb < sb_per_row; sb++) {- block_q5_K * b = &dst_row[sb];- memset(b, 0, sizeof(block_q5_K));-- float sub_scales[8];- float sub_mins[8];-- for (int kt_local = 0; kt_local < 8; kt_local++) {- const int kt = sb * 8 + kt_local;- const uint8_t * tile_src = matrix_src + ((size_t) ct * n_k_tiles + kt) * tile_size;- const uint8_t * plane = tile_src + 640;- const ggml_half * scale_src = (const ggml_half *) (tile_src + 512);-- uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;- const int shift = (kt_local & 1) ? 4 : 0;- const uint8_t hbit = (uint8_t) (1 << kt_local);-- for (int cp = 0; cp < 16; cp++) {- const uint8_t val = tile_src[cp * 32 + row];- const uint8_t q0 = val & 0x0F;- const uint8_t q1 = val >> 4;- qs_sub[2 * cp + 0] |= (uint8_t) (q0 << shift);- qs_sub[2 * cp + 1] |= (uint8_t) (q1 << shift);-- const int i = cp / 4;- const int lane = (cp % 4) * 32 + row;- if (plane[lane] & (1 << (2 * i))) {- b->qh[2 * cp + 0] |= hbit;- }- if (plane[lane] & (1 << (2 * i + 1))) {- b->qh[2 * cp + 1] |= hbit;- }- }-- const float D = GGML_FP16_TO_FP32(scale_src[2 * row + 0]);- const float M = GGML_FP16_TO_FP32(scale_src[2 * row + 1]);- sub_scales[kt_local] = (D > 0.0f) ? D : 0.0f;- sub_mins[kt_local] = (-M > 0.0f) ? -M : 0.0f;- }-- float max_scale = 0.0f;- float max_min = 0.0f;- for (int j = 0; j < 8; j++) {- if (sub_scales[j] > max_scale) max_scale = sub_scales[j];- if (sub_mins[j] > max_min) max_min = sub_mins[j];- }-- float inv_scale = 0.0f;- if (max_scale > 0.0f) {- b->d = GGML_FP32_TO_FP16(max_scale / 63.0f);- const float d_actual = GGML_FP16_TO_FP32(b->d);- inv_scale = (d_actual > 0.0f) ? (1.0f / d_actual) : 0.0f;- } else {- b->d = GGML_FP32_TO_FP16(0.0f);- }-- float inv_min = 0.0f;- if (max_min > 0.0f) {- b->dmin = GGML_FP32_TO_FP16(max_min / 63.0f);- const float dmin_actual = GGML_FP16_TO_FP32(b->dmin);- inv_min = (dmin_actual > 0.0f) ? (1.0f / dmin_actual) : 0.0f;- } else {- b->dmin = GGML_FP32_TO_FP16(0.0f);- }+ return ptr;+ }- for (int j = 0; j < 8; j++) {- uint8_t ls = (uint8_t) roundf(inv_scale * sub_scales[j]);- uint8_t lm = (uint8_t) roundf(inv_min * sub_mins[j]);- ls = (std::min)((uint8_t) 63, ls);- lm = (std::min)((uint8_t) 63, lm);- if (j < 4) {- b->scales[j] = ls;- b->scales[j + 4] = lm;- } else {- b->scales[j + 4] = (ls & 0xF) | ((lm & 0xF) << 4);- b->scales[j - 4] |= ((ls >> 4) << 6);- b->scales[j - 0] |= ((lm >> 4) << 6);- }- }- }- }+ void free_slot(void * ptr, uint32_t n_slots = 1) {+ if (!ptr) return;+ uint32_t slot = ((uint8_t *) ptr - base()) / GGML_HEXAGON_FENCE_SLOT_SIZE;+ for (uint32_t i = 0; i < n_slots; i++) {+ free_slots.push_back(slot + i);}}+};- GGML_UNUSED(size);+inline uint8_t * ggml_hexagon_session::alloc_fence(uint32_t n_slots) {+ uint8_t * ptr = fence_buf->alloc_slot(n_slots);+ GGML_ASSERT(ptr);+ return ptr;}-static void repack_tensor_tiled(ggml_tensor * tensor, const void * data, size_t size) {- switch (tensor->type) {- case GGML_TYPE_Q4_0:- repack_q4_0_tiled(tensor, data, 0, size);- break;-- case GGML_TYPE_Q4_1:- repack_q4_1_tiled(tensor, data, 0, size);- break;-- case GGML_TYPE_Q4_K:- repack_q4_K_tiled(tensor, data, 0, size);- break;-- case GGML_TYPE_Q8_0:- repack_q8_0_tiled(tensor, data, 0, size);- break;+inline void ggml_hexagon_session::free_fence(void * ptr, uint32_t n_slots) {+ if (fence_buf) {+ fence_buf->free_slot(ptr, n_slots);+ }+}- case GGML_TYPE_IQ4_NL:- repack_q4_0_tiled(tensor, data, 0, size);- break;+static void ggml_backend_hexagon_buffer_free_buffer(ggml_backend_buffer_t buffer) {+ auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(buffer->context);+ sbuf->sess->unclone_buffer(sbuf);+ delete sbuf;+}- case GGML_TYPE_MXFP4:- repack_mxfp4_tiled(tensor, data, 0, size);- break;+static void * ggml_backend_hexagon_buffer_get_base(ggml_backend_buffer_t buffer) {+ auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(buffer->context);+ return sbuf->base();+}- case GGML_TYPE_Q5_K:- repack_q5_K_tiled(tensor, data, 0, size);- break;+static enum ggml_status ggml_backend_hexagon_buffer_init_tensor(ggml_backend_buffer_t buffer, ggml_tensor * tensor) {+ auto sbuf = static_cast<ggml_hexagon_shared_buffer *>(buffer->context);+ auto sess = sbuf->sess;- case GGML_TYPE_Q6_K:- repack_q6_K_tiled(tensor, data, 0, size);- break;+ HEX_VERBOSE("ggml-hex: %s init-tensor %s : base %p data %p nbytes %zu\n", sess->c_name(),+ tensor->name, (void *) sbuf->base(), tensor->data, ggml_nbytes(tensor));- case GGML_TYPE_Q3_K:- repack_q3_K_tiled(tensor, data, 0, size);- break;+ auto extra = new ggml_hexagon_tensor_extra();+ sbuf->tensor_extra.push_back(extra);- case GGML_TYPE_Q2_K:- repack_q2_K_tiled(tensor, data, 0, size);- break;+ tensor->extra = extra;- default:- break;- }+ return GGML_STATUS_SUCCESS;}static void ggml_backend_hexagon_buffer_set_tensor(ggml_backend_buffer_t buffer,@@ -2474,21 +907,21 @@ static void ggml_backend_hexagon_buffer_set_tensor(ggml_backend_buffer_t buffer,if (ggml_backend_buffer_get_usage(buffer) == GGML_BACKEND_BUFFER_USAGE_WEIGHTS) {extra->flags |= GGML_HEXAGON_TENSOR_WEIGHT;- if (ggml_hexagon_is_repack_type(tensor->type)) {- extra->flags |= GGML_HEXAGON_TENSOR_REPACK;+ if (htp_is_tiled_weight_tensor(tensor)) {+ extra->flags |= GGML_HEXAGON_TENSOR_TILED;}}HEX_VERBOSE("ggml-hex: %s set-tensor %s : data %p offset %zu size %zu usage %d flags 0x%x\n",sess->c_name(), tensor->name, data, offset, size, (int) buffer->usage, extra->flags);- if ((extra->flags & GGML_HEXAGON_TENSOR_REPACK) == 0) {+ if ((extra->flags & GGML_HEXAGON_TENSOR_TILED) == 0) {memcpy((char *) tensor->data + offset, data, size);return;}if (offset == 0 && size == ggml_nbytes(tensor) && extra->shadow_buf.empty()) {- repack_tensor_tiled(tensor, data, size);+ htp_repack_tensor_tiled(tensor, data, size);return;}@@ -2499,7 +932,7 @@ static void ggml_backend_hexagon_buffer_set_tensor(ggml_backend_buffer_t buffer,extra->shadow_size += size;if (extra->shadow_size >= ggml_nbytes(tensor)) {- repack_tensor_tiled(tensor, extra->shadow_buf.data(), extra->shadow_buf.size());+ htp_repack_tensor_tiled(tensor, extra->shadow_buf.data(), extra->shadow_buf.size());extra->shadow_buf.clear();extra->shadow_buf.shrink_to_fit();extra->shadow_size = 0;@@ -2518,76 +951,13 @@ static void ggml_backend_hexagon_buffer_get_tensor(ggml_backend_buffer_t buffer,HEX_VERBOSE("ggml-hex: %s get-tensor %s : data %p offset %zu size %zu usage %d flags 0x%x\n",sess->c_name(), tensor->name, data, offset, size, (int) buffer->usage, extra->flags);- if ((extra->flags & GGML_HEXAGON_TENSOR_REPACK) == 0) {+ if ((extra->flags & GGML_HEXAGON_TENSOR_TILED) == 0) {memcpy(data, (const char *) tensor->data + offset, size);return;}- switch (tensor->type) {- case GGML_TYPE_Q4_0:- GGML_ASSERT(offset == 0);- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));- repack_tiled_q4_0(data, tensor, offset, size);- break;-- case GGML_TYPE_Q4_1:- GGML_ASSERT(offset == 0);- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));- repack_tiled_q4_1(data, tensor, offset, size);- break;-- case GGML_TYPE_Q4_K:- GGML_ASSERT(offset == 0);- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));- repack_tiled_q4_K(data, tensor, offset, size);- break;-- case GGML_TYPE_Q5_K:- GGML_ASSERT(offset == 0);- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));- repack_tiled_q5_K(data, tensor, offset, size);- break;-- case GGML_TYPE_Q8_0:- GGML_ASSERT(offset == 0);- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));- repack_tiled_q8_0(data, tensor, offset, size);- break;-- case GGML_TYPE_IQ4_NL:- GGML_ASSERT(offset == 0);- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));- repack_tiled_q4_0(data, tensor, offset, size);- break;-- case GGML_TYPE_MXFP4:- GGML_ASSERT(offset == 0);- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));- repack_tiled_mxfp4(data, tensor, offset, size);- break;-- case GGML_TYPE_Q6_K:- GGML_ASSERT(offset == 0);- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));- repack_tiled_q6_K(data, tensor, offset, size);- break;-- case GGML_TYPE_Q3_K:- GGML_ASSERT(offset == 0);- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));- repack_tiled_q3_K(data, tensor, offset, size);- break;-- case GGML_TYPE_Q2_K:- GGML_ASSERT(offset == 0);- GGML_ASSERT(offset + size <= ggml_nbytes(tensor));- repack_tiled_q2_K(data, tensor, offset, size);- break;-- default:- memcpy(data, (const char *) tensor->data + offset, size);- break;- }+ GGML_ASSERT(offset == 0);+ htp_unpack_tensor_tiled(data, tensor, offset, size);}static bool ggml_backend_hexagon_buffer_cpy_tensor(ggml_backend_buffer_t buffer,@@ -2615,15 +985,15 @@ static void ggml_backend_hexagon_buffer_set_tensor_2d(ggml_backend_buffer_t buffif (ggml_backend_buffer_get_usage(buffer) == GGML_BACKEND_BUFFER_USAGE_WEIGHTS) {extra->flags |= GGML_HEXAGON_TENSOR_WEIGHT;- if (ggml_hexagon_is_repack_type(tensor->type)) {- extra->flags |= GGML_HEXAGON_TENSOR_REPACK;+ if (htp_is_tiled_weight_tensor(tensor)) {+ extra->flags |= GGML_HEXAGON_TENSOR_TILED;}}HEX_VERBOSE("ggml-hex: %s set-tensor-2d %s : data %p offset %zu size %zu n_copies %zu stride_tensor %zu stride_data %zu usage %d flags 0x%x\n",sess->c_name(), tensor->name, data, offset, size, n_copies, stride_tensor, stride_data, (int) buffer->usage, extra->flags);- if ((extra->flags & GGML_HEXAGON_TENSOR_REPACK) == 0) {+ if ((extra->flags & GGML_HEXAGON_TENSOR_TILED) == 0) {for (size_t i = 0; i < n_copies; i++) {memcpy((uint8_t *) tensor->data + offset + i * stride_tensor, (const uint8_t *) data + i * stride_data, size);}@@ -2639,7 +1009,7 @@ static void ggml_backend_hexagon_buffer_set_tensor_2d(ggml_backend_buffer_t buffextra->shadow_size += n_copies * size;if (extra->shadow_size >= ggml_nbytes(tensor)) {- repack_tensor_tiled(tensor, extra->shadow_buf.data(), extra->shadow_buf.size());+ htp_repack_tensor_tiled(tensor, extra->shadow_buf.data(), extra->shadow_buf.size());extra->shadow_buf.clear();extra->shadow_buf.shrink_to_fit();extra->shadow_size = 0;@@ -2661,7 +1031,7 @@ static void ggml_backend_hexagon_buffer_get_tensor_2d(ggml_backend_buffer_t buffHEX_VERBOSE("ggml-hex: %s get-tensor-2d %s : data %p offset %zu size %zu n_copies %zu stride_tensor %zu stride_data %zu usage %d\n",sess->c_name(), tensor->name, data, offset, size, n_copies, stride_tensor, stride_data, (int) buffer->usage);- if ((extra->flags & GGML_HEXAGON_TENSOR_REPACK) == 0) {+ if ((extra->flags & GGML_HEXAGON_TENSOR_TILED) == 0) {for (size_t i = 0; i < n_copies; i++) {memcpy((uint8_t *)data + i * stride_data, (const uint8_t *)tensor->data + offset + i * stride_tensor, size);}@@ -2679,52 +1049,7 @@ static void ggml_backend_hexagon_buffer_get_tensor_2d(ggml_backend_buffer_t buffGGML_ASSERT((offset + temp_size) <= ggml_nbytes(tensor));std::vector<uint8_t> temp_buf(temp_size);-- switch (tensor->type) {- case GGML_TYPE_Q4_0:- repack_tiled_q4_0(temp_buf.data(), tensor, offset, temp_size);- break;-- case GGML_TYPE_Q4_1:- repack_tiled_q4_1(temp_buf.data(), tensor, offset, temp_size);- break;-- case GGML_TYPE_Q4_K:- repack_tiled_q4_K(temp_buf.data(), tensor, offset, temp_size);- break;-- case GGML_TYPE_Q5_K:- repack_tiled_q5_K(temp_buf.data(), tensor, offset, temp_size);- break;-- case GGML_TYPE_Q8_0:- repack_tiled_q8_0(temp_buf.data(), tensor, offset, temp_size);- break;-- case GGML_TYPE_IQ4_NL:- repack_tiled_q4_0(temp_buf.data(), tensor, offset, temp_size);- break;-- case GGML_TYPE_MXFP4:- repack_tiled_mxfp4(temp_buf.data(), tensor, offset, temp_size);- break;-- case GGML_TYPE_Q6_K:- repack_tiled_q6_K(temp_buf.data(), tensor, offset, temp_size);- break;-- case GGML_TYPE_Q3_K:- repack_tiled_q3_K(temp_buf.data(), tensor, offset, temp_size);- break;-- case GGML_TYPE_Q2_K:- repack_tiled_q2_K(temp_buf.data(), tensor, offset, temp_size);- break;-- default:- memcpy(temp_buf.data(), (const uint8_t *) tensor->data + offset, temp_size);- break;- }+ htp_unpack_tensor_tiled(temp_buf.data(), tensor, offset, temp_size);for (size_t i = 0; i < n_copies; i++) {memcpy((uint8_t *) data + i * stride_data, temp_buf.data() + i * stride_tensor, size);@@ -2834,16 +1159,15 @@ static size_t ggml_backend_hexagon_buffer_type_get_alignment(ggml_backend_buffer}static size_t ggml_backend_hexagon_buffer_type_get_alloc_size(ggml_backend_buffer_type_t buft, const struct ggml_tensor * t) {- if (ggml_hexagon_is_repack_type(t->type)) {+ if (!ggml_backend_buft_is_host(buft) && htp_is_tiled_weight_tensor(t)) {int64_t ne0 = hex_round_up(t->ne[0], 32);int64_t ne1 = hex_round_up(t->ne[1], 32);int64_t ne2 = t->ne[2];int64_t ne3 = t->ne[3];- return ggml_hexagon_tiled_row_size(t->type, ne0) * ne1 * ne2 * ne3;+ size_t tiled_size = htp_tiled_row_size(t->type, ne0) * ne1 * ne2 * ne3;+ return std::max(tiled_size, ggml_nbytes(t));}return ggml_nbytes(t);-- GGML_UNUSED(buft);}static size_t parse_size(const char * str, size_t default_unit = 1024 * 1024) {@@ -3196,20 +1520,19 @@ struct ggml_hexagon_opbatch {bool same_shape(const htp_tensor * h, const ggml_tensor * t) const {auto extra = (ggml_hexagon_tensor_extra *) t->extra;- int64_t ne0 = t->ne[0];- int64_t ne1 = t->ne[1];- const bool is_repack = (extra->flags & GGML_HEXAGON_TENSOR_REPACK) != 0;- if (is_repack) {- ne0 = hex_round_up(ne0, 32);- ne1 = hex_round_up(ne1, 32);- }- int64_t nb1 = is_repack ? (int64_t) ggml_hexagon_tiled_row_size(t->type, ne0) : t->nb[1];- int64_t nb2 = is_repack ? nb1 * ne1 : t->nb[2];- int64_t nb3 = is_repack ? nb2 * t->ne[2] : t->nb[3];+ const bool is_tiled = (extra->flags & GGML_HEXAGON_TENSOR_TILED) != 0;+ const int64_t ne0_padded = hex_round_up(t->ne[0], 32);+ const int64_t ne1_padded = hex_round_up(t->ne[1], 32);++ int64_t nb1 = is_tiled ? (int64_t) htp_tiled_row_size(t->type, ne0_padded) : t->nb[1];+ int64_t nb2 = is_tiled ? nb1 * ne1_padded : t->nb[2];+ int64_t nb3 = is_tiled ? nb2 * t->ne[2] : t->nb[3];+ int64_t nb0 = (is_tiled && (t->type == GGML_TYPE_F32 || t->type == GGML_TYPE_BF16)) ? sizeof(ggml_fp16_t) : t->nb[0];+ enum ggml_type expected_type = (is_tiled && (t->type == GGML_TYPE_F32 || t->type == GGML_TYPE_BF16)) ? GGML_TYPE_F16 : t->type;- return (h->type == t->type) &&- (h->ne[0] == ne0) && (h->ne[1] == ne1) && (h->ne[2] == t->ne[2]) && (h->ne[3] == t->ne[3]) &&- (h->nb[0] == t->nb[0]) && (h->nb[1] == nb1) && (h->nb[2] == nb2) && (h->nb[3] == nb3);+ return (h->type == expected_type) &&+ (h->ne[0] == t->ne[0]) && (h->ne[1] == t->ne[1]) && (h->ne[2] == t->ne[2]) && (h->ne[3] == t->ne[3]) &&+ (h->nb[0] == nb0) && (h->nb[1] == nb1) && (h->nb[2] == nb2) && (h->nb[3] == nb3);}// add tensor and return its index@@ -3243,23 +1566,28 @@ struct ggml_hexagon_opbatch {h.ti = ti;h.data = t_offset;h.type = t->type;-- const bool is_repack = (extra->flags & GGML_HEXAGON_TENSOR_REPACK) != 0;- if (is_repack) {- h.ne[0] = hex_round_up(t->ne[0], 32);- h.ne[1] = hex_round_up(t->ne[1], 32);- h.ne[2] = t->ne[2];- h.ne[3] = t->ne[3];-- h.nb[0] = t->nb[0];- h.nb[1] = ggml_hexagon_tiled_row_size(t->type, h.ne[0]);- h.nb[2] = h.nb[1] * h.ne[1];+ h.ne[0] = t->ne[0];+ h.ne[1] = t->ne[1];+ h.ne[2] = t->ne[2];+ h.ne[3] = t->ne[3];++ const bool is_tiled = (extra->flags & GGML_HEXAGON_TENSOR_TILED) != 0;+ if (is_tiled) {+ if (t->type == GGML_TYPE_F32 || t->type == GGML_TYPE_BF16) {+ h.type = GGML_TYPE_F16;+ h.nb[0] = sizeof(ggml_fp16_t);+ } else {+ h.nb[0] = t->nb[0];+ }+ const int64_t ne0_padded = hex_round_up(t->ne[0], 32);+ const int64_t ne1_padded = hex_round_up(t->ne[1], 32);+ h.nb[1] = htp_tiled_row_size(t->type, ne0_padded);+ h.nb[2] = h.nb[1] * ne1_padded;h.nb[3] = h.nb[2] * h.ne[2];h.size = h.nb[3] * h.ne[3];t_size = h.size;} else {h.size = t_size;- h.ne[0] = t->ne[0]; h.ne[1] = t->ne[1]; h.ne[2] = t->ne[2]; h.ne[3] = t->ne[3];h.nb[0] = t->nb[0]; h.nb[1] = t->nb[1]; h.nb[2] = t->nb[2]; h.nb[3] = t->nb[3];}@@ -3267,8 +1595,8 @@ struct ggml_hexagon_opbatch {if ((extra->flags & GGML_HEXAGON_TENSOR_WEIGHT) != 0) {h.flags |= HTP_TENSOR_WEIGHT;}- if ((extra->flags & GGML_HEXAGON_TENSOR_REPACK) != 0) {- h.flags |= HTP_TENSOR_REPACK;+ if ((extra->flags & GGML_HEXAGON_TENSOR_TILED) != 0) {+ h.flags |= HTP_TENSOR_TILED;}if ((extra->flags & GGML_HEXAGON_TENSOR_FENCE) != 0) {h.flags |= HTP_TENSOR_FENCE;@@ -3666,6 +1994,9 @@ struct ggml_hexagon_opbatch {if (src2->type != GGML_TYPE_F32) return false;+ // the fused kernel writes the shape of the MUL_MAT output: the ADD must not broadcast it+ if (!ggml_are_same_shape(mm_out, node.dst())) return false;+const struct htp_mm_kernel_params * orig_kparams = (const struct htp_mm_kernel_params *) last_node.kernel_params;struct htp_mm_kernel_params kparams;ggml_hexagon_precompute_fused_matmul_add_params(sess, src0, src1, src2, node.dst(), &kparams);@@ -3753,7 +2084,8 @@ struct ggml_hexagon_opbatch {const ggml_tensor * w0 = last_node.inputs[0];const ggml_tensor * x = last_node.inputs[curr_n];- if (x_in != x || w_in->type != w0->type || w_in->ne[0] != w0->ne[0]) {+ if (x_in != x || w_in->type != w0->type || w_in->ne[0] != w0->ne[0] ||+ ggml_hexagon_tensor_is_tiled(w_in) != ggml_hexagon_tensor_is_tiled(w0)) {return false;}if (!last_node.fused.empty() && (mm_is_hmx_eligible(last_node.fused[0]) != mm_is_hmx_eligible(node.node))) {@@ -3909,7 +2241,8 @@ struct ggml_hexagon_opbatch {const ggml_tensor * x = last_node.inputs[curr_n];const ggml_tensor * ids = last_node.inputs[curr_n + 1];- if (x_in != x || ids_in != ids || w_in->type != w0->type || w_in->ne[0] != w0->ne[0] || w_in->ne[2] != w0->ne[2]) {+ if (x_in != x || ids_in != ids || w_in->type != w0->type || w_in->ne[0] != w0->ne[0] || w_in->ne[2] != w0->ne[2] ||+ ggml_hexagon_tensor_is_tiled(w_in) != ggml_hexagon_tensor_is_tiled(w0)) {return false;}if (!last_node.fused.empty() && (mm_is_hmx_eligible(last_node.fused[0]) != mm_is_hmx_eligible(node.node))) {@@ -5433,7 +3766,6 @@ static bool ggml_hexagon_matmul_is_hmx_eligible(const struct ggml_tensor * src0,const struct ggml_tensor * src1,const struct ggml_tensor * dst,- int ne01_padded,bool is_matmul_id,bool is_batched) {@@ -5442,17 +3774,18 @@ static bool ggml_hexagon_matmul_is_hmx_eligible(}const int ne00 = src0->ne[0];+ const int ne01 = src0->ne[1];const int ne11 = src1->ne[1];const int ne12 = src1->ne[2];const int wtype = src0->type;// HMX weight tiles accept non-32-aligned N for non-matmul_id.- if (ne01_padded % 32 != 0 && is_matmul_id) {+ if (ne01 % 32 != 0 && is_matmul_id) {return false;}// HMX supports F16, F32, and repack quantized types.- if (!ggml_hexagon_is_hmx_weight_type((ggml_type) wtype)) {+ if (!htp_is_hmx_weight_type((ggml_type) wtype)) {return false;}@@ -5461,8 +3794,12 @@ static bool ggml_hexagon_matmul_is_hmx_eligible(return false;}- // Quantized HMX kernels only handle flat 2D matmul (or matmul_id wrapping flat 2D matmuls).- if (!is_matmul_id && is_batched && wtype != GGML_TYPE_F16) {+ // Quantized and tiled HMX kernels only handle flat 2D matmul (or matmul_id wrapping flat 2D matmuls).+ const bool is_tiled = ggml_hexagon_tensor_is_tiled(src0);+ if (ggml_is_quantized((ggml_type) wtype) && !is_tiled) {+ return false;+ }+ if (!is_matmul_id && is_batched && is_tiled) {return false;}@@ -5501,7 +3838,8 @@ static bool ggml_hexagon_precompute_hmx_mm_params(size_t vtcm_budget,struct htp_mm_kernel_params * kparams) {- const int aligned_tile_size = htp_mm_get_weight_aligned_tile_size(wtype);+ const bool is_tiled = ggml_hexagon_tensor_is_tiled(src0);+ const int aligned_tile_size = is_tiled ? htp_tensor_get_tile_padded_size(wtype) : 0;const int n_threads = (int)sess->n_threads;const int ne10 = act->ne[0];@@ -5528,7 +3866,7 @@ static bool ggml_hexagon_precompute_hmx_mm_params(bool use_grouped = false;int act_threads_selected = 0;- if (is_batched_val && wtype == GGML_TYPE_F16 && group_size > 1) {+ if (is_batched_val && wtype == GGML_TYPE_F16 && !is_tiled && group_size > 1) {// Try grouped path firstif (htp_mm_hmx_solve_batched_params(wtype, ne00_padded, ne01_padded, m_for_solver, group_size, n_threads, pipeline, src2_size, vtcm_budget, &m_chunk, &n_chunk, &act_threads_selected, &vtcm_size)) {use_grouped = true;@@ -5559,9 +3897,9 @@ static bool ggml_hexagon_precompute_hmx_mm_params(kparams->n_chunk = n_chunk;kparams->n_threads = n_threads;kparams->n_act_threads = act_threads_selected;- kparams->tile_size = htp_mm_get_weight_tile_size(wtype);+ kparams->tile_size = is_tiled ? htp_tensor_get_tile_size(wtype) : 0;kparams->aligned_tile_size = aligned_tile_size;- kparams->act_row_size = htp_mm_weight_has_offset(wtype) ? htp_mm_q8_1_tiled_row_size(ne10) : htp_mm_q8_0_tiled_row_size(ne10);+ kparams->act_row_size = htp_mm_tiled_act_row_size(wtype, ne10);kparams->vtcm_size = vtcm_size;kparams->vtcm_src0_size = 0;kparams->div_n_act_threads = init_fastdiv_values(act_threads_selected);@@ -5571,13 +3909,11 @@ static bool ggml_hexagon_precompute_hmx_mm_params(kparams->vtcm_dst_size = 0;if (is_batched && !is_matmul_id) {- kparams->kernel_type = HTP_MM_KERNEL_HMX_F16_BATCHED;+ kparams->kernel_type = HTP_MM_KERNEL_HMX_BATCHED;} else {kparams->kernel_type = HTP_MM_KERNEL_HMX_2D;}return true;-- GGML_UNUSED(src0);}static void ggml_hexagon_precompute_hvx_mm_params(@@ -5586,6 +3922,7 @@ static void ggml_hexagon_precompute_hvx_mm_params(const struct ggml_tensor * src1,const struct ggml_tensor * dst,int wtype,+ bool is_tiled,int ne02,int ne03,int ne10,@@ -5608,19 +3945,19 @@ static void ggml_hexagon_precompute_hvx_mm_params(GGML_UNUSED(ne02);GGML_UNUSED(ne03);- const bool is_quant = (wtype != GGML_TYPE_F16 && wtype != GGML_TYPE_F32);const int src1_nrows = ne11 * ne12 * ne13;- if (is_quant) {- // Quantized HVX- kparams->tile_size = htp_mm_get_weight_tile_size(wtype);- kparams->aligned_tile_size = htp_mm_get_weight_aligned_tile_size(wtype);+ if (is_tiled) {+ // Tiled HVX (quantized weights or tiled F16/F32/BF16)+ kparams->tile_size = htp_tensor_get_tile_size(wtype);+ kparams->aligned_tile_size = htp_tensor_get_tile_padded_size(wtype);const bool k_align = (ne10 % 32 == 0);if (is_matmul_id) {- kparams->kernel_type = (src1_nrows < (int) sess->n_threads) ? HTP_MM_KERNEL_HVX_QUANT_BLOCK : HTP_MM_KERNEL_HVX_QUANT_ROW;- kparams->act_row_size = htp_mm_weight_has_offset(wtype) ? htp_mm_q8_1_tiled_row_size(ne10) : htp_mm_q8_0_tiled_row_size(ne10);+ kparams->kernel_type = (src1_nrows < (int) sess->n_threads)+ ? HTP_MM_KERNEL_HVX_QUANT_BLOCK : HTP_MM_KERNEL_HVX_QUANT_ROW;+ kparams->act_row_size = htp_mm_tiled_act_row_size(wtype, ne10);struct htp_mm_hvx_vtcm_layout L;uint32_t max_prefetch = (src1_nrows > HTP_MM_HMX_MIN_NROWS) ? 2 : 16;@@ -5645,13 +3982,11 @@ static void ggml_hexagon_precompute_hvx_mm_params(kparams->vtcm_act_size = L.act_bytes;kparams->vtcm_bias_size = 0;kparams->vtcm_dst_size = L.dst_bytes;- goto done_quant;+ return;} else {bool try_tiled = (k_align && opt_mm_select >= 1);if (try_tiled) {- kparams->act_row_size = htp_mm_weight_has_offset(wtype)- ? htp_mm_q8_1_tiled_row_size(ne10)- : htp_mm_q8_0_tiled_row_size(ne10);+ kparams->act_row_size = htp_mm_tiled_act_row_size(wtype, ne10);if (src1_nrows < (int) sess->n_threads) {kparams->kernel_type = HTP_MM_KERNEL_HVX_QUANT_BLOCK;} else {@@ -5684,17 +4019,18 @@ static void ggml_hexagon_precompute_hvx_mm_params(kparams->vtcm_act_size = L.act_bytes;kparams->vtcm_bias_size = L.bias_bytes;kparams->vtcm_dst_size = L.dst_bytes;- goto done_quant;+ return;}}kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;return;}-- done_quant:;+ } else if (is_matmul_id) {+ kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;+ return;} else if (wtype == GGML_TYPE_F16) {- // F16 HVX+ // Flat F16 HVXstruct htp_mm_hvx_vtcm_layout L;uint32_t m_chunk = 0;if (htp_mm_hvx_solve_vtcm_params(@@ -5715,8 +4051,8 @@ static void ggml_hexagon_precompute_hvx_mm_params(kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;return;- } else {- // F32 HVX+ } else if (wtype == GGML_TYPE_F32) {+ // Flat F32 HVXstruct htp_mm_hvx_vtcm_layout L;uint32_t m_chunk = 0;if (htp_mm_hvx_solve_vtcm_params(@@ -5735,6 +4071,31 @@ static void ggml_hexagon_precompute_hvx_mm_params(return;}+ kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;+ return;+ } else if (wtype == GGML_TYPE_BF16) {+ // Flat BF16 HVX+ struct htp_mm_hvx_vtcm_layout L;+ uint32_t m_chunk = 0;+ if (htp_mm_hvx_solve_vtcm_params(+ HTP_MM_KERNEL_HVX_BF16_F32_VTCM, wtype, ne10, src1_nrows, sess->n_threads,+ dst->nb[1], src0->nb[1], src1->nb[1], src2_row_size, 16, vtcm_budget,+ &L, &m_chunk)) {+ kparams->kernel_type = HTP_MM_KERNEL_HVX_BF16_F32_VTCM;+ kparams->m_chunk = (m_chunk < (uint32_t) src1_nrows) ? m_chunk : 0;+ kparams->act_row_size = hex_round_up(ne10 * 4, 128);+ kparams->vtcm_size = L.total_bytes;+ kparams->vtcm_src0_size = L.src0_bytes;+ kparams->vtcm_act_size = L.act_bytes;+ kparams->vtcm_bias_size = L.bias_bytes;+ kparams->vtcm_dst_size = L.dst_bytes;+ kparams->n_prefetch = 16;+ return;+ }++ kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;+ return;+ } else {kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;return;}@@ -5761,14 +4122,13 @@ static void ggml_hexagon_precompute_matmul_params_impl(const int ne12 = src1->ne[2];const int ne13 = src1->ne[3];- const int wtype = src0->type;- const bool is_repack = ggml_hexagon_is_repack_type((ggml_type) wtype);- const int ne00_padded = is_repack ? hex_round_up(ne00, 32) : ne00;- const int ne01_padded = is_repack ? hex_round_up(ne01, 32) : ne01;+ const bool is_tiled = ggml_hexagon_tensor_is_tiled(src0);+ const int wtype = (is_tiled && (src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_BF16)) ? GGML_TYPE_F16 : src0->type;+ const int ne00_padded = is_tiled ? hex_round_up(ne00, 32) : ne00;const int ne11_padded = hex_round_up(ne11, 32);// VTCM has to hold whole 32-row weight tiles, so size for the rounded-up N// even when the tensor itself is ragged.- const int ne01_tiled = hex_round_up(ne01_padded, 32);+ const int ne01_tiled = hex_round_up(ne01, 32);const bool is_matmul_id = (dst->op == GGML_OP_MUL_MAT_ID);const bool is_batched = (ne02 * ne03 > 1 || ne12 * ne13 > 1);@@ -5777,14 +4137,14 @@ static void ggml_hexagon_precompute_matmul_params_impl(// Check HMX eligibility and try precomputing HMX parametersbool hmx_enabled = (sess->n_hmx > 0) && (opt_mm_select >= 2);- if (hmx_enabled && ggml_hexagon_matmul_is_hmx_eligible(src0, src1, dst, ne01_padded, is_matmul_id, is_batched)) {+ if (hmx_enabled && ggml_hexagon_matmul_is_hmx_eligible(src0, src1, dst, is_matmul_id, is_batched)) {if (ggml_hexagon_precompute_hmx_mm_params(sess, src0, src1, dst, wtype, ne00_padded, ne01_tiled, ne02, ne11, ne12, ne11_padded, is_matmul_id, is_batched, src2_size, vtcm_budget, kparams)) {goto finalize;}}// Fallback to HVX parameter computation- ggml_hexagon_precompute_hvx_mm_params(sess, src0, src1, dst, wtype, ne02, ne03, ne10, ne11, ne12, ne13, is_matmul_id, src2_row_size, vtcm_budget, kparams);+ ggml_hexagon_precompute_hvx_mm_params(sess, src0, src1, dst, wtype, is_tiled, ne02, ne03, ne10, ne11, ne12, ne13, is_matmul_id, src2_row_size, vtcm_budget, kparams);finalize:kparams->div_ne12_ne1 = init_fastdiv_values(ne12 * ne11);@@ -6170,29 +4530,24 @@ static void ggml_hexagon_precompute_get_rows_params(const uint32_t ne12 = src1->ne[2];const uint32_t nr = ne10 * ne11 * ne12;- const ggml_tensor * src0_base = src0->view_src ? src0->view_src : src0;- const auto * extra = src0_base->buffer && ggml_backend_buffer_is_hexagon(src0_base->buffer) ?- (const ggml_hexagon_tensor_extra *) src0_base->extra : nullptr;- const bool tiled = src0->type == GGML_TYPE_Q4_0 || (extra && (extra->flags & GGML_HEXAGON_TENSOR_REPACK) != 0) ||- sess->needs_repack.count(src0_base) || sess->needs_repack.count(src0);+ const bool tiled = ggml_hexagon_tensor_is_tiled(src0);+ const int wtype = (tiled && (src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_BF16)) ? GGML_TYPE_F16 : src0->type;- if (src0->type == dst->type) {- kparams->kernel_type = HTP_GET_ROWS_KERNEL_SAMETYPE;- } else if (tiled) {+ if (tiled) {kparams->kernel_type = HTP_GET_ROWS_KERNEL_TILED;+ } else if (src0->type == dst->type) {+ kparams->kernel_type = HTP_GET_ROWS_KERNEL_SAMETYPE;} else {kparams->kernel_type = HTP_GET_ROWS_KERNEL_FLAT;}- const uint32_t chunks_per_row = 1;- const uint32_t chunk_size = ne00;const uint32_t total_tasks = nr;kparams->n_threads = (std::min)((uint32_t)sess->n_threads, total_tasks);struct htp_get_rows_vtcm_layout vtcm_layout = {};while (kparams->n_threads > 0) {- htp_get_rows_vtcm_layout_build(&vtcm_layout, kparams->kernel_type, src0->type, ne00, kparams->n_threads);+ htp_get_rows_vtcm_layout_build(&vtcm_layout, kparams->kernel_type, wtype, ne00, total_tasks, src1->type, kparams->n_threads);if (vtcm_layout.total_bytes <= sess->vtcm_size) {break;}@@ -6200,19 +4555,15 @@ static void ggml_hexagon_precompute_get_rows_params(}if (kparams->n_threads == 0 && total_tasks > 0) {- htp_get_rows_vtcm_layout_build(&vtcm_layout, kparams->kernel_type, src0->type, ne00, 1);+ htp_get_rows_vtcm_layout_build(&vtcm_layout, kparams->kernel_type, wtype, ne00, total_tasks, src1->type, 1);}kparams->vtcm_size = (total_tasks == 0) ? 0 : vtcm_layout.total_bytes;kparams->tasks_per_thread = kparams->n_threads > 0 ? (total_tasks + kparams->n_threads - 1) / kparams->n_threads : 0;-- kparams->chunks_per_row = chunks_per_row;- kparams->chunk_size = chunk_size;kparams->total_tasks = total_tasks;kparams->div_ne10 = ne10 > 0 ? init_fastdiv_values(ne10) : fastdiv_values{0, 0};kparams->div_ne10_ne11 = (ne10 * ne11) > 0 ? init_fastdiv_values(ne10 * ne11) : fastdiv_values{0, 0};- kparams->div_chunks_per_row = chunks_per_row > 0 ? init_fastdiv_values(chunks_per_row) : fastdiv_values{0, 0};kparams->div_ne02 = ne02 > 0 ? init_fastdiv_values(ne02) : fastdiv_values{0, 0};kparams->div_ne03 = ne03 > 0 ? init_fastdiv_values(ne03) : fastdiv_values{0, 0};}@@ -6920,49 +5271,44 @@ static void ggml_hexagon_precompute_fused_mmnx_params(const int ne12 = act->ne[2];const int ne13 = act->ne[3];- const int wtype = src0->type;- const bool is_repack = ggml_hexagon_is_repack_type((ggml_type) wtype);- const int ne00_padded = is_repack ? hex_round_up(ne00, 32) : ne00;- const int ne01_padded = is_repack ? hex_round_up(ne01, 32) : ne01;+ const bool is_tiled = ggml_hexagon_tensor_is_tiled(src0);+ const int wtype = (is_tiled && (src0->type == GGML_TYPE_F32 || src0->type == GGML_TYPE_BF16)) ? GGML_TYPE_F16 : src0->type;+ const int ne00_padded = is_tiled ? hex_round_up(ne00, 32) : ne00;const int ne11_padded = hex_round_up(ne11, 32);- const int ne01_tiled = hex_round_up(ne01_padded, 32);+ const int ne01_tiled = hex_round_up(ne01, 32);const size_t vtcm_budget = sess->vtcm_size;const bool is_batched = (ne02 * ne03 > 1 || ne12 * ne13 > 1);bool hmx_enabled = (sess->n_hmx > 0) && (opt_mm_select >= 2);- if (hmx_enabled && ggml_hexagon_matmul_is_hmx_eligible(src0, act, nullptr, ne01_padded, false, is_batched)) {+ if (hmx_enabled && ggml_hexagon_matmul_is_hmx_eligible(src0, act, nullptr, false, is_batched)) {if (ggml_hexagon_precompute_hmx_mm_params(sess, src0, act, nullptr, wtype, ne00_padded, ne01_tiled, ne02, ne11, ne12, ne11_padded, false, is_batched, 0, vtcm_budget, kparams)) {kparams->n_weights = n_weights;goto finalize;}}- if (!is_repack) {+ if (!is_tiled || ne10 % 32 != 0) {kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;return;}{const int act_nrows = ne11 * ne12 * ne13;- const size_t act_row_size = htp_mm_weight_has_offset(wtype) ? htp_mm_q8_1_tiled_row_size(ne10) : htp_mm_q8_0_tiled_row_size(ne10);+ const size_t act_row_size = htp_mm_tiled_act_row_size(wtype, ne10);const size_t src0_row_size = src0->nb[1];- uint32_t best_n_prefetch = 16;-- if (is_repack) {- const uint32_t max_prefetch = (act_nrows > HTP_MM_HMX_MIN_NROWS) ? 2 : 16;- best_n_prefetch = 2;- for (uint32_t d = max_prefetch; d >= 2; d /= 2) {- struct htp_mm_hvx_vtcm_layout L;- htp_mm_hvx_vtcm_layout_build(- &L, HTP_MM_KERNEL_HVX_QUANT_ROW, wtype, ne10, act_nrows, sess->n_threads,- 0, src0_row_size, act_row_size, 0, d, false, true- );- if (L.total_bytes <= sess->vtcm_size) {- best_n_prefetch = d;- break;- }+ const uint32_t max_prefetch = (act_nrows > HTP_MM_HMX_MIN_NROWS) ? 2 : 16;+ uint32_t best_n_prefetch = 2;+ for (uint32_t d = max_prefetch; d >= 2; d /= 2) {+ struct htp_mm_hvx_vtcm_layout L;+ htp_mm_hvx_vtcm_layout_build(+ &L, HTP_MM_KERNEL_HVX_QUANT_ROW, wtype, ne10, act_nrows, sess->n_threads,+ 0, src0_row_size, act_row_size, 0, d, false, true+ );+ if (L.total_bytes <= sess->vtcm_size) {+ best_n_prefetch = d;+ break;}}@@ -6976,15 +5322,17 @@ static void ggml_hexagon_precompute_fused_mmnx_params();if (try_tiled && L.total_bytes <= sess->vtcm_size) {- kparams->kernel_type = HTP_MM_KERNEL_HVX_QUANT_ROW;- kparams->act_row_size = act_row_size;- kparams->vtcm_src0_size = L.src0_bytes;- kparams->vtcm_act_size = L.act_bytes;- kparams->vtcm_bias_size = 0;- kparams->vtcm_dst_size = L.dst_bytes;- kparams->vtcm_size = L.total_bytes;- kparams->n_prefetch = best_n_prefetch;- kparams->n_weights = n_weights;+ kparams->kernel_type = HTP_MM_KERNEL_HVX_QUANT_ROW;+ kparams->tile_size = htp_tensor_get_tile_size(wtype);+ kparams->aligned_tile_size = htp_tensor_get_tile_padded_size(wtype);+ kparams->act_row_size = act_row_size;+ kparams->vtcm_src0_size = L.src0_bytes;+ kparams->vtcm_act_size = L.act_bytes;+ kparams->vtcm_bias_size = 0;+ kparams->vtcm_dst_size = L.dst_bytes;+ kparams->vtcm_size = L.total_bytes;+ kparams->n_prefetch = best_n_prefetch;+ kparams->n_weights = n_weights;} else {kparams->kernel_type = HTP_MM_KERNEL_UNSUPPORTED;return;@@ -7011,16 +5359,6 @@ static void ggml_hexagon_precompute_fused_mmidnx_params(kparams->n_weights = n_weights;}-static bool ggml_hexagon_tensor_is_host(const struct ggml_hexagon_session * sess, const struct ggml_tensor * t) {- return t && t->buffer && ggml_backend_buft_is_host(t->buffer->buft);- GGML_UNUSED(sess);-}--static bool ggml_hexagon_tensor_is_non_host(const struct ggml_hexagon_session * sess, const struct ggml_tensor * t) {- return t && t->buffer && !ggml_backend_buft_is_host(t->buffer->buft);- GGML_UNUSED(sess);-}-static bool ggml_hexagon_supported_mul_mat(const struct ggml_hexagon_session * sess, const struct ggml_tensor * dst) {const struct ggml_tensor * src0 = dst->src[0];const struct ggml_tensor * src1 = dst->src[1];@@ -7044,7 +5382,7 @@ static bool ggml_hexagon_supported_mul_mat(const struct ggml_hexagon_session * scase GGML_TYPE_Q6_K:case GGML_TYPE_Q3_K:case GGML_TYPE_Q2_K:- if (!ggml_is_contiguous(src0) || ggml_is_permuted(src0)) {+ if (!ggml_is_contiguous(src0) || ggml_is_permuted(src0) || ggml_is_view(src0)) {return false;}@@ -7059,10 +5397,6 @@ static bool ggml_hexagon_supported_mul_mat(const struct ggml_hexagon_session * sif (src1->ne[2] % src0->ne[2] != 0 || src1->ne[3] % src0->ne[3] != 0) {return false;}-- if (!src0->buffer) {- sess->needs_repack.insert(src0);- }break;case GGML_TYPE_F16:@@ -7092,6 +5426,18 @@ static bool ggml_hexagon_supported_mul_mat(const struct ggml_hexagon_session * s}break;+ case GGML_TYPE_BF16:+ if (src0->nb[1] < src0->nb[0]) {+ return false;+ }+ if (src1->ne[2] < src0->ne[2] || src1->ne[3] < src0->ne[3]) {+ return false;+ }+ if (src1->ne[2] % src0->ne[2] != 0 || src1->ne[3] % src0->ne[3] != 0) {+ return false;+ }+ break;+default:return false;}@@ -7131,7 +5477,7 @@ static bool ggml_hexagon_supported_mul_mat_id(const struct ggml_hexagon_sessioncase GGML_TYPE_Q6_K:case GGML_TYPE_Q3_K:case GGML_TYPE_Q2_K:- if (!ggml_is_contiguous(src0) || ggml_is_permuted(src0)) {+ if (!ggml_is_contiguous(src0) || ggml_is_permuted(src0) || ggml_is_view(src0)) {return false;}@@ -7139,9 +5485,16 @@ static bool ggml_hexagon_supported_mul_mat_id(const struct ggml_hexagon_sessionsrc0->type == GGML_TYPE_Q3_K || src0->type == GGML_TYPE_Q2_K) ? QK_K : 32)) {return false;}+ break;- if (!src0->buffer) {- sess->needs_repack.insert(src0);+ case GGML_TYPE_F16:+ case GGML_TYPE_F32:+ case GGML_TYPE_BF16:+ if (!ggml_is_contiguous(src0) || ggml_is_permuted(src0)) {+ return false;+ }+ if (src0->ne[0] % 32 != 0) {+ return false;}break;@@ -7564,40 +5917,12 @@ static bool ggml_hexagon_supported_get_rows(const struct ggml_hexagon_session *const struct ggml_tensor * src1 = op->src[1]; // indicesconst struct ggml_tensor * dst = op;- if ((src0->type == GGML_TYPE_Q4_0 || src0->type == GGML_TYPE_Q4_K || src0->type == GGML_TYPE_Q6_K) && src0->view_src) {- return false;- }-- const ggml_tensor * src0_base = src0->view_src ? src0->view_src : src0;- bool is_repacked = false;- if (src0_base->buffer && ggml_backend_buffer_is_hexagon(src0_base->buffer) && src0_base->extra) {- const auto * extra = (const ggml_hexagon_tensor_extra *) src0_base->extra;- is_repacked = (extra->flags & GGML_HEXAGON_TENSOR_REPACK) != 0;- if (is_repacked && src0->type != GGML_TYPE_Q4_0 && src0->type != GGML_TYPE_Q4_K && src0->type != GGML_TYPE_Q6_K && src0->type != GGML_TYPE_Q8_0) {- return false;- }- }- is_repacked = is_repacked || sess->needs_repack.count(src0_base) || sess->needs_repack.count(src0);-- // View offsets use the raw quantized layout and cannot address a tiled allocation.- if (src0->view_src && is_repacked) {- return false;- }-- if ((src0->type == GGML_TYPE_Q4_0 || src0->type == GGML_TYPE_Q4_K || src0->type == GGML_TYPE_Q6_K) && src0->buffer && ggml_backend_buffer_get_size(src0->buffer) != 0 && !is_repacked) {- return false;- }-- if (src0->type != dst->type && src0->ne[0] < 32) {+ if (src0->type != GGML_TYPE_F32 && src0->type != GGML_TYPE_F16 && src0->type != GGML_TYPE_BF16 &&+ src0->type != GGML_TYPE_Q4_0 && src0->type != GGML_TYPE_Q4_K && src0->type != GGML_TYPE_Q5_K && src0->type != GGML_TYPE_Q6_K && src0->type != GGML_TYPE_Q8_0 && src0->type != GGML_TYPE_I32) {return false;}- if (src0->type != GGML_TYPE_F32 && src0->type != GGML_TYPE_F16 &&- src0->type != GGML_TYPE_Q4_0 && src0->type != GGML_TYPE_Q4_K && src0->type != GGML_TYPE_Q6_K && src0->type != GGML_TYPE_Q8_0 && src0->type != GGML_TYPE_I32) {- return false;- }-- if ((src0->type == GGML_TYPE_Q4_K || src0->type == GGML_TYPE_Q6_K) && (!ggml_is_contiguous(src0) || ggml_is_permuted(src0) || src0->ne[0] % QK_K)) {+ if (!ggml_is_contiguous_rows(src0) || !ggml_is_contiguous_rows(src1) || !ggml_is_contiguous_rows(dst)) {return false;}@@ -7626,11 +5951,6 @@ static bool ggml_hexagon_supported_get_rows(const struct ggml_hexagon_session *return false;}- // Tiled quantized weights have no raw fallback. Mark only accepted tensors for repacking.- if ((src0->type == GGML_TYPE_Q4_0 || src0->type == GGML_TYPE_Q4_K || src0->type == GGML_TYPE_Q6_K) && !src0->buffer) {- sess->needs_repack.insert(src0);- }-return true;GGML_UNUSED(sess);@@ -8436,20 +6756,13 @@ static bool mm_is_hmx_eligible(const ggml_tensor * t) {if (ggml_hexagon_matmul_can_collapse(src0, src1, t)) {const ggml_tensor src1_c = ggml_hexagon_tensor_collapse_rows(src1);- const int wtype = src0->type;- const bool is_repack = ggml_hexagon_is_repack_type((ggml_type) wtype);- const int ne01_padded = is_repack ? hex_round_up(src0->ne[1], 32) : src0->ne[1];- return ggml_hexagon_matmul_is_hmx_eligible(src0, &src1_c, t, ne01_padded, false, false);+ return ggml_hexagon_matmul_is_hmx_eligible(src0, &src1_c, t, false, false);}- const int wtype = src0->type;- const bool is_repack = ggml_hexagon_is_repack_type((ggml_type) wtype);const bool is_matmul_id = (t->op == GGML_OP_MUL_MAT_ID);const bool is_batched = (src0->ne[2] * src0->ne[3] > 1 || src1->ne[2] * src1->ne[3] > 1);- const int ne01_padded = is_repack ? hex_round_up(src0->ne[1], 32) : src0->ne[1];-- return ggml_hexagon_matmul_is_hmx_eligible(src0, src1, t, ne01_padded, is_matmul_id, is_batched);+ return ggml_hexagon_matmul_is_hmx_eligible(src0, src1, t, is_matmul_id, is_batched);}static bool is_supported_mul_mat_nx_kernel(const ggml_tensor * src0, const struct htp_mm_kernel_params * kparams) {@@ -8457,8 +6770,8 @@ static bool is_supported_mul_mat_nx_kernel(const ggml_tensor * src0, const strucreturn kparams->kernel_type == HTP_MM_KERNEL_HMX_2D;}- if (!ggml_hexagon_is_repack_type(src0->type) || src0->type == GGML_TYPE_Q6_K) {- return false; // Q6_K has no fused HVX kernel+ if (!ggml_hexagon_tensor_is_tiled(src0)) {+ return false;}return kparams->kernel_type == HTP_MM_KERNEL_HVX_QUANT_ROW;@@ -8469,7 +6782,7 @@ static bool is_supported_mul_mat_id_nx_kernel(const ggml_tensor * src0, const streturn kparams->kernel_type == HTP_MM_KERNEL_HMX_2D;}- if (!ggml_hexagon_is_repack_type(src0->type)) {+ if (!ggml_hexagon_tensor_is_tiled(src0)) {return false;}@@ -8484,13 +6797,13 @@ static bool is_mergeable_mul_mat(const ggml_tensor * t) {if (src0->ne[2] != 1 || src0->ne[3] != 1) return false;if (mm_is_hmx_eligible(t)) {- return ggml_hexagon_is_hmx_weight_type(src0->type);+ return htp_is_hmx_weight_type(src0->type);}- // HVX path requires F32 activations and repacked weights (except Q6_K)+ // HVX path requires F32 activations and tiled weightsif (src1->type != GGML_TYPE_F32) return false;- return ggml_hexagon_is_repack_type(src0->type) && src0->type != GGML_TYPE_Q6_K;+ return ggml_hexagon_tensor_is_tiled(src0);}static bool is_mergeable_mul_mat_pair(const ggml_tensor * n1, const ggml_tensor * n2) {@@ -8506,6 +6819,9 @@ static bool is_mergeable_mul_mat_pair(const ggml_tensor * n1, const ggml_tensorif (n1->src[0]->type != n2->src[0]->type) {return false;}+ if (ggml_hexagon_tensor_is_tiled(n1->src[0]) != ggml_hexagon_tensor_is_tiled(n2->src[0])) {+ return false;+ }if (mm_is_hmx_eligible(n1) != mm_is_hmx_eligible(n2)) {return false;}@@ -8516,7 +6832,11 @@ static bool is_mergeable_mul_mat_id(const ggml_tensor * t) {if (t->op != GGML_OP_MUL_MAT_ID) return false;const ggml_tensor * src0 = t->src[0];- return ggml_hexagon_is_repack_type(src0->type);+ if (mm_is_hmx_eligible(t)) {+ return htp_is_hmx_weight_type(src0->type);+ }++ return ggml_hexagon_tensor_is_tiled(src0);}static bool is_mergeable_mul_mat_id_pair(const ggml_tensor * n1, const ggml_tensor * n2) {@@ -8538,6 +6858,9 @@ static bool is_mergeable_mul_mat_id_pair(const ggml_tensor * n1, const ggml_tensif (n1->src[0]->type != n2->src[0]->type) {return false;}+ if (ggml_hexagon_tensor_is_tiled(n1->src[0]) != ggml_hexagon_tensor_is_tiled(n2->src[0])) {+ return false;+ }if (mm_is_hmx_eligible(n1) != mm_is_hmx_eligible(n2)) {return false;}@@ -8588,6 +6911,26 @@ static ggml_status ggml_backend_hexagon_graph_compute(ggml_backend_t backend, gghtp_opnode node(HTP_OP_INVALID, n);node.opcode = op_remap_to_htp(n);+ if (node.opcode == HTP_OP_MUL_MAT || node.opcode == HTP_OP_MUL_MAT_ID || node.opcode == HTP_OP_GET_ROWS) {+ ggml_tensor * src0 = node.node->src[0];+ if (ggml_hexagon_needs_repack(src0)) {+ if (opt_verbose) {+ char dims_buf[64];+ char strides_buf[64];+ htp_opformat fmt;+ fmt.format_tensor_dims(dims_buf, sizeof(dims_buf), src0);+ fmt.format_tensor_strides(strides_buf, sizeof(strides_buf), src0);+ GGML_LOG_DEBUG("ggml-hex: %s rt-repack %s|%s|%s|%s|size %zu\n",+ sess->c_name(), src0->name, dims_buf, ggml_type_name(src0->type), strides_buf, ggml_nbytes(src0));+ }+ std::vector<uint8_t> raw(ggml_nbytes(src0));+ memcpy(raw.data(), src0->data, raw.size());+ htp_repack_tensor_tiled(src0, raw.data(), raw.size());+ auto * extra = (ggml_hexagon_tensor_extra *) src0->extra;+ extra->flags |= GGML_HEXAGON_TENSOR_TILED;+ }+ }+if (node.opcode == HTP_OP_MUL_MAT || node.opcode == HTP_OP_MUL_MAT_ID) {ggml_hexagon_precompute_matmul_params(sess,node.node->src[0], node.node->src[1], node.node,@@ -9378,6 +7721,9 @@ static bool ggml_hexagon_supported_roll(const struct ggml_hexagon_session * sessreturn true;}+// IMPORTANT: supports_op runs before buffer allocation and before tensor data is populated.+// Never check buffer usage, buffer properties, or repacking state (is_tiled) here.+// Support must depend only on static tensor properties (shape, type, strides).static bool ggml_backend_hexagon_device_supports_op(ggml_backend_dev_t dev, const struct ggml_tensor * op) {auto dev_ctx = static_cast<ggml_backend_hexagon_device_context *>(dev->context);auto sess = dev_ctx->session();@@ -9971,12 +8317,15 @@ static void ggml_hexagon_init(ggml_backend_reg * reg) {"please update hexagon_type to match ggml_type");static_assert((unsigned int) HTP_TYPE_Q2_K == (unsigned int) GGML_TYPE_Q2_K,"please update hexagon_type to match ggml_type");+ static_assert((unsigned int) HTP_TYPE_BF16 == (unsigned int) GGML_TYPE_BF16,+ "please update hexagon_type to match ggml_type");const char * str_verbose = getenv("GGML_HEXAGON_VERBOSE");const char * str_opbatch = getenv("GGML_HEXAGON_OPBATCH");const char * str_opqueue = getenv("GGML_HEXAGON_OPQUEUE");const char * str_oppoll = getenv("GGML_HEXAGON_OPPOLL");const char * str_opfusion = getenv("GGML_HEXAGON_OPFUSION");+ const char * str_rt_repack = getenv("GGML_HEXAGON_RT_REPACK");const char * str_opfilter = getenv("GGML_HEXAGON_OPFILTER");const char * str_profile = getenv("GGML_HEXAGON_PROFILE");const char * str_etm = getenv("GGML_HEXAGON_ETM");@@ -10034,6 +8383,7 @@ static void ggml_hexagon_init(ggml_backend_reg * reg) {opt_optrace = str_optrace ? strtoul(str_optrace, NULL, 0) : (opt_opbatch * 256);opt_oppoll = str_oppoll ? strtoul(str_oppoll, NULL, 0) : opt_oppoll;opt_opfusion = str_opfusion ? atoi(str_opfusion) : opt_opfusion;+ opt_rt_repack = str_rt_repack ? atoi(str_rt_repack) : opt_rt_repack;opt_profile = str_profile ? atoi(str_profile) : 0;opt_etm = str_etm ? atoi(str_etm) : 0;if (str_power_mode) {diff --git a/ggml/src/ggml-hexagon/htp-opnode.h b/ggml/src/ggml-hexagon/htp-opnode.hindex 8ff38545c7..d62ce1d000 100644--- a/ggml/src/ggml-hexagon/htp-opnode.h+++ b/ggml/src/ggml-hexagon/htp-opnode.h@@ -326,10 +326,12 @@ struct htp_opformat {const auto * kparams = (const struct htp_mm_kernel_params *) node.kernel_params;const char * path = "unknown";int32_t type = kparams->kernel_type;- if (type == HTP_MM_KERNEL_HMX_2D || type == HTP_MM_KERNEL_HMX_F16_BATCHED) {- path = "hmx-tiled";+ if (type == HTP_MM_KERNEL_HMX_2D || type == HTP_MM_KERNEL_HMX_BATCHED) {+ path = (kparams->tile_size > 0) ? "hmx-tiled" : "hmx-flat";} else if (type == HTP_MM_KERNEL_HVX_F16_F16_VTCM || type == HTP_MM_KERNEL_HVX_F32_F32_VTCM ||- type == HTP_MM_KERNEL_HVX_QUANT_ROW || type == HTP_MM_KERNEL_HVX_QUANT_BLOCK) {+ type == HTP_MM_KERNEL_HVX_BF16_F32_VTCM) {+ path = "hvx-flat";+ } else if (type == HTP_MM_KERNEL_HVX_QUANT_ROW || type == HTP_MM_KERNEL_HVX_QUANT_BLOCK) {path = "hvx-tiled";}snprintf(str, max_size, "%s vtcm %d", path, (int) kparams->vtcm_size);diff --git a/ggml/src/ggml-hexagon/htp-repack.cpp b/ggml/src/ggml-hexagon/htp-repack.cppnew file mode 100644index 0000000000..38beabe82a--- /dev/null+++ b/ggml/src/ggml-hexagon/htp-repack.cpp@@ -0,0 +1,1000 @@+#include "htp-repack.h"++#define GGML_COMMON_IMPL_CPP+#include "ggml-common.h"+#include "ggml-impl.h"+#include "ggml-quants.h"++#include <algorithm>+#include <cmath>+#include <cstring>+#include <initializer_list>++// --- 4-bit block-32 nibble helpers (shared by Q4_0, IQ4_NL, Q4_1, MXFP4) ---++static inline void unpack_nibbles_32(uint8_t * dst, const uint8_t * qs) {+ for (unsigned int i = 0; i < 16; ++i) {+ dst[i + 0] = qs[i] & 0x0F;+ dst[i + 16] = qs[i] >> 4;+ }+}++static inline void pack_nibbles_32(uint8_t * qs, const uint8_t * src) {+ for (unsigned int i = 0; i < 16; ++i) {+ qs[i] = src[i + 0] | (src[i + 16] << 4);+ }+}++static inline void transpose_nibbles_to_tile(uint8_t * tile_dst, const uint8_t tile_quants[32][32]) {+ for (int cp = 0; cp < 16; cp++) {+ for (int row = 0; row < 32; row++) {+ tile_dst[cp * 32 + row] = (tile_quants[row][2 * cp + 1] << 4) | tile_quants[row][2 * cp];+ }+ }+}++static inline void transpose_tile_to_nibbles(uint8_t tile_quants[32][32], const uint8_t * tile_src) {+ for (int cp = 0; cp < 16; cp++) {+ for (int row = 0; row < 32; row++) {+ const uint8_t val = tile_src[cp * 32 + row];+ tile_quants[row][2 * cp + 0] = val & 0x0F;+ tile_quants[row][2 * cp + 1] = val >> 4;+ }+ }+}++// --- Generic 32x32 tile grid walkers for block-32 and float types ---++template <typename SrcElemT, int ELEMS_PER_KTILE, typename PackTileFn>+static void repack_to_tiled_grid_32(ggml_tensor * t, const void * data, size_t offset, size_t size,+ size_t tile_size, PackTileFn && pack_tile) {+ const SrcElemT * src_matrix = (const SrcElemT *) data;+ const int64_t ne0 = t->ne[0];+ const int64_t ne1 = t->ne[1];+ const int64_t ne2 = t->ne[2];+ const int64_t ne3 = t->ne[3];+ const int n_col_tiles = hex_round_up(ne1, 32) / 32;+ const int n_k_tiles = hex_round_up(ne0, 32) / 32;+ const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;++ const size_t slice_size = (size_t) ne1 * ggml_row_size(t->type, ne0);+ const int64_t start_slice = offset / slice_size;+ int64_t end_slice = (offset + size + slice_size - 1) / slice_size;+ if (end_slice > ne2 * ne3) {+ end_slice = ne2 * ne3;+ }++ const int64_t row_stride = (ne0 / (32 / ELEMS_PER_KTILE));++ for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {+ const SrcElemT * src_slice = src_matrix + (slice_idx - start_slice) * (ne1 * row_stride);+ uint8_t * matrix_dst = (uint8_t *) t->data + slice_idx * matrix_size;++ for (int ct = 0; ct < n_col_tiles; ct++) {+ for (int kt = 0; kt < n_k_tiles; kt++) {+ uint8_t * tile_dst = matrix_dst + ((size_t) ct * n_k_tiles + kt) * tile_size;+ pack_tile(tile_dst, src_slice, ne0, ne1, row_stride, ct, kt);+ }+ }+ }+}++template <typename DstElemT, int ELEMS_PER_KTILE, typename UnpackTileFn>+static void unpack_from_tiled_grid_32(void * data, const ggml_tensor * t, size_t offset, size_t size,+ size_t tile_size, UnpackTileFn && unpack_tile) {+ uint8_t * dst_bytes = (uint8_t *) data;+ const int64_t ne0 = t->ne[0];+ const int64_t ne1 = t->ne[1];+ const int64_t ne2 = t->ne[2];+ const int64_t ne3 = t->ne[3];+ const int n_col_tiles = hex_round_up(ne1, 32) / 32;+ const int n_k_tiles = hex_round_up(ne0, 32) / 32;+ const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;++ const size_t row_size_bytes = ggml_row_size(t->type, ne0);+ const size_t slice_size = (size_t) ne1 * row_size_bytes;+ const int64_t start_slice = offset / slice_size;+ int64_t end_slice = (offset + size + slice_size - 1) / slice_size;+ if (end_slice > ne2 * ne3) {+ end_slice = ne2 * ne3;+ }++ const int64_t row_stride = (ne0 / (32 / ELEMS_PER_KTILE));++ for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {+ const size_t cur_start_byte = (std::max)(offset, (size_t) slice_idx * slice_size);+ const size_t cur_end_byte = (std::min)(offset + size, (size_t) (slice_idx + 1) * slice_size);+ const size_t slice_offset_start = cur_start_byte - (size_t) slice_idx * slice_size;+ const size_t slice_offset_end = cur_end_byte - (size_t) slice_idx * slice_size;++ const int64_t start_row = slice_offset_start / row_size_bytes;+ const int64_t end_row = (std::min)((int64_t) ((slice_offset_end + row_size_bytes - 1) / row_size_bytes), ne1);++ const int start_ct = (int) (start_row / 32);+ const int end_ct = (std::min)((int) ((end_row + 31) / 32), n_col_tiles);++ DstElemT * dst_slice = (DstElemT *) (dst_bytes + (cur_start_byte - offset));+ const uint8_t * matrix_src = (const uint8_t *) t->data + slice_idx * matrix_size;++ for (int ct = start_ct; ct < end_ct; ct++) {+ for (int kt = 0; kt < n_k_tiles; kt++) {+ const uint8_t * tile_src = matrix_src + ((size_t) ct * n_k_tiles + kt) * tile_size;+ unpack_tile(dst_slice, tile_src, ne0, start_row, end_row, row_stride, ct, kt);+ }+ }+ }+}++// --- Block-32 Quantized Types (Q4_0, IQ4_NL, Q4_1, Q8_0, MXFP4) ---++static void repack_q4_0_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {+ repack_to_tiled_grid_32<block_q4_0, 1>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q4_0,+ [](uint8_t * tile_dst, const block_q4_0 * src_slice, int64_t ne0, int64_t ne1, int64_t row_stride, int ct, int kt) {+ uint8_t tile_quants[32][32];+ ggml_half * scale_dst = (ggml_half *) (tile_dst + 512);+ for (int row = 0; row < 32; row++) {+ const int64_t r = ct * 32 + row;+ if (r < ne1 && kt < ne0 / 32) {+ const block_q4_0 & b = src_slice[r * row_stride + kt];+ unpack_nibbles_32(tile_quants[row], b.qs);+ scale_dst[row] = b.d;+ } else {+ memset(tile_quants[row], 8, 32);+ scale_dst[row] = 0;+ }+ }+ transpose_nibbles_to_tile(tile_dst, tile_quants);+ });+}++static void repack_tiled_q4_0(void * data, const ggml_tensor * t, size_t offset, size_t size) {+ unpack_from_tiled_grid_32<block_q4_0, 1>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q4_0,+ [](block_q4_0 * dst_slice, const uint8_t * tile_src, int64_t ne0, int64_t start_row, int64_t end_row, int64_t row_stride, int ct, int kt) {+ uint8_t tile_quants[32][32];+ transpose_tile_to_nibbles(tile_quants, tile_src);+ const ggml_half * scale_src = (const ggml_half *) (tile_src + 512);+ for (int row = 0; row < 32; row++) {+ const int64_t r = ct * 32 + row;+ if (r >= start_row && r < end_row && kt < ne0 / 32) {+ block_q4_0 & b = dst_slice[(r - start_row) * row_stride + kt];+ pack_nibbles_32(b.qs, tile_quants[row]);+ b.d = scale_src[row];+ }+ }+ });+}++static void repack_q4_1_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {+ repack_to_tiled_grid_32<block_q4_1, 1>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q4_1,+ [](uint8_t * tile_dst, const block_q4_1 * src_slice, int64_t ne0, int64_t ne1, int64_t row_stride, int ct, int kt) {+ uint8_t tile_quants[32][32];+ ggml_half * scale_dst = (ggml_half *) (tile_dst + 512);+ for (int row = 0; row < 32; row++) {+ const int64_t r = ct * 32 + row;+ if (r < ne1 && kt < ne0 / 32) {+ const block_q4_1 & b = src_slice[r * row_stride + kt];+ unpack_nibbles_32(tile_quants[row], b.qs);+ scale_dst[2 * row + 0] = b.d;+ scale_dst[2 * row + 1] = b.m;+ } else {+ memset(tile_quants[row], 0, 32);+ scale_dst[2 * row + 0] = 0;+ scale_dst[2 * row + 1] = 0;+ }+ }+ transpose_nibbles_to_tile(tile_dst, tile_quants);+ });+}++static void repack_tiled_q4_1(void * data, const ggml_tensor * t, size_t offset, size_t size) {+ unpack_from_tiled_grid_32<block_q4_1, 1>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q4_1,+ [](block_q4_1 * dst_slice, const uint8_t * tile_src, int64_t ne0, int64_t start_row, int64_t end_row, int64_t row_stride, int ct, int kt) {+ uint8_t tile_quants[32][32];+ transpose_tile_to_nibbles(tile_quants, tile_src);+ const ggml_half * scale_src = (const ggml_half *) (tile_src + 512);+ for (int row = 0; row < 32; row++) {+ const int64_t r = ct * 32 + row;+ if (r >= start_row && r < end_row && kt < ne0 / 32) {+ block_q4_1 & b = dst_slice[(r - start_row) * row_stride + kt];+ pack_nibbles_32(b.qs, tile_quants[row]);+ b.d = scale_src[2 * row + 0];+ b.m = scale_src[2 * row + 1];+ }+ }+ });+}++static void repack_mxfp4_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {+ repack_to_tiled_grid_32<block_mxfp4, 1>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_MXFP4,+ [](uint8_t * tile_dst, const block_mxfp4 * src_slice, int64_t ne0, int64_t ne1, int64_t row_stride, int ct, int kt) {+ uint8_t tile_quants[32][32];+ uint8_t * scale_dst = tile_dst + 512;+ for (int row = 0; row < 32; row++) {+ const int64_t r = ct * 32 + row;+ if (r < ne1 && kt < ne0 / 32) {+ const block_mxfp4 & b = src_slice[r * row_stride + kt];+ unpack_nibbles_32(tile_quants[row], b.qs);+ scale_dst[row] = b.e;+ } else {+ memset(tile_quants[row], 0, 32);+ scale_dst[row] = 0;+ }+ }+ transpose_nibbles_to_tile(tile_dst, tile_quants);+ });+}++static void repack_tiled_mxfp4(void * data, const ggml_tensor * t, size_t offset, size_t size) {+ unpack_from_tiled_grid_32<block_mxfp4, 1>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_MXFP4,+ [](block_mxfp4 * dst_slice, const uint8_t * tile_src, int64_t ne0, int64_t start_row, int64_t end_row, int64_t row_stride, int ct, int kt) {+ uint8_t tile_quants[32][32];+ transpose_tile_to_nibbles(tile_quants, tile_src);+ const uint8_t * scale_src = tile_src + 512;+ for (int row = 0; row < 32; row++) {+ const int64_t r = ct * 32 + row;+ if (r >= start_row && r < end_row && kt < ne0 / 32) {+ block_mxfp4 & b = dst_slice[(r - start_row) * row_stride + kt];+ pack_nibbles_32(b.qs, tile_quants[row]);+ b.e = scale_src[row];+ }+ }+ });+}++static void repack_q8_0_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {+ repack_to_tiled_grid_32<block_q8_0, 1>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q8_0,+ [](uint8_t * tile_dst, const block_q8_0 * src_slice, int64_t ne0, int64_t ne1, int64_t row_stride, int ct, int kt) {+ for (int cp = 0; cp < 16; cp++) {+ const int col0 = cp * 2;+ const int col1 = col0 + 1;+ for (int row = 0; row < 32; row++) {+ const int64_t r = ct * 32 + row;+ const block_q8_0 * b = (r < ne1 && kt < ne0 / 32) ? &src_slice[r * row_stride + kt] : nullptr;+ tile_dst[cp * 64 + 2 * row + 0] = b ? b->qs[col0] : 0;+ tile_dst[cp * 64 + 2 * row + 1] = b ? b->qs[col1] : 0;+ }+ }+ ggml_half * scale_dst = (ggml_half *) (tile_dst + 1024);+ for (int row = 0; row < 32; row++) {+ const int64_t r = ct * 32 + row;+ scale_dst[row] = (r < ne1 && kt < ne0 / 32) ? src_slice[r * row_stride + kt].d : 0;+ }+ });+}++static void repack_tiled_q8_0(void * data, const ggml_tensor * t, size_t offset, size_t size) {+ unpack_from_tiled_grid_32<block_q8_0, 1>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q8_0,+ [](block_q8_0 * dst_slice, const uint8_t * tile_src, int64_t ne0, int64_t start_row, int64_t end_row, int64_t row_stride, int ct, int kt) {+ for (int cp = 0; cp < 16; cp++) {+ const int col0 = cp * 2;+ const int col1 = col0 + 1;+ for (int row = 0; row < 32; row++) {+ const int64_t r = ct * 32 + row;+ if (r >= start_row && r < end_row && kt < ne0 / 32) {+ block_q8_0 & b = dst_slice[(r - start_row) * row_stride + kt];+ b.qs[col0] = tile_src[cp * 64 + 2 * row + 0];+ b.qs[col1] = tile_src[cp * 64 + 2 * row + 1];+ }+ }+ }+ const ggml_half * scale_src = (const ggml_half *) (tile_src + 1024);+ for (int row = 0; row < 32; row++) {+ const int64_t r = ct * 32 + row;+ if (r >= start_row && r < end_row && kt < ne0 / 32) {+ dst_slice[(r - start_row) * row_stride + kt].d = scale_src[row];+ }+ }+ });+}++// --- Float Types (F16, F32, BF16 -> f16_tiled HMX layout) ---++template <typename SrcT, typename ToFp16Fn>+static void repack_float_to_f16_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size, ToFp16Fn && to_fp16) {+ repack_to_tiled_grid_32<SrcT, 32>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_F16,+ [&](uint8_t * tile_bytes, const SrcT * src_slice, int64_t ne0, int64_t ne1, int64_t row_stride, int ct, int kt) {+ uint16_t * tile_dst = (uint16_t *) tile_bytes;+ for (int cp = 0; cp < 16; cp++) {+ const int64_t c0 = kt * 32 + cp * 2;+ const int64_t c1 = c0 + 1;+ for (int row = 0; row < 32; row++) {+ const int64_t r = ct * 32 + row;+ const SrcT * r_ptr = (r < ne1) ? &src_slice[r * row_stride] : nullptr;+ tile_dst[cp * 64 + row * 2 + 0] = (r_ptr && c0 < ne0) ? to_fp16(r_ptr[c0]) : 0;+ tile_dst[cp * 64 + row * 2 + 1] = (r_ptr && c1 < ne0) ? to_fp16(r_ptr[c1]) : 0;+ }+ }+ });+}++template <typename DstT, typename FromFp16Fn>+static void unpack_f16_tiled_to_float(void * data, const ggml_tensor * t, size_t offset, size_t size, FromFp16Fn && from_fp16) {+ unpack_from_tiled_grid_32<DstT, 32>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_F16,+ [&](DstT * dst_slice, const uint8_t * tile_bytes, int64_t ne0, int64_t start_row, int64_t end_row, int64_t row_stride, int ct, int kt) {+ const uint16_t * tile_src = (const uint16_t *) tile_bytes;+ for (int cp = 0; cp < 16; cp++) {+ const int64_t c0 = kt * 32 + cp * 2;+ const int64_t c1 = c0 + 1;+ for (int row = 0; row < 32; row++) {+ const int64_t r = ct * 32 + row;+ if (r >= start_row && r < end_row) {+ DstT * r_ptr = dst_slice + (r - start_row) * row_stride;+ if (c0 < ne0) r_ptr[c0] = from_fp16(tile_src[cp * 64 + row * 2 + 0]);+ if (c1 < ne0) r_ptr[c1] = from_fp16(tile_src[cp * 64 + row * 2 + 1]);+ }+ }+ }+ });+}++// --- Generic K-Quant (super-block 256) grid walkers ---++template <typename BlockT, typename PackKtFn>+static void repack_kquant_to_tiled_grid(ggml_tensor * t, const void * data, size_t offset, size_t size,+ size_t tile_size, PackKtFn && pack_kt) {+ GGML_ASSERT(offset == 0);+ GGML_UNUSED(size);++ const BlockT * src_matrix = (const BlockT *) data;+ const int64_t ne0 = t->ne[0];+ const int64_t ne1 = t->ne[1];+ const int64_t ne2 = t->ne[2];+ const int64_t ne3 = t->ne[3];+ GGML_ASSERT(ne0 % QK_K == 0);++ const int n_col_tiles = hex_round_up(ne1, 32) / 32;+ const int n_k_tiles = hex_round_up(ne0, 32) / 32;+ const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;+ const int64_t sb_per_row = ne0 / QK_K;++ for (int64_t slice_idx = 0; slice_idx < ne2 * ne3; slice_idx++) {+ const BlockT * src_slice = src_matrix + slice_idx * (ne1 * sb_per_row);+ uint8_t * matrix_dst = (uint8_t *) t->data + slice_idx * matrix_size;++ // Padding rows and OR-ed bit planes require zeroed tiles+ memset(matrix_dst, 0, matrix_size);++ for (int64_t r = 0; r < ne1; r++) {+ const int ct = (int) (r / 32);+ const int row = (int) (r % 32);+ const BlockT * src_row = src_slice + r * sb_per_row;++ for (int kt = 0; kt < n_k_tiles; kt++) {+ uint8_t * tile = matrix_dst + ((size_t) ct * n_k_tiles + kt) * tile_size;+ pack_kt(tile, &src_row[kt / 8], row, kt % 8);+ }+ }+ }+}++template <typename BlockT, typename UnpackSbFn>+static void unpack_kquant_from_tiled_grid(void * data, const ggml_tensor * t, size_t offset, size_t size,+ size_t tile_size, UnpackSbFn && unpack_sb) {+ uint8_t * dst_bytes = (uint8_t *) data;+ const int64_t ne0 = t->ne[0];+ const int64_t ne1 = t->ne[1];+ const int64_t ne2 = t->ne[2];+ const int64_t ne3 = t->ne[3];+ GGML_ASSERT(ne0 % QK_K == 0);++ const int n_col_tiles = hex_round_up(ne1, 32) / 32;+ const int n_k_tiles = hex_round_up(ne0, 32) / 32;+ const size_t matrix_size = (size_t) n_col_tiles * n_k_tiles * tile_size;+ const int64_t sb_per_row = ne0 / QK_K;++ const size_t row_size_bytes = ggml_row_size(t->type, ne0);+ const size_t slice_size = (size_t) ne1 * row_size_bytes;+ const int64_t start_slice = offset / slice_size;+ int64_t end_slice = (offset + size + slice_size - 1) / slice_size;+ if (end_slice > ne2 * ne3) {+ end_slice = ne2 * ne3;+ }++ for (int64_t slice_idx = start_slice; slice_idx < end_slice; slice_idx++) {+ const size_t cur_start_byte = (std::max)(offset, (size_t) slice_idx * slice_size);+ const size_t cur_end_byte = (std::min)(offset + size, (size_t) (slice_idx + 1) * slice_size);+ const size_t slice_offset_start = cur_start_byte - (size_t) slice_idx * slice_size;+ const size_t slice_offset_end = cur_end_byte - (size_t) slice_idx * slice_size;++ const int64_t start_row = slice_offset_start / row_size_bytes;+ const int64_t end_row = (std::min)((int64_t) ((slice_offset_end + row_size_bytes - 1) / row_size_bytes), ne1);++ BlockT * dst_slice = (BlockT *) (dst_bytes + (cur_start_byte - offset));+ const uint8_t * matrix_src = (const uint8_t *) t->data + slice_idx * matrix_size;++ for (int64_t r = start_row; r < end_row; r++) {+ const int ct = (int) (r / 32);+ const int row = (int) (r % 32);+ BlockT * dst_row = dst_slice + (r - start_row) * sb_per_row;++ for (int64_t sb = 0; sb < sb_per_row; sb++) {+ BlockT * b = &dst_row[sb];+ memset(b, 0, sizeof(BlockT));+ const uint8_t * sb_tiles = matrix_src + ((size_t) ct * n_k_tiles + sb * 8) * tile_size;+ unpack_sb(b, sb_tiles, tile_size, row);+ }+ }+ }+}++// --- K-Quant Helpers & Per-Type Implementations (Q2_K, Q3_K, Q4_K, Q5_K, Q6_K) ---++// unsigned 6-bit value (0..63) of element e of a Q6_K block, same bit layout as dequantize_row_q6_K+static inline uint8_t q6_K_get_quant(const block_q6_K * b, int e) {+ const int c = e / 128;+ const int w = e % 128;+ const int g = w / 32;+ const int l = w % 32;+ const uint8_t * ql = b->ql + c * 64;+ const uint8_t * qh = b->qh + c * 32;+ uint8_t lo, hi;+ switch (g) {+ case 0: lo = ql[l] & 0xF; hi = (qh[l] >> 0) & 3; break;+ case 1: lo = ql[l + 32] & 0xF; hi = (qh[l] >> 2) & 3; break;+ case 2: lo = ql[l] >> 4; hi = (qh[l] >> 4) & 3; break;+ default: lo = ql[l + 32] >> 4; hi = (qh[l] >> 6) & 3; break;+ }+ return (uint8_t) (lo | (hi << 4));+}++static void repack_q6_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {+ repack_kquant_to_tiled_grid<block_q6_K>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q6_K,+ [](uint8_t * tile, const block_q6_K * b, int row, int kt_local) {+ const float d = GGML_FP16_TO_FP32(b->d);+ uint8_t * lo_pl = tile;+ uint8_t * hi_pl = tile + 512;+ ggml_half * sc_pl = (ggml_half *) (tile + 768);++ for (int lk = 0; lk < 32; lk++) {+ const uint8_t q6 = q6_K_get_quant(b, kt_local * 32 + lk);+ const int g = lk >> 2;+ const int pos = row * 4 + (lk & 3);+ lo_pl[(g >> 1) * 128 + pos] |= (uint8_t) ((q6 & 0xF) << ((g & 1) * 4));+ hi_pl[(g >> 2) * 128 + pos] |= (uint8_t) ((q6 >> 4) << ((g & 3) * 2));+ }+ for (int sub = 0; sub < 2; sub++) {+ sc_pl[sub * 32 + row] = GGML_FP32_TO_FP16(d * (float) b->scales[kt_local * 2 + sub]);+ }+ });+}++static void repack_tiled_q6_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {+ unpack_kquant_from_tiled_grid<block_q6_K>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q6_K,+ [](block_q6_K * b, const uint8_t * sb_tiles, size_t tile_size, int row) {+ float sub_scales[16];+ for (int kt_local = 0; kt_local < 8; kt_local++) {+ const uint8_t * tile = sb_tiles + kt_local * tile_size;+ const uint8_t * lo_pl = tile;+ const uint8_t * hi_pl = tile + 512;+ const ggml_half * sc_pl = (const ggml_half *) (tile + 768);++ const int c = kt_local / 4;+ const int g = kt_local % 4;+ uint8_t * ql = b->ql + c * 64;+ uint8_t * qh = b->qh + c * 32;++ for (int lk = 0; lk < 32; lk++) {+ const int g_tile = lk >> 2;+ const int pos = row * 4 + (lk & 3);+ const uint8_t lo = (lo_pl[(g_tile >> 1) * 128 + pos] >> ((g_tile & 1) * 4)) & 0xF;+ const uint8_t hi = (hi_pl[(g_tile >> 2) * 128 + pos] >> ((g_tile & 3) * 2)) & 3;++ switch (g) {+ case 0: ql[lk] |= lo; qh[lk] |= (hi << 0); break;+ case 1: ql[lk + 32] |= lo; qh[lk] |= (hi << 2); break;+ case 2: ql[lk] |= (lo << 4); qh[lk] |= (hi << 4); break;+ default: ql[lk + 32] |= (lo << 4); qh[lk] |= (hi << 6); break;+ }+ }++ for (int sub = 0; sub < 2; sub++) {+ sub_scales[kt_local * 2 + sub] = GGML_FP16_TO_FP32(sc_pl[sub * 32 + row]);+ }+ }++ float max_abs_scale = 0.0f;+ for (int s = 0; s < 16; s++) {+ max_abs_scale = (std::max)(max_abs_scale, fabsf(sub_scales[s]));+ }++ if (max_abs_scale == 0.0f) {+ b->d = GGML_FP32_TO_FP16(0.0f);+ } else {+ b->d = GGML_FP32_TO_FP16(max_abs_scale / 127.0f);+ const float d_actual = GGML_FP16_TO_FP32(b->d);+ const float inv_d = (d_actual != 0.0f) ? (1.0f / d_actual) : 0.0f;+ for (int s = 0; s < 16; s++) {+ const int sc = (int) roundf(sub_scales[s] * inv_d);+ b->scales[s] = (int8_t) (std::max)(-128, (std::min)(127, sc));+ }+ }+ });+}++// low 2 bits (0..3) of element e of a Q2_K or Q3_K block, same bit layout as dequantize_row_q2_K / q3_K+static inline uint8_t q2_3_K_get_low2(const uint8_t * qs, int e) {+ const int c = e / 128;+ const int j = (e % 128) / 32;+ const int l = e % 32;+ return (qs[c * 32 + l] >> (2 * j)) & 3;+}++// hmask bit of element e of a Q3_K block, same bit layout as dequantize_row_q3_K+static inline bool q3_K_get_hbit(const block_q3_K * b, int e) {+ const int c = e / 128;+ const int j = (e % 128) / 32;+ const int l = e % 32;+ return (b->hmask[l] >> (c * 4 + j)) & 1;+}++// signed 6-bit scale j (-32..31) of a Q3_K block, same packing as quantize_row_q3_K_ref+static inline int q3_K_get_scale(const uint8_t * scales, int j) {+ const int lo = (j < 8) ? (scales[j] & 0xF) : (scales[j - 8] >> 4);+ const int hi = (scales[8 + j % 4] >> (2 * (j / 4))) & 3;+ return (lo | (hi << 4)) - 32;+}++// read-back: find fp16 d and l[j] in [lmin, lmax] with fp16(d * l[j]) == prod[j] for all j, false if none+static bool hexagon_recover_k_scales(const ggml_half * prod, int n, int lmin, int lmax, ggml_half * d_out, int * l_out) {+ int jmax = 0;+ for (int j = 1; j < n; j++) {+ if (fabsf(GGML_FP16_TO_FP32(prod[j])) > fabsf(GGML_FP16_TO_FP32(prod[jmax]))) {+ jmax = j;+ }+ }+ const float pmax = GGML_FP16_TO_FP32(prod[jmax]);+ if (pmax == 0.0f) {+ *d_out = GGML_FP32_TO_FP16(0.0f);+ for (int j = 0; j < n; j++) {+ l_out[j] = 0;+ }+ return true;+ }++ // the quantizers put the largest scale at or near the range end, so try large |l| first+ const int lext = (std::max)(-lmin, lmax);+ for (int a = lext; a >= 1; a--) {+ for (int sign : { 1, -1 }) {+ const int lj = sign * a;+ if (lj < lmin || lj > lmax) {+ continue;+ }+ const ggml_half d0 = GGML_FP32_TO_FP16(pmax / (float) lj);+ for (int ulp : { 0, -1, 1 }) {+ ggml_half d = d0;+ uint16_t bits;+ memcpy(&bits, &d, sizeof(bits));+ bits = (uint16_t) (bits + ulp);+ memcpy(&d, &bits, sizeof(bits));++ const float df = GGML_FP16_TO_FP32(d);+ if (!std::isfinite(df) || df == 0.0f) {+ continue;+ }+ bool ok = true;+ for (int j = 0; j < n && ok; j++) {+ const int l = (int) roundf(GGML_FP16_TO_FP32(prod[j]) / df);+ const ggml_half p = GGML_FP32_TO_FP16(df * (float) l);+ ok = l >= lmin && l <= lmax && memcmp(&p, &prod[j], sizeof(p)) == 0;+ l_out[j] = l;+ }+ if (ok) {+ *d_out = d;+ return true;+ }+ }+ }+ }+ return false;+}++static void repack_q3_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {+ repack_kquant_to_tiled_grid<block_q3_K>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q3_K,+ [](uint8_t * tile, const block_q3_K * b, int row, int kt_local) {+ const float d = GGML_FP16_TO_FP32(b->d);+ uint8_t * lo_pl = tile;+ uint8_t * neg_pl = tile + 256;+ ggml_half * sc_pl = (ggml_half *) (tile + 384);++ for (int lk = 0; lk < 32; lk++) {+ const int e = kt_local * 32 + lk;+ const int g = lk >> 2;+ const int pos = row * 4 + (lk & 3);+ lo_pl[(g >> 2) * 128 + pos] |= (uint8_t) (q2_3_K_get_low2(b->qs, e) << ((g & 3) * 2));+ if (!q3_K_get_hbit(b, e)) {+ neg_pl[pos] |= (uint8_t) (1 << g);+ }+ }+ for (int sub = 0; sub < 2; sub++) {+ sc_pl[sub * 32 + row] = GGML_FP32_TO_FP16(d * (float) q3_K_get_scale(b->scales, kt_local * 2 + sub));+ }+ });+}++static void repack_tiled_q3_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {+ unpack_kquant_from_tiled_grid<block_q3_K>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q3_K,+ [](block_q3_K * b, const uint8_t * sb_tiles, size_t tile_size, int row) {+ ggml_half sub_scales[16];+ for (int kt_local = 0; kt_local < 8; kt_local++) {+ const uint8_t * tile = sb_tiles + kt_local * tile_size;+ const uint8_t * lo_pl = tile;+ const uint8_t * neg_pl = tile + 256;+ const ggml_half * sc_pl = (const ggml_half *) (tile + 384);++ for (int lk = 0; lk < 32; lk++) {+ const int e = kt_local * 32 + lk;+ const int g = lk >> 2;+ const int pos = row * 4 + (lk & 3);+ const uint8_t lo = (lo_pl[(g >> 2) * 128 + pos] >> ((g & 3) * 2)) & 3;++ const int c = e / 128;+ const int j = (e % 128) / 32;+ const int l = e % 32;+ b->qs[c * 32 + l] |= (uint8_t) (lo << (2 * j));+ if (!((neg_pl[pos] >> g) & 1)) {+ b->hmask[l] |= (uint8_t) (1 << (c * 4 + j));+ }+ }++ for (int sub = 0; sub < 2; sub++) {+ sub_scales[kt_local * 2 + sub] = sc_pl[sub * 32 + row];+ }+ }++ int ls[16];+ if (!hexagon_recover_k_scales(sub_scales, 16, -32, 31, &b->d, ls)) {+ float max_scale = 0.0f;+ for (int s = 0; s < 16; s++) {+ if (fabsf(GGML_FP16_TO_FP32(sub_scales[s])) > fabsf(max_scale)) {+ max_scale = GGML_FP16_TO_FP32(sub_scales[s]);+ }+ }+ b->d = GGML_FP32_TO_FP16(-max_scale / 32.0f);+ const float d_actual = GGML_FP16_TO_FP32(b->d);+ const float inv_d = (d_actual != 0.0f) ? (1.0f / d_actual) : 0.0f;+ for (int s = 0; s < 16; s++) {+ ls[s] = (std::max)(-32, (std::min)(31, (int) roundf(GGML_FP16_TO_FP32(sub_scales[s]) * inv_d)));+ }+ }++ for (int s = 0; s < 16; s++) {+ const int l = ls[s] + 32;+ if (s < 8) {+ b->scales[s] = l & 0xF;+ } else {+ b->scales[s - 8] |= (uint8_t) ((l & 0xF) << 4);+ }+ b->scales[s % 4 + 8] |= (uint8_t) ((l >> 4) << (2 * (s / 4)));+ }+ });+}++static void repack_q2_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {+ repack_kquant_to_tiled_grid<block_q2_K>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q2_K,+ [](uint8_t * tile, const block_q2_K * b, int row, int kt_local) {+ uint8_t * lo_pl = tile;+ uint8_t * sc_pl = tile + 256;+ ggml_half * dd_pl = (ggml_half *) (tile + 320);++ for (int lk = 0; lk < 32; lk++) {+ const int g = lk >> 2;+ const int pos = row * 4 + (lk & 3);+ lo_pl[(g >> 2) * 128 + pos] |= (uint8_t) (q2_3_K_get_low2(b->qs, kt_local * 32 + lk) << ((g & 3) * 2));+ }+ for (int sub = 0; sub < 2; sub++) {+ sc_pl[sub * 32 + row] = b->scales[kt_local * 2 + sub];+ }+ if (row / 8 == kt_local % 4) {+ dd_pl[row % 8] = kt_local < 4 ? b->d : (ggml_half) (b->dmin ^ 0x8000);+ }+ });+}++static void repack_tiled_q2_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {+ unpack_kquant_from_tiled_grid<block_q2_K>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q2_K,+ [](block_q2_K * b, const uint8_t * sb_tiles, size_t tile_size, int row) {+ for (int kt_local = 0; kt_local < 8; kt_local++) {+ const uint8_t * tile = sb_tiles + kt_local * tile_size;+ const uint8_t * lo_pl = tile;+ const uint8_t * sc_pl = tile + 256;+ const ggml_half * dd_pl = (const ggml_half *) (tile + 320);++ for (int lk = 0; lk < 32; lk++) {+ const int e = kt_local * 32 + lk;+ const int g = lk >> 2;+ const int pos = row * 4 + (lk & 3);+ const uint8_t lo = (lo_pl[(g >> 2) * 128 + pos] >> ((g & 3) * 2)) & 3;+ b->qs[(e / 128) * 32 + e % 32] |= (uint8_t) (lo << (2 * ((e % 128) / 32)));+ }++ for (int sub = 0; sub < 2; sub++) {+ b->scales[kt_local * 2 + sub] = sc_pl[sub * 32 + row];+ }+ if (row / 8 == kt_local % 4) {+ if (kt_local < 4) {+ b->d = dd_pl[row % 8];+ } else {+ b->dmin = (ggml_half) (dd_pl[row % 8] ^ 0x8000);+ }+ }+ }+ });+}++static inline void get_scale_min_k4(int j, const uint8_t * q, uint8_t * d, uint8_t * m) {+ if (j < 4) {+ *d = q[j] & 63;+ *m = q[j + 4] & 63;+ } else {+ *d = (q[j + 4] & 0xF) | ((q[j - 4] >> 6) << 4);+ *m = (q[j + 4] >> 4) | ((q[j - 0] >> 6) << 4);+ }+}++static inline void pack_k4_scales_mins(uint8_t * scales, ggml_half * d, ggml_half * dmin,+ const float * sub_scales, const float * sub_mins) {+ float max_scale = 0.0f;+ float max_min = 0.0f;+ for (int j = 0; j < 8; j++) {+ if (sub_scales[j] > max_scale) max_scale = sub_scales[j];+ if (sub_mins[j] > max_min) max_min = sub_mins[j];+ }++ float inv_scale = 0.0f;+ if (max_scale > 0.0f) {+ *d = GGML_FP32_TO_FP16(max_scale / 63.0f);+ const float d_actual = GGML_FP16_TO_FP32(*d);+ inv_scale = (d_actual > 0.0f) ? (1.0f / d_actual) : 0.0f;+ } else {+ *d = GGML_FP32_TO_FP16(0.0f);+ }++ float inv_min = 0.0f;+ if (max_min > 0.0f) {+ *dmin = GGML_FP32_TO_FP16(max_min / 63.0f);+ const float dmin_actual = GGML_FP16_TO_FP32(*dmin);+ inv_min = (dmin_actual > 0.0f) ? (1.0f / dmin_actual) : 0.0f;+ } else {+ *dmin = GGML_FP32_TO_FP16(0.0f);+ }++ for (int j = 0; j < 8; j++) {+ uint8_t ls = (std::min)((uint8_t) 63, (uint8_t) roundf(inv_scale * sub_scales[j]));+ uint8_t lm = (std::min)((uint8_t) 63, (uint8_t) roundf(inv_min * sub_mins[j]));+ if (j < 4) {+ scales[j] = ls;+ scales[j + 4] = lm;+ } else {+ scales[j + 4] = (ls & 0xF) | ((lm & 0xF) << 4);+ scales[j - 4] |= ((ls >> 4) << 6);+ scales[j - 0] |= ((lm >> 4) << 6);+ }+ }+}++static void repack_q4_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {+ repack_kquant_to_tiled_grid<block_q4_K>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q4_K,+ [](uint8_t * tile_dst, const block_q4_K * b, int row, int kt_local) {+ const float d = GGML_FP16_TO_FP32(b->d);+ const float dmin = GGML_FP16_TO_FP32(b->dmin);++ uint8_t sc, m;+ get_scale_min_k4(kt_local, b->scales, &sc, &m);++ const uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;+ const int shift = (kt_local & 1) ? 4 : 0;++ for (int cp = 0; cp < 16; cp++) {+ const uint8_t q0 = (qs_sub[2 * cp + 0] >> shift) & 0x0F;+ const uint8_t q1 = (qs_sub[2 * cp + 1] >> shift) & 0x0F;+ tile_dst[cp * 32 + row] = (uint8_t) ((q1 << 4) | q0);+ }++ ggml_half * scale_dst = (ggml_half *) (tile_dst + 512);+ scale_dst[2 * row + 0] = GGML_FP32_TO_FP16(d * (float) sc);+ scale_dst[2 * row + 1] = GGML_FP32_TO_FP16(-dmin * (float) m);+ });+}++static void repack_tiled_q4_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {+ unpack_kquant_from_tiled_grid<block_q4_K>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q4_K,+ [](block_q4_K * b, const uint8_t * sb_tiles, size_t tile_size, int row) {+ float sub_scales[8];+ float sub_mins[8];++ for (int kt_local = 0; kt_local < 8; kt_local++) {+ const uint8_t * tile_src = sb_tiles + kt_local * tile_size;+ const ggml_half * scale_src = (const ggml_half *) (tile_src + 512);++ uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;+ const int shift = (kt_local & 1) ? 4 : 0;++ for (int cp = 0; cp < 16; cp++) {+ const uint8_t val = tile_src[cp * 32 + row];+ qs_sub[2 * cp + 0] |= (uint8_t) ((val & 0x0F) << shift);+ qs_sub[2 * cp + 1] |= (uint8_t) ((val >> 4) << shift);+ }++ const float D = GGML_FP16_TO_FP32(scale_src[2 * row + 0]);+ const float M = GGML_FP16_TO_FP32(scale_src[2 * row + 1]);+ sub_scales[kt_local] = (D > 0.0f) ? D : 0.0f;+ sub_mins[kt_local] = (-M > 0.0f) ? -M : 0.0f;+ }++ pack_k4_scales_mins(b->scales, &b->d, &b->dmin, sub_scales, sub_mins);+ });+}++static void repack_q5_K_tiled(ggml_tensor * t, const void * data, size_t offset, size_t size) {+ repack_kquant_to_tiled_grid<block_q5_K>(t, data, offset, size, HTP_TENSOR_TILE_SIZE_Q5_K,+ [](uint8_t * tile_dst, const block_q5_K * b, int row, int kt_local) {+ const float d = GGML_FP16_TO_FP32(b->d);+ const float dmin = GGML_FP16_TO_FP32(b->dmin);+ uint8_t * plane = tile_dst + 640;++ uint8_t sc, m;+ get_scale_min_k4(kt_local, b->scales, &sc, &m);++ const uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;+ const int shift = (kt_local & 1) ? 4 : 0;+ const uint8_t hbit = (uint8_t) (1 << kt_local);++ for (int cp = 0; cp < 16; cp++) {+ const uint8_t q0 = (qs_sub[2 * cp + 0] >> shift) & 0x0F;+ const uint8_t q1 = (qs_sub[2 * cp + 1] >> shift) & 0x0F;+ tile_dst[cp * 32 + row] = (uint8_t) ((q1 << 4) | q0);++ const int i = cp / 4;+ const int lane = (cp % 4) * 32 + row;+ if (b->qh[2 * cp + 0] & hbit) {+ plane[lane] |= (uint8_t) (1 << (2 * i));+ }+ if (b->qh[2 * cp + 1] & hbit) {+ plane[lane] |= (uint8_t) (1 << (2 * i + 1));+ }+ }++ ggml_half * scale_dst = (ggml_half *) (tile_dst + 512);+ scale_dst[2 * row + 0] = GGML_FP32_TO_FP16(d * (float) sc);+ scale_dst[2 * row + 1] = GGML_FP32_TO_FP16(-dmin * (float) m);+ });+}++static void repack_tiled_q5_K(void * data, const ggml_tensor * t, size_t offset, size_t size) {+ unpack_kquant_from_tiled_grid<block_q5_K>(data, t, offset, size, HTP_TENSOR_TILE_SIZE_Q5_K,+ [](block_q5_K * b, const uint8_t * sb_tiles, size_t tile_size, int row) {+ float sub_scales[8];+ float sub_mins[8];++ for (int kt_local = 0; kt_local < 8; kt_local++) {+ const uint8_t * tile_src = sb_tiles + kt_local * tile_size;+ const uint8_t * plane = tile_src + 640;+ const ggml_half * scale_src = (const ggml_half *) (tile_src + 512);++ uint8_t * qs_sub = b->qs + (kt_local / 2) * 32;+ const int shift = (kt_local & 1) ? 4 : 0;+ const uint8_t hbit = (uint8_t) (1 << kt_local);++ for (int cp = 0; cp < 16; cp++) {+ const uint8_t val = tile_src[cp * 32 + row];+ qs_sub[2 * cp + 0] |= (uint8_t) ((val & 0x0F) << shift);+ qs_sub[2 * cp + 1] |= (uint8_t) ((val >> 4) << shift);++ const int i = cp / 4;+ const int lane = (cp % 4) * 32 + row;+ if (plane[lane] & (1 << (2 * i))) {+ b->qh[2 * cp + 0] |= hbit;+ }+ if (plane[lane] & (1 << (2 * i + 1))) {+ b->qh[2 * cp + 1] |= hbit;+ }+ }++ const float D = GGML_FP16_TO_FP32(scale_src[2 * row + 0]);+ const float M = GGML_FP16_TO_FP32(scale_src[2 * row + 1]);+ sub_scales[kt_local] = (D > 0.0f) ? D : 0.0f;+ sub_mins[kt_local] = (-M > 0.0f) ? -M : 0.0f;+ }++ pack_k4_scales_mins(b->scales, &b->d, &b->dmin, sub_scales, sub_mins);+ });+}++// --- Top-level Repack / Unpack Dispatchers ---++void htp_repack_tensor_tiled(ggml_tensor * tensor, const void * data, size_t size) {+ switch (tensor->type) {+ case GGML_TYPE_Q4_0:+ case GGML_TYPE_IQ4_NL:+ repack_q4_0_tiled(tensor, data, 0, size);+ break;+ case GGML_TYPE_Q4_1:+ repack_q4_1_tiled(tensor, data, 0, size);+ break;+ case GGML_TYPE_Q8_0:+ repack_q8_0_tiled(tensor, data, 0, size);+ break;+ case GGML_TYPE_MXFP4:+ repack_mxfp4_tiled(tensor, data, 0, size);+ break;+ case GGML_TYPE_Q2_K:+ repack_q2_K_tiled(tensor, data, 0, size);+ break;+ case GGML_TYPE_Q3_K:+ repack_q3_K_tiled(tensor, data, 0, size);+ break;+ case GGML_TYPE_Q4_K:+ repack_q4_K_tiled(tensor, data, 0, size);+ break;+ case GGML_TYPE_Q5_K:+ repack_q5_K_tiled(tensor, data, 0, size);+ break;+ case GGML_TYPE_Q6_K:+ repack_q6_K_tiled(tensor, data, 0, size);+ break;+ case GGML_TYPE_F16:+ repack_float_to_f16_tiled<uint16_t>(tensor, data, 0, size,+ [](uint16_t v) { return v; });+ break;+ case GGML_TYPE_F32:+ repack_float_to_f16_tiled<float>(tensor, data, 0, size,+ [](float v) { return (uint16_t) ggml_fp32_to_fp16(v); });+ break;+ case GGML_TYPE_BF16:+ repack_float_to_f16_tiled<ggml_bf16_t>(tensor, data, 0, size,+ [](ggml_bf16_t v) { return (uint16_t) ggml_fp32_to_fp16(ggml_bf16_to_fp32(v)); });+ break;+ default:+ break;+ }+}++void htp_unpack_tensor_tiled(void * data, const ggml_tensor * tensor, size_t offset, size_t size) {+ GGML_ASSERT(offset + size <= ggml_nbytes(tensor));++ switch (tensor->type) {+ case GGML_TYPE_Q4_0:+ case GGML_TYPE_IQ4_NL:+ repack_tiled_q4_0(data, tensor, offset, size);+ break;+ case GGML_TYPE_Q4_1:+ repack_tiled_q4_1(data, tensor, offset, size);+ break;+ case GGML_TYPE_Q8_0:+ repack_tiled_q8_0(data, tensor, offset, size);+ break;+ case GGML_TYPE_MXFP4:+ repack_tiled_mxfp4(data, tensor, offset, size);+ break;+ case GGML_TYPE_Q2_K:+ repack_tiled_q2_K(data, tensor, offset, size);+ break;+ case GGML_TYPE_Q3_K:+ repack_tiled_q3_K(data, tensor, offset, size);+ break;+ case GGML_TYPE_Q4_K:+ repack_tiled_q4_K(data, tensor, offset, size);+ break;+ case GGML_TYPE_Q5_K:+ repack_tiled_q5_K(data, tensor, offset, size);+ break;+ case GGML_TYPE_Q6_K:+ repack_tiled_q6_K(data, tensor, offset, size);+ break;+ case GGML_TYPE_F16:+ unpack_f16_tiled_to_float<uint16_t>(data, tensor, offset, size,+ [](uint16_t v) { return v; });+ break;+ case GGML_TYPE_F32:+ unpack_f16_tiled_to_float<float>(data, tensor, offset, size,+ [](uint16_t v) { return ggml_fp16_to_fp32((ggml_fp16_t) v); });+ break;+ case GGML_TYPE_BF16:+ unpack_f16_tiled_to_float<ggml_bf16_t>(data, tensor, offset, size,+ [](uint16_t v) { return ggml_fp32_to_bf16(ggml_fp16_to_fp32((ggml_fp16_t) v)); });+ break;+ default:+ memcpy(data, (const char *) tensor->data + offset, size);+ break;+ }+}diff --git a/ggml/src/ggml-hexagon/htp-repack.h b/ggml/src/ggml-hexagon/htp-repack.hnew file mode 100644index 0000000000..ae66faf61f--- /dev/null+++ b/ggml/src/ggml-hexagon/htp-repack.h@@ -0,0 +1,64 @@+#ifndef HTP_REPACK_H+#define HTP_REPACK_H++#include <stddef.h>+#include <stdint.h>++#include "ggml.h"+#include "htp/htp-tensor.h"++static inline bool htp_is_tiled_weight_type(enum ggml_type type) {+ return type == GGML_TYPE_Q4_0 || type == GGML_TYPE_Q4_1 ||+ type == GGML_TYPE_Q8_0 || type == GGML_TYPE_IQ4_NL ||+ type == GGML_TYPE_MXFP4 || type == GGML_TYPE_Q6_K ||+ type == GGML_TYPE_Q4_K || type == GGML_TYPE_Q5_K ||+ type == GGML_TYPE_Q3_K || type == GGML_TYPE_Q2_K ||+ type == GGML_TYPE_F16 || type == GGML_TYPE_F32 ||+ type == GGML_TYPE_BF16;+}++static inline bool htp_is_tiled_weight_tensor(const struct ggml_tensor * t) {+ if (!ggml_is_contiguous(t) || ggml_is_view(t)) {+ return false;+ }+ if (!htp_is_tiled_weight_type(t->type)) {+ return false;+ }+ if (t->ne[0] < 32) {+ return false;+ }+ if (!ggml_is_quantized(t->type)) {+ return (t->ne[0] % 32 == 0) && (t->ne[1] % 32 == 0) && (t->ne[2] == 1) && (t->ne[3] == 1);+ }+ return t->ne[3] == 1;+}++// Size of one tiled row in the DSP tiled layout. The Q3_K..Q6_K tiles store uncompressed scales/mins,+// so they are larger than the ggml blocks. For the other tiled types (Q2_K included) the tile has the same size.+static inline size_t htp_tiled_row_size(enum ggml_type type, int64_t ne0) {+ if (type == GGML_TYPE_F16 || type == GGML_TYPE_F32 || type == GGML_TYPE_BF16) {+ return (size_t) ne0 * sizeof(ggml_fp16_t);+ }+ if (type == GGML_TYPE_Q6_K) {+ return (size_t) (ne0 / 32) * (HTP_TENSOR_TILE_SIZE_Q6_K / 32);+ }+ if (type == GGML_TYPE_Q4_K) {+ return (size_t) (ne0 / 32) * (HTP_TENSOR_TILE_SIZE_Q4_K / 32);+ }+ if (type == GGML_TYPE_Q5_K) {+ return (size_t) (ne0 / 32) * (HTP_TENSOR_TILE_SIZE_Q5_K / 32);+ }+ if (type == GGML_TYPE_Q3_K) {+ return (size_t) (ne0 / 32) * (HTP_TENSOR_TILE_SIZE_Q3_K / 32);+ }+ return ggml_row_size(type, ne0);+}++static inline bool htp_is_hmx_weight_type(enum ggml_type type) {+ return htp_is_tiled_weight_type(type);+}++void htp_repack_tensor_tiled(struct ggml_tensor * t, const void * data, size_t size);+void htp_unpack_tensor_tiled(void * data, const struct ggml_tensor * t, size_t offset, size_t size);++#endif // HTP_REPACK_Hdiff --git a/ggml/src/ggml-hexagon/htp/CMakeLists.txt b/ggml/src/ggml-hexagon/htp/CMakeLists.txtindex bc1474ae83..b1131317f5 100644--- a/ggml/src/ggml-hexagon/htp/CMakeLists.txt+++ b/ggml/src/ggml-hexagon/htp/CMakeLists.txt@@ -20,7 +20,7 @@ add_library(${HTP_LIB} SHAREDwork-queue.cdma-queue.chmx-queue.c- htp-tensor.c+ htp-dcache.cmatmul-ops.cflash-attn-ops.cgated-delta-net-ops.cdiff --git a/ggml/src/ggml-hexagon/htp/flash-attn-ops.c b/ggml/src/ggml-hexagon/htp/flash-attn-ops.cindex 2ee55feb9a..4b27346952 100644--- a/ggml/src/ggml-hexagon/htp/flash-attn-ops.c+++ b/ggml/src/ggml-hexagon/htp/flash-attn-ops.c@@ -361,7 +361,7 @@ static void flash_attn_ext_f16_thread(unsigned int nth, unsigned int ith, void *htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_FA_K_PREP, ir);for (uint32_t r = 0; r < current_block_size; ++r) {__fp16 * row_k = (__fp16 *)(k_base + r * factx->size_k_row_padded);- hvx_dequantize_row_q8_0_f16(row_k, row_k, DK);+ hvx_dequantize_row_flat_q8_0_f16(row_k, row_k, DK);}htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_FA_K_PREP, ir);}@@ -369,7 +369,7 @@ static void flash_attn_ext_f16_thread(unsigned int nth, unsigned int ith, void *htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_FA_V_PREP, ir);for (uint32_t r = 0; r < current_block_size; ++r) {__fp16 * row_v = (__fp16 *)(v_base + r * factx->size_v_row_padded);- hvx_dequantize_row_q8_0_f16(row_v, row_v, DV);+ hvx_dequantize_row_flat_q8_0_f16(row_v, row_v, DV);}htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_FA_V_PREP, ir);}@@ -625,7 +625,7 @@ static void fa_k_interleave_thread(unsigned int n, unsigned int i, void * data)if (factx->k->type == HTP_TYPE_Q8_0) {for (uint32_t r = start; r < end; ++r) {__fp16 * row_k = (__fp16 *)((char *)args->curr_k + r * args->src_stride * sizeof(__fp16));- hvx_dequantize_row_q8_0_f16(row_k, row_k, factx->DK);+ hvx_dequantize_row_flat_q8_0_f16(row_k, row_k, factx->DK);}}hmx_interleave_rows_to_tiles(factx->vtcm_k_tiles[args->buf_idx], (const __fp16 *) args->curr_k, total_rows, factx->DK_pad,@@ -679,7 +679,7 @@ static void fa_v_interleave_thread(unsigned int n, unsigned int i, void * data)if (factx->v->type == HTP_TYPE_Q8_0) {for (uint32_t r = start; r < end; ++r) {__fp16 * row_v = (__fp16 *)((char *)args->v_src + r * args->src_stride * sizeof(__fp16));- hvx_dequantize_row_q8_0_f16(row_v, row_v, factx->DV);+ hvx_dequantize_row_flat_q8_0_f16(row_v, row_v, factx->DV);}}hmx_interleave_cols_to_tiles(v_tiles_dst, (const __fp16 *) args->v_src, total_rows, factx->DV_pad,diff --git a/ggml/src/ggml-hexagon/htp/get-rows-ops.c b/ggml/src/ggml-hexagon/htp/get-rows-ops.cindex 90f38fa56b..27948c37ac 100644--- a/ggml/src/ggml-hexagon/htp/get-rows-ops.c+++ b/ggml/src/ggml-hexagon/htp/get-rows-ops.c@@ -17,21 +17,32 @@#include "htp-tensor.h"#include "hvx-utils.h"#include "hvx-quant.h"-#include "matmul-ops.h"#include "get-rows-ops.h"#include "work-queue.h"+struct get_rows_context;++typedef uint32_t (*get_rows_read_index_fn_t)(const void * src1, uint32_t idx, int idx_type);+typedef void (*get_rows_flat_compute_fn_t)(float * dst_spad, const void * src_spad, uint32_t cur_elems);+typedef void (*get_rows_tiled_compute_fn_t)(float * dst_spad, const uint8_t * src_spad, uint32_t n_k_tiles, uint32_t tile_stride, uint32_t row);+struct get_rows_context {struct htp_ops_context * octx;const struct htp_get_rows_kernel_params * kparams;struct htp_get_rows_vtcm_layout vtcm_layout;- uint8_t * vtcm_base;+ uint8_t * vtcm_base;+ const void * src1_data;+ get_rows_read_index_fn_t read_index;+ get_rows_flat_compute_fn_t compute_flat;+ get_rows_tiled_compute_fn_t compute_tiled;uint32_t task_start;uint32_t tasks;uint32_t tasks_per_thread;uint32_t tile_size;uint32_t tile_stride;- bool index_i32;+ uint32_t src0_row_size;+ uint32_t dst_row_size;+ int idx_type;};#define get_rows_preamble \@@ -64,229 +75,169 @@ struct get_rows_context {\const uint32_t nr = ne10 * ne11 * ne12;-#define GET_ROWS_THREAD_ST_FN(IDX_TYPE) \-static void get_rows_thread_st_##IDX_TYPE(unsigned int nth, unsigned int ith, void *data) { \- struct get_rows_context * grctx = (struct get_rows_context *)data; \- struct htp_ops_context * octx = grctx->octx; \- const struct htp_get_rows_kernel_params * kparams = grctx->kparams; \- get_rows_preamble; \- const uint32_t dr = grctx->tasks_per_thread; \- const uint32_t ir0 = grctx->task_start + dr * ith; \- if (ir0 >= grctx->task_start + grctx->tasks) { \- return; \- } \- const uint32_t ir1 = MIN(ir0 + dr, grctx->task_start + grctx->tasks); \- const uint32_t row_size_bytes = htp_tensor_get_row_size(octx->src[0]->type, ne00); \- dma_queue * dma_q = octx->ctx->dma[ith]; \- for (uint32_t i = ir0; i < ir1; ++i) { \- const uint32_t i12 = fastdiv(i, &kparams->div_ne10_ne11); \- const uint32_t rem = i - i12 * ne11 * ne10; \- const uint32_t i11 = fastdiv(rem, &kparams->div_ne10); \- const uint32_t i10 = rem - i11 * ne10; \- const IDX_TYPE * src1_ptr = (const IDX_TYPE *)(uintptr_t)(octx->src[1]->data + i10*nb10 + i11*nb11 + i12*nb12); \- const uint32_t i01 = (uint32_t)*src1_ptr; \- assert(i01 < ne01); \- const uint32_t q02 = fastdiv(i11, &kparams->div_ne02); \- const uint32_t i02 = i11 - q02 * ne02; \- const uint32_t q03 = fastdiv(i12, &kparams->div_ne03); \- const uint32_t i03 = i12 - q03 * ne03; \- const dma_addr_t src0_data = octx->src[0]->data + i01*nb01 + i02*nb02 + i03*nb03; \- const dma_addr_t dst_data = octx->dst->data + i10*nb1 + i11*nb2 + i12*nb3; \- while (!dma_queue_push(dma_q, dma_make_data(dst_data, src0_data), nb1, nb01, \- row_size_bytes, 1)) { \- dma_queue_pop(dma_q); \- } \- } \- dma_queue_flush(dma_q); \+static uint32_t get_rows_read_index_ddr(+ const void * src1,+ uint32_t idx,+ int idx_type+) {+ const uint32_t off = idx * ((idx_type == HTP_TYPE_I64) ? sizeof(int64_t) : sizeof(int32_t));+ return *(const uint32_t *) ((const uint8_t *) src1 + off);}-GET_ROWS_THREAD_ST_FN(int32_t)-GET_ROWS_THREAD_ST_FN(int64_t)--#define GET_ROWS_THREAD_DT_FN(TYPE_NAME, SRC0_SIZE_EXPR, IDX_TYPE, COMPUTE_EXPR) \-static void get_rows_thread_##TYPE_NAME##_##IDX_TYPE(unsigned int nth, unsigned int ith, void *data) { \- struct get_rows_context * grctx = (struct get_rows_context *)data; \- struct htp_ops_context * octx = grctx->octx; \- const struct htp_get_rows_kernel_params * kparams = grctx->kparams; \- get_rows_preamble; \- struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \- const uint32_t dr = grctx->tasks_per_thread; \- const uint32_t ir0 = grctx->task_start + dr * ith; \- if (ir0 >= grctx->task_start + grctx->tasks) { \- return; \- } \- const uint32_t ir1 = MIN(ir0 + dr, grctx->task_start + grctx->tasks); \- const uint32_t chunks_per_row = kparams->chunks_per_row; \- const uint32_t chunk_size = kparams->chunk_size; \- dma_queue * dma_q = octx->ctx->dma[ith]; \- const struct htp_get_rows_vtcm_layout * vtcm_layout = &grctx->vtcm_layout; \- uint8_t * vtcm_src0 = grctx->vtcm_base + vtcm_layout->off_src0 + ith * vtcm_layout->src0_bytes_per_thread; \- uint8_t * vtcm_dst = grctx->vtcm_base + vtcm_layout->off_dst + ith * vtcm_layout->dst_bytes_per_thread; \- for (uint32_t step = 0, spad_idx = 0; step < ir1 - ir0 && spad_idx < 2; ++step, spad_idx++) { \- const uint32_t i = ir0 + step; \- const uint32_t row_idx = fastdiv(i, &kparams->div_chunks_per_row); \- const uint32_t chunk_idx = i - row_idx * chunks_per_row; \- const uint32_t i12 = fastdiv(row_idx, &kparams->div_ne10_ne11); \- const uint32_t rem = row_idx - i12 * ne11 * ne10; \- const uint32_t i11 = fastdiv(rem, &kparams->div_ne10); \- const uint32_t i10 = rem - i11 * ne10; \- const IDX_TYPE * src1_ptr = (const IDX_TYPE *)(uintptr_t)(octx->src[1]->data + i10*nb10 + i11*nb11 + i12*nb12); \- const uint32_t i01 = (uint32_t)*src1_ptr; \- assert(i01 < ne01); \- const uint32_t q02 = fastdiv(i11, &kparams->div_ne02); \- const uint32_t i02 = i11 - q02 * ne02; \- const uint32_t q03 = fastdiv(i12, &kparams->div_ne03); \- const uint32_t i03 = i12 - q03 * ne03; \- const uint32_t offset = chunk_idx * chunk_size; \- const uint32_t cur_elems = (offset < ne00) ? MIN(chunk_size, ne00 - offset) : 0; \- const uint32_t cur_src0_bytes = SRC0_SIZE_EXPR(cur_elems); \- const uint32_t cur_dst_bytes = cur_elems * sizeof(float); \- const dma_addr_t src0_data = octx->src[0]->data + i01*nb01 + i02*nb02 + i03*nb03 + SRC0_SIZE_EXPR(offset); \- dma_queue_push(dma_q, \- dma_make_data(octx->dst->data, \- vtcm_dst + spad_idx * vtcm_layout->dst_spad_half_size), \- cur_dst_bytes, vtcm_layout->dst_spad_half_size, cur_dst_bytes, 0); \- dma_queue_push(dma_q, \- dma_make_data(vtcm_src0 + spad_idx * vtcm_layout->src0_spad_half_size, src0_data), \- vtcm_layout->src0_spad_half_size, cur_src0_bytes, cur_src0_bytes, 1); \- } \- for (uint32_t step = 0; step < ir1 - ir0; ++step) { \- const uint32_t i = ir0 + step; \- void * dst_spad = (void *) dma_queue_pop(dma_q).src; \- void * src_spad = (void *) dma_queue_pop(dma_q).dst; \- const uint32_t row_idx = fastdiv(i, &kparams->div_chunks_per_row); \- const uint32_t chunk_idx = i - row_idx * chunks_per_row; \- const uint32_t i12 = fastdiv(row_idx, &kparams->div_ne10_ne11); \- const uint32_t rem = row_idx - i12 * ne11 * ne10; \- const uint32_t i11 = fastdiv(rem, &kparams->div_ne10); \- const uint32_t i10 = rem - i11 * ne10; \- const uint32_t offset = chunk_idx * chunk_size; \- const uint32_t cur_elems = (offset < ne00) ? MIN(chunk_size, ne00 - offset) : 0; \- const uint32_t cur_dst_bytes = cur_elems * sizeof(float); \- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, i); \- COMPUTE_EXPR; \- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, i); \- const dma_addr_t dst_data = octx->dst->data + i10*nb1 + i11*nb2 + i12*nb3 + offset * sizeof(float); \- dma_queue_push(dma_q, \- dma_make_data(dst_data, dst_spad), \- cur_dst_bytes, vtcm_layout->dst_spad_half_size, cur_dst_bytes, 1); \- const uint32_t next_step = step + 2; \- if (next_step < ir1 - ir0) { \- const uint32_t pi = ir0 + next_step; \- const uint32_t prow_idx = fastdiv(pi, &kparams->div_chunks_per_row); \- const uint32_t pchunk_idx = pi - prow_idx * chunks_per_row; \- const uint32_t pi12 = fastdiv(prow_idx, &kparams->div_ne10_ne11); \- const uint32_t prem = prow_idx - pi12 * ne11 * ne10; \- const uint32_t pi11 = fastdiv(prem, &kparams->div_ne10); \- const uint32_t pi10 = prem - pi11 * ne10; \- const IDX_TYPE * psrc1_ptr = (const IDX_TYPE *)(uintptr_t)(octx->src[1]->data + pi10*nb10 + pi11*nb11 + pi12*nb12); \- const uint32_t pi01 = (uint32_t)*psrc1_ptr; \- assert(pi01 < ne01); \- const uint32_t pq02 = fastdiv(pi11, &kparams->div_ne02); \- const uint32_t pi02 = pi11 - pq02 * ne02; \- const uint32_t pq03 = fastdiv(pi12, &kparams->div_ne03); \- const uint32_t pi03 = pi12 - pq03 * ne03; \- const uint32_t poffset = pchunk_idx * chunk_size; \- const uint32_t pcur_elems = (poffset < ne00) ? MIN(chunk_size, ne00 - poffset) : 0; \- const uint32_t pcur_src0_bytes = SRC0_SIZE_EXPR(pcur_elems); \- const dma_addr_t psrc0_data = \- octx->src[0]->data + pi01*nb01 + pi02*nb02 + pi03*nb03 + SRC0_SIZE_EXPR(poffset); \- dma_queue_push(dma_q, \- dma_make_data(src_spad, psrc0_data), \- vtcm_layout->src0_spad_half_size, pcur_src0_bytes, pcur_src0_bytes, 1); \- } \- } \- dma_queue_flush(dma_q); \+static __attribute__((noinline)) uint32_t get_rows_read_index_vtcm(+ const void * src1,+ uint32_t idx,+ int idx_type+) {+ const uint32_t off = idx * ((idx_type == HTP_TYPE_I64) ? sizeof(int64_t) : sizeof(int32_t));+ const HVX_Vector v = hvx_vmem((const uint8_t *) src1 + off); // off & ~127u in HW+ // vextract is normally slow, but here it avoids an HVX->L2 store and L1D miss+ return (uint32_t) Q6_R_vextract_VR(v, off);}-#define F16_BYTES(n) ((n) * sizeof(__fp16))-#define Q8_0_BYTES(n) (((n) / 32) * sizeof(block_q8_0))+static void get_rows_thread_sametype(unsigned int nth, unsigned int ith, void * data) {+ struct get_rows_context * grctx = (struct get_rows_context *) data;+ struct htp_ops_context * octx = grctx->octx;+ const struct htp_get_rows_kernel_params * kparams = grctx->kparams;+ get_rows_preamble;-static __attribute__((noinline)) void compute_get_rows_f16(float * dst_spad, const void * src_spad, uint32_t cur_elems) {- hvx_dequantize_row_f16_f32(dst_spad, src_spad, cur_elems);-}+ const uint32_t dr = grctx->tasks_per_thread;+ const uint32_t ir0 = grctx->task_start + dr * ith;+ if (ir0 >= grctx->task_start + grctx->tasks) {+ return;+ }+ const uint32_t ir1 = MIN(ir0 + dr, grctx->task_start + grctx->tasks);+ const uint32_t row_size_bytes = grctx->src0_row_size;+ dma_queue * dma_q = octx->ctx->dma[ith];-static __attribute__((noinline)) void compute_get_rows_q8_0(float * dst_spad, const void * src_spad, uint32_t cur_elems) {- hvx_dequantize_row_q8_0_f32(dst_spad, src_spad, cur_elems);-}+ for (uint32_t i = ir0; i < ir1; ++i) {+ const uint32_t i12 = fastdiv(i, &kparams->div_ne10_ne11);+ const uint32_t rem = i - i12 * ne11 * ne10;+ const uint32_t i11 = fastdiv(rem, &kparams->div_ne10);+ const uint32_t i10 = rem - i11 * ne10;-GET_ROWS_THREAD_DT_FN(f16, F16_BYTES, int32_t, { compute_get_rows_f16((float *)dst_spad, src_spad, cur_elems); })-GET_ROWS_THREAD_DT_FN(f16, F16_BYTES, int64_t, { compute_get_rows_f16((float *)dst_spad, src_spad, cur_elems); })--GET_ROWS_THREAD_DT_FN(q8_0, Q8_0_BYTES, int32_t, { compute_get_rows_q8_0((float *)dst_spad, src_spad, cur_elems); })-GET_ROWS_THREAD_DT_FN(q8_0, Q8_0_BYTES, int64_t, { compute_get_rows_q8_0((float *)dst_spad, src_spad, cur_elems); })---static __attribute__((noinline)) void compute_get_rows_tiled(float * dst, const uint8_t * tile, uint32_t row, bool q4, bool q4_k) {- const HVX_VectorPred first2 = Q6_Q_vsetq_R(2);- const HVX_VectorPred first4 = Q6_Q_vsetq_R(4);- HVX_Vector vq = Q6_V_vzero();- if (q4) {- const HVX_VectorPred first1 = Q6_Q_vsetq_R(1);- const HVX_VectorPred first3 = Q6_Q_vsetq_R(3);- for (int group = 3; group >= 0; --group) {- const HVX_Vector v = Q6_V_vror_VR(hvx_vmem(tile + group * VLEN), row);- // Four planes contribute bytes at 0, 32, 64 and 96 after rotation.- HVX_Vector packed = Q6_V_vmux_QVV(first1, v, Q6_V_vror_VR(v, 31));- packed = Q6_V_vmux_QVV(first2, packed, Q6_V_vror_VR(v, 62));- packed = Q6_V_vmux_QVV(first3, packed, Q6_V_vror_VR(v, 93));- vq = Q6_V_vmux_QVV(first4, packed, Q6_V_vror_VR(vq, VLEN - 4));- }- const HVX_Vector lo = Q6_V_vand_VV(vq, Q6_Vb_vsplat_R(0x0F));- const HVX_Vector hi = Q6_Vub_vlsr_VubR(vq, 4);- vq = Q6_V_lo_W(Q6_W_vshuff_VVR(hi, lo, -1));- if (!q4_k) {- vq = Q6_Vb_vsub_VbVb(vq, Q6_Vb_vsplat_R(8));- }- } else {- for (int group = 7; group >= 0; --group) {- const HVX_Vector v = Q6_V_vror_VR(hvx_vmem(tile + group * VLEN), 2 * row);- // Two planes contribute halfwords at 0 and 64 after rotation.- const HVX_Vector packed = Q6_V_vmux_QVV(first2, v, Q6_V_vror_VR(v, 62));- vq = Q6_V_vmux_QVV(first4, packed, Q6_V_vror_VR(vq, VLEN - 4));+ const uint32_t i01 = grctx->read_index(grctx->src1_data, i, grctx->idx_type);+ assert(i01 < ne01);++ const uint32_t q02 = fastdiv(i11, &kparams->div_ne02);+ const uint32_t i02 = i11 - q02 * ne02;+ const uint32_t q03 = fastdiv(i12, &kparams->div_ne03);+ const uint32_t i03 = i12 - q03 * ne03;++ const dma_addr_t src0_data = octx->src[0]->data + i01*nb01 + i02*nb02 + i03*nb03;+ const dma_addr_t dst_data = octx->dst->data + i10*nb1 + i11*nb2 + i12*nb3;++ while (!dma_queue_push(dma_q, dma_make_data(dst_data, src0_data), nb1, nb01, row_size_bytes, 1)) {+ dma_queue_pop(dma_q);}}- const HVX_Vector scales = hvx_vmem(tile + (q4 ? 512 : 1024));- const HVX_Vector scale_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, (q4_k ? 4 : 2) * row));- const HVX_Vector scale = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_hf));- const HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(vq);- const HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16));- HVX_Vector values = hvx_vec_mul_f32_f32(Q6_Vsf_equals_Vw(Q6_V_lo_W(p32)), scale);- if (q4_k) {- const HVX_Vector offset_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 4 * row + 2));- const HVX_Vector offset = Q6_V_lo_W(hvx_vec_f16_to_f32(offset_hf));- values = hvx_vec_add_f32_f32(values, offset);+ dma_queue_flush(dma_q);+}++++struct get_rows_flat_task {+ dma_addr_t src0_data;+ dma_addr_t dst_data;+};++static inline struct get_rows_flat_task get_rows_flat_calc_task(+ const struct htp_ops_context * octx,+ const struct get_rows_context * grctx,+ uint32_t i,+ bool need_src0+) {+ const struct htp_get_rows_kernel_params * kparams = grctx->kparams;+ get_rows_preamble;++ const uint32_t i12 = fastdiv(i, &kparams->div_ne10_ne11);+ const uint32_t rem = i - i12 * ne11 * ne10;+ const uint32_t i11 = fastdiv(rem, &kparams->div_ne10);+ const uint32_t i10 = rem - i11 * ne10;++ struct get_rows_flat_task task;+ task.src0_data = 0;+ task.dst_data = octx->dst->data + i10*nb1 + i11*nb2 + i12*nb3;++ if (need_src0) {+ const uint32_t i01 = grctx->read_index(grctx->src1_data, i, grctx->idx_type);+ assert(i01 < ne01);++ const uint32_t q02 = fastdiv(i11, &kparams->div_ne02);+ const uint32_t i02 = i11 - q02 * ne02;+ const uint32_t q03 = fastdiv(i12, &kparams->div_ne03);+ const uint32_t i03 = i12 - q03 * ne03;++ task.src0_data = octx->src[0]->data + i01*nb01 + i02*nb02 + i03*nb03;}- *(HVX_Vector *) dst = values;++ return task;}-static __attribute__((noinline)) void compute_get_rows_q6_k(float * dst, const uint8_t * tile, uint32_t row) {- const HVX_VectorPred first4 = Q6_Q_vsetq_R(4);- const HVX_VectorPred first16 = Q6_Q_vsetq_R(16 * sizeof(float));- const HVX_Vector mask_0f = Q6_Vb_vsplat_R(0x0F);- const HVX_Vector mask_03 = Q6_Vb_vsplat_R(0x03);- HVX_Vector vq = Q6_V_vzero();-- for (int group = 7; group >= 0; --group) {- const HVX_Vector lo_plane = Q6_V_vror_VR(hvx_vmem(tile + (group >> 1) * VLEN), 4 * row);- const HVX_Vector hi_plane = Q6_V_vror_VR(hvx_vmem(tile + 512 + (group >> 2) * VLEN), 4 * row);- const HVX_Vector lo = (group & 1) ? Q6_Vub_vlsr_VubR(lo_plane, 4) : Q6_V_vand_VV(lo_plane, mask_0f);- const HVX_Vector hi = Q6_Vub_vlsr_VubR(hi_plane, 2 * (group & 3));- const HVX_Vector packed = Q6_V_vor_VV(lo, Q6_Vw_vasl_VwR(Q6_V_vand_VV(hi, mask_03), 4));- vq = Q6_V_vmux_QVV(first4, packed, Q6_V_vror_VR(vq, VLEN - 4));+static void get_rows_thread_flat(unsigned int nth, unsigned int ith, void * data) {+ struct get_rows_context * grctx = (struct get_rows_context *) data;+ struct htp_ops_context * octx = grctx->octx;+ get_rows_preamble;++ const uint32_t dr = grctx->tasks_per_thread;+ const uint32_t ir0 = grctx->task_start + dr * ith;+ if (ir0 >= grctx->task_start + grctx->tasks) {+ return;+ }+ const uint32_t ir1 = MIN(ir0 + dr, grctx->task_start + grctx->tasks);++ dma_queue * dma_q = octx->ctx->dma[ith];+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ const struct htp_get_rows_vtcm_layout * vtcm_layout = &grctx->vtcm_layout;+ uint8_t * vtcm_src0 = VTCM_LAYOUT_PTR(uint8_t, grctx->vtcm_base, vtcm_layout->off_src0) + ith * vtcm_layout->src0_bytes_per_thread;+ uint8_t * vtcm_dst = VTCM_LAYOUT_PTR(uint8_t, grctx->vtcm_base, vtcm_layout->off_dst) + ith * vtcm_layout->dst_bytes_per_thread;+ const uint32_t src0_bytes = grctx->src0_row_size;+ const uint32_t dst_bytes = grctx->dst_row_size;+ get_rows_flat_compute_fn_t compute = grctx->compute_flat;++ for (uint32_t step = 0, spad_idx = 0; step < ir1 - ir0 && spad_idx < 2; ++step, spad_idx++) {+ const uint32_t i = ir0 + step;+ struct get_rows_flat_task task = get_rows_flat_calc_task(octx, grctx, i, true);++ dma_queue_push(dma_q,+ dma_make_data(octx->dst->data,+ vtcm_dst + spad_idx * vtcm_layout->dst_spad_half_size),+ dst_bytes, vtcm_layout->dst_spad_half_size, dst_bytes, 0);+ dma_queue_push(dma_q,+ dma_make_data(vtcm_src0 + spad_idx * vtcm_layout->src0_spad_half_size, task.src0_data),+ vtcm_layout->src0_spad_half_size, src0_bytes, src0_bytes, 1);+ }++ for (uint32_t step = 0; step < ir1 - ir0; ++step) {+ const uint32_t i = ir0 + step;+ void * dst_spad = (void *) dma_queue_pop(dma_q).src;+ void * src_spad = (void *) dma_queue_pop(dma_q).dst;++ struct get_rows_flat_task task = get_rows_flat_calc_task(octx, grctx, i, false);++ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) i);+ compute((float *) dst_spad, src_spad, ne00);+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) i);++ dma_queue_push(dma_q,+ dma_make_data(task.dst_data, dst_spad),+ dst_bytes, vtcm_layout->dst_spad_half_size, dst_bytes, 1);++ const uint32_t next_step = step + 2;+ if (next_step < ir1 - ir0) {+ const uint32_t pi = ir0 + next_step;+ struct get_rows_flat_task next_task = get_rows_flat_calc_task(octx, grctx, pi, true);+ dma_queue_push(dma_q,+ dma_make_data(src_spad, next_task.src0_data),+ vtcm_layout->src0_spad_half_size, src0_bytes, src0_bytes, 1);+ }}- const HVX_Vector scales = hvx_vmem(tile + 768);- const HVX_Vector scale_lo_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 2 * row));- const HVX_Vector scale_hi_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 64 + 2 * row));- const HVX_Vector scale_lo = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_lo_hf));- const HVX_Vector scale_hi = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_hi_hf));- const HVX_Vector scale = Q6_V_vmux_QVV(first16, scale_lo, scale_hi);- const HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(Q6_Vb_vsub_VbVb(vq, Q6_Vb_vsplat_R(32)));- const HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16));- *(HVX_Vector *) dst = hvx_vec_mul_f32_f32(Q6_Vsf_equals_Vw(Q6_V_lo_W(p32)), scale);+ dma_queue_flush(dma_q);}++struct get_rows_tiled_task {dma_addr_t tile_src_base;dma_addr_t dst_data;@@ -307,16 +258,15 @@ static inline struct get_rows_tiled_task get_rows_tiled_calc_task(const uint32_t rem = i - i12 * ne11 * ne10;const uint32_t i11 = fastdiv(rem, &kparams->div_ne10);const uint32_t i10 = rem - i11 * ne10;- const dma_addr_t src1_data = octx->src[1]->data + i10*nb10 + i11*nb11 + i12*nb12;- const uint32_t i01 = grctx->index_i32 ? *(const int32_t *)(uintptr_t) src1_data : (uint32_t) *(const int64_t *)(uintptr_t) src1_data;+ const uint32_t i01 = grctx->read_index(grctx->src1_data, i, grctx->idx_type);assert(i01 < ne01);const uint32_t q02 = fastdiv(i11, &kparams->div_ne02);const uint32_t i02 = i11 - q02 * ne02;const uint32_t q03 = fastdiv(i12, &kparams->div_ne03);const uint32_t i03 = i12 - q03 * ne03;- const uint32_t column_tile = i01 / HTP_MM_HMX_TILE_N_ROWS;- const uint32_t row = i01 % HTP_MM_HMX_TILE_N_ROWS;+ const uint32_t column_tile = i01 / HTP_TENSOR_TILE_N_ROWS;+ const uint32_t row = i01 % HTP_TENSOR_TILE_N_ROWS;const dma_addr_t matrix = octx->src[0]->data + i02*nb02 + i03*nb03;struct get_rows_tiled_task task;@@ -329,7 +279,6 @@ static inline struct get_rows_tiled_task get_rows_tiled_calc_task(static void get_rows_thread_tiled(unsigned int nth, unsigned int ith, void * data) {struct get_rows_context * grctx = (struct get_rows_context *) data;struct htp_ops_context * octx = grctx->octx;- const struct htp_get_rows_kernel_params * kparams = grctx->kparams;get_rows_preamble;const uint32_t dr = grctx->tasks_per_thread;@@ -339,19 +288,17 @@ static void get_rows_thread_tiled(unsigned int nth, unsigned int ith, void * dat}const uint32_t ir1 = MIN(ir0 + dr, grctx->task_start + grctx->tasks);- const uint32_t n_k_tiles = ne00 / HTP_MM_HMX_TILE_N_COLS;+ const uint32_t n_k_tiles = ne00 / HTP_TENSOR_TILE_N_COLS;const struct htp_get_rows_vtcm_layout * vtcm_layout = &grctx->vtcm_layout;- uint8_t * src_spad_base = grctx->vtcm_base + vtcm_layout->off_src0 + ith * vtcm_layout->src0_bytes_per_thread;- uint8_t * dst_spad_base = grctx->vtcm_base + vtcm_layout->off_dst + ith * vtcm_layout->dst_bytes_per_thread;+ uint8_t * src_spad_base = VTCM_LAYOUT_PTR(uint8_t, grctx->vtcm_base, vtcm_layout->off_src0) + ith * vtcm_layout->src0_bytes_per_thread;+ uint8_t * dst_spad_base = VTCM_LAYOUT_PTR(uint8_t, grctx->vtcm_base, vtcm_layout->off_dst) + ith * vtcm_layout->dst_bytes_per_thread;dma_queue * dma_q = octx->ctx->dma[ith];struct htp_thread_trace * tr = &octx->ctx->trace[ith];const uint32_t tile_size = grctx->tile_size;const uint32_t tile_stride = grctx->tile_stride;- const uint32_t dst_bytes = ne00 * sizeof(float);- const bool is_q4 = octx->src[0]->type == HTP_TYPE_Q4_0 || octx->src[0]->type == HTP_TYPE_Q4_K;- const bool is_q4_k = octx->src[0]->type == HTP_TYPE_Q4_K;- const bool is_q6_k = octx->src[0]->type == HTP_TYPE_Q6_K;+ const uint32_t dst_bytes = grctx->dst_row_size;+ get_rows_tiled_compute_fn_t compute_tiled = grctx->compute_tiled;for (uint32_t step = 0, spad_idx = 0; step < ir1 - ir0 && spad_idx < 2; ++step, ++spad_idx) {const uint32_t i = ir0 + step;@@ -376,15 +323,7 @@ static void get_rows_thread_tiled(unsigned int nth, unsigned int ith, void * datstruct get_rows_tiled_task task = get_rows_tiled_calc_task(octx, grctx, i, n_k_tiles, tile_size);htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) i);- for (uint32_t k_tile = 0; k_tile < n_k_tiles; ++k_tile) {- const uint8_t * tile = src_spad + k_tile * tile_stride;- float * dst_block = dst_spad + k_tile * HTP_MM_HMX_TILE_N_COLS;- if (is_q6_k) {- compute_get_rows_q6_k(dst_block, tile, task.row);- } else {- compute_get_rows_tiled(dst_block, tile, task.row, is_q4, is_q4_k);- }- }+ compute_tiled(dst_spad, src_spad, n_k_tiles, tile_stride, task.row);htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, (uint16_t) i);// Real writeback of dst_spad@@ -405,16 +344,40 @@ static void get_rows_thread_tiled(unsigned int nth, unsigned int ith, void * datdma_queue_flush(dma_q);}+static __attribute__((noinline)) void get_rows_dma_src1(+ struct htp_ops_context * octx,+ uint8_t * vtcm_src1,+ uint32_t elem_size+) {+ const uint32_t ne10 = octx->src[1]->ne[0];+ const uint32_t ne11 = octx->src[1]->ne[1];+ const uint32_t ne12 = octx->src[1]->ne[2];+ const uint32_t ne13 = octx->src[1]->ne[3];+ const uint32_t nb11 = octx->src[1]->nb[1];+ const uint32_t nb12 = octx->src[1]->nb[2];+ const uint32_t nb13 = octx->src[1]->nb[3];++ dma_queue * dma_q = octx->ctx->dma[0];+ uint8_t * cur_dst = vtcm_src1;+ for (uint32_t i13 = 0; i13 < ne13; ++i13) {+ for (uint32_t i12 = 0; i12 < ne12; ++i12) {+ const dma_addr_t src_plane = octx->src[1]->data + i12 * nb12 + i13 * nb13;+ while (!dma_queue_push(dma_q, dma_make_data(cur_dst, src_plane),+ ne10 * elem_size, nb11, ne10 * elem_size, ne11)) {+ dma_queue_pop(dma_q);+ }+ cur_dst += ne10 * ne11 * elem_size;+ }+ }+ dma_queue_flush(dma_q);+}+int op_get_rows(struct htp_ops_context * octx) {const struct htp_get_rows_kernel_params * kparams = (const struct htp_get_rows_kernel_params *) octx->kernel_params;- if (octx->src[0]->type != HTP_TYPE_F32 &&- octx->src[0]->type != HTP_TYPE_F16 &&- octx->src[0]->type != HTP_TYPE_Q4_0 &&- octx->src[0]->type != HTP_TYPE_Q4_K &&- octx->src[0]->type != HTP_TYPE_Q6_K &&- octx->src[0]->type != HTP_TYPE_Q8_0 &&- octx->src[0]->type != HTP_TYPE_I32) {+ const uint32_t ne00 = octx->src[0]->ne[0];+ const uint32_t src0_row_size = htp_tensor_get_row_size(octx->src[0]->type, ne00);+ if (src0_row_size == 0) {return HTP_STATUS_NO_SUPPORT;}@@ -432,20 +395,16 @@ int op_get_rows(struct htp_ops_context * octx) {return HTP_STATUS_NO_SUPPORT;}- if (htp_tensor_is_extended(octx->src[1])) {- return HTP_STATUS_NO_SUPPORT;- }-const struct htp_tensor * dst = octx->dst;const uint32_t total_tasks = kparams->total_tasks;- const size_t dst_row_size = htp_tensor_get_row_size(dst->type, dst->ne[0]);+ const uint32_t dst_row_size = (kparams->kernel_type == HTP_GET_ROWS_KERNEL_SAMETYPE) ? src0_row_size : ne00 * sizeof(float);uint32_t task_start = 0;uint32_t tasks = total_tasks;if (octx->ctx->mdev.count > 1) {uint32_t tasks_per_chunk = 1;- htp_tensor_mdev_rows_per_chunk(dst, dst_row_size / dst->ne[0], (uint32_t) dst_row_size, &tasks_per_chunk);+ htp_tensor_mdev_rows_per_chunk(dst, dst_row_size / dst->ne[0], dst_row_size, &tasks_per_chunk);const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(total_tasks, tasks_per_chunk, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);task_start = range.start;tasks = range.count;@@ -465,15 +424,18 @@ int op_get_rows(struct htp_ops_context * octx) {grctx.octx = octx;grctx.kparams = kparams;grctx.vtcm_base = (uint8_t *)octx->ctx->vtcm_base;+ grctx.compute_flat = NULL;+ grctx.compute_tiled = NULL;grctx.task_start = task_start;grctx.tasks = tasks;grctx.tasks_per_thread = octx->ctx->mdev.count == 1 ? kparams->tasks_per_thread : fastdiv(tasks + n_threads - 1, &octx->n_threads_div);- grctx.tile_size = htp_mm_get_weight_tile_size(octx->src[0]->type);- grctx.tile_stride = (grctx.tile_size + 127) & ~127;- grctx.index_i32 = octx->src[1]->type == HTP_TYPE_I32;+ grctx.tile_size = 0;+ grctx.tile_stride = 0;+ grctx.src0_row_size = src0_row_size;+ grctx.dst_row_size = dst_row_size;+ grctx.idx_type = octx->src[1]->type;- const uint32_t ne00 = octx->src[0]->ne[0];- htp_get_rows_vtcm_layout_build(&grctx.vtcm_layout, kparams->kernel_type, octx->src[0]->type, ne00, n_threads);+ htp_get_rows_vtcm_layout_build(&grctx.vtcm_layout, kparams->kernel_type, octx->src[0]->type, ne00, total_tasks, grctx.idx_type, n_threads);if (grctx.vtcm_layout.total_bytes > octx->ctx->vtcm_size) {FARF(ERROR, "get-rows: VTCM reservation %zu is too small, needed %zu\n",@@ -481,27 +443,56 @@ int op_get_rows(struct htp_ops_context * octx) {return HTP_STATUS_INVAL_PARAMS;}- const bool is_i32 = (octx->src[1]->type == HTP_TYPE_I32);-work_queue_func_t q_func = NULL;switch (kparams->kernel_type) {case HTP_GET_ROWS_KERNEL_SAMETYPE:- q_func = (work_queue_func_t)(is_i32 ? get_rows_thread_st_int32_t : get_rows_thread_st_int64_t);+ q_func = get_rows_thread_sametype;break;case HTP_GET_ROWS_KERNEL_TILED:+ switch (octx->src[0]->type) {+ case HTP_TYPE_F16: grctx.compute_tiled = hvx_dequantize_row_tiled_f16_f32; break;+ case HTP_TYPE_Q8_0: grctx.compute_tiled = hvx_dequantize_row_tiled_q8_0_f32; break;+ case HTP_TYPE_Q4_0: grctx.compute_tiled = hvx_dequantize_row_tiled_q4_0_f32; break;+ case HTP_TYPE_Q4_K: grctx.compute_tiled = hvx_dequantize_row_tiled_q4_k_f32; break;+ case HTP_TYPE_Q5_K: grctx.compute_tiled = hvx_dequantize_row_tiled_q5_k_f32; break;+ case HTP_TYPE_Q6_K: grctx.compute_tiled = hvx_dequantize_row_tiled_q6_k_f32; break;+ default: return HTP_STATUS_NO_SUPPORT;+ }+ grctx.tile_size = htp_tensor_get_tile_size(octx->src[0]->type);+ grctx.tile_stride = htp_tensor_get_tile_padded_size(octx->src[0]->type);q_func = get_rows_thread_tiled;break;case HTP_GET_ROWS_KERNEL_FLAT:switch (octx->src[0]->type) {- case HTP_TYPE_F16: q_func = (work_queue_func_t)(is_i32 ? get_rows_thread_f16_int32_t : get_rows_thread_f16_int64_t); break;- case HTP_TYPE_Q8_0: q_func = (work_queue_func_t)(is_i32 ? get_rows_thread_q8_0_int32_t : get_rows_thread_q8_0_int64_t); break;+ case HTP_TYPE_F16: grctx.compute_flat = hvx_dequantize_row_flat_f16_f32; break;+ case HTP_TYPE_BF16: grctx.compute_flat = hvx_dequantize_row_flat_bf16_f32; break;+ case HTP_TYPE_Q8_0: grctx.compute_flat = hvx_dequantize_row_flat_q8_0_f32; break;+ case HTP_TYPE_Q4_0: grctx.compute_flat = hvx_dequantize_row_flat_q4_0_f32; break;+ case HTP_TYPE_Q4_K: grctx.compute_flat = hvx_dequantize_row_flat_q4_k_f32; break;+ case HTP_TYPE_Q5_K: grctx.compute_flat = hvx_dequantize_row_flat_q5_k_f32; break;+ case HTP_TYPE_Q6_K: grctx.compute_flat = hvx_dequantize_row_flat_q6_k_f32; break;default: return HTP_STATUS_NO_SUPPORT;}+ q_func = get_rows_thread_flat;break;default:return HTP_STATUS_NO_SUPPORT;}+ const uint32_t elem_size = (grctx.idx_type == HTP_TYPE_I64) ? sizeof(int64_t) : sizeof(int32_t);+ const size_t src1_bytes = (size_t) total_tasks * elem_size;++ if (!htp_tensor_is_extended(octx->src[1]) && src1_bytes <= 128 &&+ htp_tensor_is_contiguous(octx->src[1], elem_size)) {+ grctx.src1_data = (const void *) (uintptr_t) octx->src[1]->data;+ grctx.read_index = get_rows_read_index_ddr;+ } else {+ uint8_t * vtcm_src1 = VTCM_LAYOUT_PTR(uint8_t, grctx.vtcm_base, grctx.vtcm_layout.off_src1);+ get_rows_dma_src1(octx, vtcm_src1, elem_size);+ grctx.src1_data = vtcm_src1;+ grctx.read_index = get_rows_read_index_vtcm;+ }+FARF(HIGH, "get-rows: (%ux%ux%ux%u) x (%ux%ux%ux%u) -> (%ux%ux%ux%u) : src0-vtcm-size %zu dst-vtcm-size %zu kernel-type %d n-threads %d\n",octx->src[0]->ne[0], octx->src[0]->ne[1], octx->src[0]->ne[2], octx->src[0]->ne[3],octx->src[1]->ne[0], octx->src[1]->ne[1], octx->src[1]->ne[2], octx->src[1]->ne[3],diff --git a/ggml/src/ggml-hexagon/htp/get-rows-ops.h b/ggml/src/ggml-hexagon/htp/get-rows-ops.hindex 69fa544649..f44e58d164 100644--- a/ggml/src/ggml-hexagon/htp/get-rows-ops.h+++ b/ggml/src/ggml-hexagon/htp/get-rows-ops.h@@ -5,7 +5,8 @@#include <string.h>#include "hex-fastdiv.h"-#include "matmul-ops.h"+#include "htp-tensor.h"+#include "htp-vtcm.h"enum htp_get_rows_kernel_type {HTP_GET_ROWS_KERNEL_SAMETYPE = 0,@@ -16,8 +17,6 @@ enum htp_get_rows_kernel_type {struct htp_get_rows_kernel_params {int32_t n_threads;int32_t kernel_type;- int32_t chunks_per_row;- int32_t chunk_size;int32_t total_tasks;int32_t tasks_per_thread;int32_t vtcm_size;@@ -25,13 +24,14 @@ struct htp_get_rows_kernel_params {// Fastdiv helpersstruct fastdiv_values div_ne10;struct fastdiv_values div_ne10_ne11;- struct fastdiv_values div_chunks_per_row;struct fastdiv_values div_ne02;struct fastdiv_values div_ne03;};struct htp_get_rows_vtcm_layout {size_t total_bytes;+ size_t off_src1;+ size_t src1_bytes;size_t off_src0;size_t off_dst;@@ -43,57 +43,44 @@ struct htp_get_rows_vtcm_layout {};static inline void htp_get_rows_vtcm_layout_build(- struct htp_get_rows_vtcm_layout * vtcm_layout,+ struct htp_get_rows_vtcm_layout * L,int kernel_type,int type,uint32_t ne00,+ uint32_t total_tasks,+ int idx_type,uint32_t n_threads) {- if (kernel_type == HTP_GET_ROWS_KERNEL_SAMETYPE) {- memset(vtcm_layout, 0, sizeof(*vtcm_layout));- return;+ memset(L, 0, sizeof(*L));++ const size_t elem_size = (idx_type == HTP_TYPE_I64) ? sizeof(int64_t) : sizeof(int32_t);+ const size_t src1_raw_bytes = total_tasks * elem_size;+ const size_t src1_bytes = ((src1_raw_bytes + 127) & ~127) + 128;++ L->src1_bytes = src1_bytes;+ const size_t src1_alloc_bytes = (src1_bytes + 255) & ~255;++ if (kernel_type != HTP_GET_ROWS_KERNEL_SAMETYPE) {+ size_t src0_row_size = 0;+ if (kernel_type == HTP_GET_ROWS_KERNEL_TILED) {+ const size_t tile_stride = htp_tensor_get_tile_padded_size(type);+ const uint32_t n_k_tiles = ne00 / HTP_TENSOR_TILE_N_COLS;+ src0_row_size = n_k_tiles > 0 ? (n_k_tiles * tile_stride) : tile_stride;+ } else {+ src0_row_size = htp_tensor_get_row_size(type, ne00);+ }++ L->src0_spad_half_size = (src0_row_size + 255) & ~255;+ L->dst_spad_half_size = (ne00 * sizeof(float) + 255) & ~255;+ L->src0_bytes_per_thread = 2 * L->src0_spad_half_size;+ L->dst_bytes_per_thread = 2 * L->dst_spad_half_size;}- if (kernel_type == HTP_GET_ROWS_KERNEL_TILED) {- const size_t tile_size = htp_mm_get_weight_tile_size(type);- const size_t tile_stride = (tile_size + 127) & ~127;- const uint32_t n_k_tiles = ne00 / HTP_MM_HMX_TILE_N_COLS;- const size_t row_tiles_size = n_k_tiles > 0 ? (n_k_tiles * tile_stride) : tile_stride;- vtcm_layout->src0_spad_half_size = (row_tiles_size + 255) & ~255;- vtcm_layout->dst_spad_half_size = (ne00 * sizeof(float) + 255) & ~255;- vtcm_layout->src0_bytes_per_thread = 2 * vtcm_layout->src0_spad_half_size;- vtcm_layout->dst_bytes_per_thread = 2 * vtcm_layout->dst_spad_half_size;- vtcm_layout->off_src0 = 0;- vtcm_layout->off_dst = vtcm_layout->src0_bytes_per_thread * n_threads;- vtcm_layout->total_bytes = vtcm_layout->off_dst + vtcm_layout->dst_bytes_per_thread * n_threads;- return;- }-- uint32_t src0_row_size = 0;- switch (type) {- case HTP_TYPE_F16:- src0_row_size = ne00 * 2;- break;- case HTP_TYPE_Q8_0:- src0_row_size = (ne00 / 32) * 34;- break;- default:- src0_row_size = 0;- break;- }-- size_t src0_row_size_aligned = (src0_row_size + 255) & ~255;- size_t dst_row_size_aligned = (ne00 * sizeof(float) + 255) & ~255;-- vtcm_layout->src0_spad_half_size = src0_row_size_aligned;- vtcm_layout->dst_spad_half_size = dst_row_size_aligned;-- vtcm_layout->src0_bytes_per_thread = src0_row_size_aligned * 2;- vtcm_layout->dst_bytes_per_thread = dst_row_size_aligned * 2;-- vtcm_layout->off_src0 = 0;- vtcm_layout->off_dst = vtcm_layout->off_src0 + vtcm_layout->src0_bytes_per_thread * n_threads;- vtcm_layout->total_bytes = vtcm_layout->off_dst + vtcm_layout->dst_bytes_per_thread * n_threads;+ size_t off = 0;+ VTCM_LAYOUT_ALLOC(off, off_src1, src1_alloc_bytes);+ VTCM_LAYOUT_ALLOC(off, off_src0, L->src0_bytes_per_thread * n_threads);+ VTCM_LAYOUT_ALLOC(off, off_dst, L->dst_bytes_per_thread * n_threads);+ L->total_bytes = off;}#if defined(__cplusplus)diff --git a/ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h b/ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.hindex f0a0d9de19..309ec48777 100644--- a/ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h+++ b/ggml/src/ggml-hexagon/htp/hmx-mm-kernels-tiled.h@@ -14,11 +14,15 @@ static const __fp16 iq4_nl_to_fp16_lut[64] __attribute__((aligned(VLEN))) = {// --- tiled format dequantizers ----typedef struct {+typedef struct tiled_dequantize_state tiled_dequantize_state_t;+typedef void (*tiled_dequant_task_fn_t)(const tiled_dequantize_state_t * state, uint32_t start_tile, uint32_t end_tile);++struct tiled_dequantize_state {struct htp_context * ctx;struct htp_thread_trace * traces;__fp16 * dst;const uint8_t * src;+ tiled_dequant_task_fn_t task_fn;struct fastdiv_values n_k_tiles_div;uint32_t n_k_tiles;@@ -31,7 +35,7 @@ typedef struct {uint32_t k_block;size_t row_stride;uint32_t weight_type;-} tiled_dequantize_state_t;+};// Dequantize a single tile from tiled weight data (already in VTCM) to tile-major FP16.static void dequantize_tiled_weight_to_fp16_task_q4_0(@@ -611,7 +615,7 @@ static void dequantize_tiled_weight_to_fp16_task_q5_k(}}-// Q6_K stores 6-bit weights and one fp16 scale per 16 k, see HTP_MM_WEIGHT_TILE_SIZE_Q6_K.+// Q6_K stores 6-bit weights and one fp16 scale per 16 k, see HTP_TENSOR_TILE_SIZE_Q6_K.// A k-group holds 4 k per row, the HMX tile holds 2, so each group is dealt into two tiles.static void dequantize_tiled_weight_to_fp16_task_q6_k(const tiled_dequantize_state_t *state,@@ -669,7 +673,7 @@ static void dequantize_tiled_weight_to_fp16_task_q6_k(}}-// Q3_K stores 3-bit weights and one fp16 scale per 16 k, see HTP_MM_WEIGHT_TILE_SIZE_Q3_K.+// Q3_K stores 3-bit weights and one fp16 scale per 16 k, see HTP_TENSOR_TILE_SIZE_Q3_K.static void dequantize_tiled_weight_to_fp16_task_q3_k(const tiled_dequantize_state_t *state,uint32_t start_tile, uint32_t end_tile) {@@ -701,7 +705,7 @@ static void dequantize_tiled_weight_to_fp16_task_q3_k(}}-// Q2_K stores 2-bit weights, the ggml scale bytes and d / -dmin per super-block, see HTP_MM_WEIGHT_TILE_SIZE_Q2_K.+// Q2_K stores 2-bit weights, the ggml scale bytes and d / -dmin per super-block, see HTP_TENSOR_TILE_SIZE_Q2_K.static void dequantize_tiled_weight_to_fp16_task_q2_k(const tiled_dequantize_state_t *state,uint32_t start_tile, uint32_t end_tile) {@@ -837,6 +841,56 @@ void quantize_f32_weight_to_fp16_tiles_task(}}+static __attribute__((noinline))+void convert_bf16_weight_to_fp16_tiles_task(+ const tiled_dequantize_state_t *state,+ uint32_t start_tile, uint32_t end_tile) {++ const uint32_t n_k_tiles = state->n_k_tiles;+ const struct fastdiv_values n_k_tiles_div = state->n_k_tiles_div;++ const HVX_Vector v_scat_base = hvx_vmem(hmx_transpose_scatter_offsets);+ const HVX_Vector v_scat_step = Q6_V_vsplat_R(4);+ const HVX_VectorPred q_mask64 = Q6_Q_vsetq_R(64);++ unsigned ct = fastdiv((unsigned)start_tile, &n_k_tiles_div);+ unsigned kt = fastmodulo((unsigned)start_tile, n_k_tiles, &n_k_tiles_div);++ for (unsigned t = start_tile; t < (unsigned)end_tile; ) {+ if (kt >= (unsigned)n_k_tiles) { kt = 0; ct++; }++ __fp16 *tile_base = state->dst + t * HTP_MM_HMX_TILE_N_ELMS;+ {+ uint32_t byte_off = kt * 32 * sizeof(uint16_t);++ HVX_Vector v_off = v_scat_base;+ for (uint32_t r = 0; r < HTP_MM_HMX_TILE_N_ROWS; r += 2) {+ uint32_t row0 = ct * HTP_MM_HMX_TILE_N_COLS + r;+ uint32_t row1 = row0 + 1;++ const uint8_t *r0 = state->src + row0 * state->row_stride;+ const uint8_t *r1 = state->src + row1 * state->row_stride;++ HVX_Vector v0_raw = (row0 < state->n_cols) ? hvx_vmemu((const void *)(r0 + byte_off)) : Q6_V_vzero();+ HVX_Vector v1_raw = (row1 < state->n_cols) ? hvx_vmemu((const void *)(r1 + byte_off)) : Q6_V_vzero();++ HVX_Vector v0_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(Q6_Ww_vunpack_Vh(v0_raw)), 16);+ HVX_Vector v1_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(Q6_Ww_vunpack_Vh(v1_raw)), 16);++ HVX_Vector v_out = hvx_vec_f32_to_f16(v0_f32, v1_f32);++ Q6_vscatter_QRMVwV(q_mask64, (size_t)tile_base, HTP_MM_HMX_TILE_SIZE - 1, v_off, v_out);+ v_off = Q6_Vw_vadd_VwVw(v_off, v_scat_step);++ HVX_Vector v_out_hi = Q6_V_vror_VR(v_out, 64);+ Q6_vscatter_QRMVwV(q_mask64, (size_t)tile_base, HTP_MM_HMX_TILE_SIZE - 1, v_off, v_out_hi);+ v_off = Q6_Vw_vadd_VwVw(v_off, v_scat_step);+ }+ }+ ++t; ++kt;+ }+}+// --- End tiled dequantizers ---// dot-chunk functions require external HMX lockdiff --git a/ggml/src/ggml-hexagon/htp/htp-ctx.h b/ggml/src/ggml-hexagon/htp/htp-ctx.hindex b36322ae81..22a05b9179 100644--- a/ggml/src/ggml-hexagon/htp/htp-ctx.h+++ b/ggml/src/ggml-hexagon/htp/htp-ctx.h@@ -3,6 +3,7 @@#include "dma-queue.h"#include "hmx-queue.h"+#include "htp-dcache.h"#include "htp-ops.h"#include "hex-profile.h"#include "work-queue.h"@@ -18,8 +19,7 @@#define HTP_MAX_NTHREADS 10#endif-#define HTP_MAX_MMAPS 64-#define HTP_MAX_DIRTY_RANGES 64+#define HTP_MAX_MMAPS 64// Memory mappingstruct htp_mmap {@@ -29,11 +29,6 @@ struct htp_mmap {uint32_t flags;};-struct htp_dirty_range {- uint32_t start;- uint32_t end;-};-// Scratchpad statestruct htp_spad {const struct htp_tensor * src; // original src of the data (for reuse)diff --git a/ggml/src/ggml-hexagon/htp/htp-tensor.c b/ggml/src/ggml-hexagon/htp/htp-dcache.csimilarity index 97%rename from ggml/src/ggml-hexagon/htp/htp-tensor.crename to ggml/src/ggml-hexagon/htp/htp-dcache.cindex 03b0070d78..840b1edb7a 100644--- a/ggml/src/ggml-hexagon/htp/htp-tensor.c+++ b/ggml/src/ggml-hexagon/htp/htp-dcache.c@@ -1,4 +1,4 @@-#include "htp-tensor.h"+#include "htp-dcache.h"#include <qurt.h>#include <qurt_memory.h>@@ -94,7 +94,7 @@ static void merge_dirty_ranges(struct htp_context * ctx) {}}-void htp_tensor_dirty_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n) {+void htp_dcache_mark_dirty_tensors(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n) {const struct htp_tensor * pending[HTP_OP_MAX_OUTPUTS];uint32_t n_pending = 0;@@ -301,7 +301,7 @@ static void flush_dirty_ranges(struct htp_context * ctx, const struct htp_dirty_}}-void htp_flush_dirty_ranges(struct htp_context * ctx) {+void htp_dcache_flush_dirty_ranges(struct htp_context * ctx) {struct htp_dirty_range ranges[HTP_MAX_DIRTY_RANGES];uint32_t n_ranges = 0;uint64_t total_dirty = 0;@@ -328,7 +328,7 @@ void htp_flush_dirty_ranges(struct htp_context * ctx) {memset(ctx->dirty_ranges, 0, sizeof(ctx->dirty_ranges));}-void htp_tensor_flush_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n) {+void htp_dcache_flush_dirty_tensors(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n) {const struct htp_tensor * dirty_tensors[HTP_OP_MAX_INPUTS];struct htp_dirty_range ranges[HTP_OP_MAX_INPUTS];uint32_t n_dirty = 0;diff --git a/ggml/src/ggml-hexagon/htp/htp-dcache.h b/ggml/src/ggml-hexagon/htp/htp-dcache.hnew file mode 100644index 0000000000..1996ab258d--- /dev/null+++ b/ggml/src/ggml-hexagon/htp/htp-dcache.h@@ -0,0 +1,20 @@+#ifndef HTP_DCACHE_H+#define HTP_DCACHE_H++#include <stdint.h>+#include "htp-ops.h"++#define HTP_MAX_DIRTY_RANGES 64++struct htp_dirty_range {+ uint32_t start;+ uint32_t end;+};++struct htp_context;++void htp_dcache_flush_dirty_ranges(struct htp_context * ctx);+void htp_dcache_flush_dirty_tensors(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n);+void htp_dcache_mark_dirty_tensors(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n);++#endif // HTP_DCACHE_Hdiff --git a/ggml/src/ggml-hexagon/htp/htp-ops.h b/ggml/src/ggml-hexagon/htp/htp-ops.hindex 70d56e181a..bb358d09ad 100644--- a/ggml/src/ggml-hexagon/htp/htp-ops.h+++ b/ggml/src/ggml-hexagon/htp/htp-ops.h@@ -30,21 +30,14 @@ enum htp_data_type {HTP_TYPE_IQ4_NL = 20,HTP_TYPE_I32 = 26,HTP_TYPE_I64 = 27,+ HTP_TYPE_BF16 = 30,HTP_TYPE_MXFP4 = 39,- // types used internally for repack, dyn.quant, etc- HTP_TYPE_Q4_0_TILED = 200,- HTP_TYPE_Q4_1_TILED,- HTP_TYPE_Q8_0_TILED,- HTP_TYPE_MXFP4_TILED,-HTP_TYPE_INVALID};-// Constats for internal types-#define QK_Q4_0_TILED 256 // 32x32 Q4_0 tiled layout+// Constants for internal types#define QK_Q8_0_TILED 128 // 32x32 Q8_0 tiled layout-#define QK_MXFP4_TILED 256 // 32x32 MXFP4 tiled layout@@ -141,7 +134,7 @@ enum htp_op_code {enum htp_tensor_flags {HTP_TENSOR_WEIGHT = (1U << 0), // Tensor buffer model weight data (not compute)- HTP_TENSOR_REPACK = (1U << 1), // Tensor is in repacked tiled format+ HTP_TENSOR_TILED = (1U << 1), // Tensor is in tiled formatHTP_TENSOR_FENCE = (1U << 2) // Tensor is synchronization fence (explicitly managed)};diff --git a/ggml/src/ggml-hexagon/htp/htp-tensor.h b/ggml/src/ggml-hexagon/htp/htp-tensor.hindex f7a9668361..0bce68dd06 100644--- a/ggml/src/ggml-hexagon/htp/htp-tensor.h+++ b/ggml/src/ggml-hexagon/htp/htp-tensor.h@@ -8,27 +8,59 @@#include "hex-common.h"#include "hex-fastdiv.h"-enum {- HTP_TENSOR_MDEV_LINE_SIZE = 128,-};+#define HTP_TENSOR_MDEV_LINE_SIZE 128++// Currently HMX requires 32x32 tile formats (do not change)+#define HTP_TENSOR_TILE_N_COLS 32+#define HTP_TENSOR_TILE_N_ROWS 32++#define HTP_TENSOR_TILE_SIZE_F16 2048+#define HTP_TENSOR_TILE_SIZE_Q4_0 576+#define HTP_TENSOR_TILE_SIZE_Q4_1 640+#define HTP_TENSOR_TILE_SIZE_Q8_0 1088+#define HTP_TENSOR_TILE_SIZE_IQ4_NL 576+#define HTP_TENSOR_TILE_SIZE_MXFP4 544+#define HTP_TENSOR_TILE_SIZE_Q4_K 640+#define HTP_TENSOR_TILE_SIZE_Q5_K 768+#define HTP_TENSOR_TILE_SIZE_Q6_K 896+#define HTP_TENSOR_TILE_SIZE_Q3_K 512+#define HTP_TENSOR_TILE_SIZE_Q2_K 336++// Q2_K, Q3_K, Q6_K use vrmpy byte order within each 128B HVX vector:+// k = 0..31 is split into 8 groups g = k/4 of 4 k each; byte 4*r + (k%4) holds row r+//+// Q2_K (336B, lossless tiling with same size as block_q2_K):+// bytes 0..255 (vec 0..1): unsigned 2-bit quants, vec m holds groups 4m..4m+3 at bit offsets 0,2,4,6+// bytes 256..319: raw scales[] bytes (min << 4 | sc): [0..31] for k 0..15, [32..63] for k 16..31+// bytes 320..335: 8 fp16 in tile j (0..7) of super-block: d (j < 4) or -dmin (j >= 4) for rows 8*(j%4)..8*(j%4)+7+//+// Q3_K (512B):+// bytes 0..255 (vec 0..1): low 2 bits, vec m holds groups 4m..4m+3 at bit offsets 0,2,4,6+// bytes 256..383 (vec 2): sign mask, bit g set when hmask is 0 (quant = low2 - 4)+// bytes 384..511 (vec 3): fp16 scales d * (scales[] - 32): fp16[0..31] for k 0..15, fp16[32..63] for k 16..31+//+// Q4_K (640B, same layout as Q4_1 tile; each 256-k super-block splits into 8 tiles of 32 k):+// bytes 0..511 (vec 0..3): 4-bit quants, byte 32*cp + r holds k = 2*cp (low nibble) and 2*cp + 1 (high nibble)+// bytes 512..639 (vec 4): 32x2 fp16 per row: (d * sc, -dmin * m)+//+// Q5_K (768B):+// bytes 0..639 (vec 0..4): Q4_K tile (low 4 bits + fp16 scale/min pairs)+// bytes 640..767 (vec 5): 5th-bit plane; byte l holds the 8 high bits for lane l across vec i = 0..3 (bit 2i = lo, 2i+1 = hi)+//+// Q6_K (896B):+// bytes 0..511 (vec 0..3): low 4 bits, vec i holds group 2i (low nibble) and 2i+1 (high nibble)+// bytes 512..767 (vec 4..5): high 2 bits, vec 4+m holds groups 4m..4m+3 at bit offsets 0,2,4,6+// bytes 768..895 (vec 6): fp16 scales d * scales[]: fp16[0..31] for k 0..15, fp16[32..63] for k 16..31struct htp_tensor_mdev_range {uint32_t start;uint32_t count;};-static inline void * htp_tensor_data(const struct htp_tensor * t) {- return (void *) (uintptr_t) t->data;-}-static inline bool htp_tensor_is_extended(const struct htp_tensor * t) {return t && (t->data >> 32) != 0;}-static inline uint32_t * htp_tensor_flags(const struct htp_tensor * t) {- return (uint32_t *) &t->flags;-}-static inline bool htp_tensor_is_contiguous(const struct htp_tensor * t, uint32_t type_size) {uint32_t next_nb = type_size;if (t->ne[0] != 1 && t->nb[0] != next_nb) {@@ -125,19 +157,50 @@ static inline struct htp_tensor_mdev_range htp_tensor_mdev_partition(uint32_t toreturn range;}-static inline uint32_t htp_tensor_get_row_size(int type, uint32_t ne00) {+static inline uint32_t htp_tensor_get_tile_size(int type) {switch (type) {- case HTP_TYPE_F32: return ne00 * 4;- case HTP_TYPE_F16: return ne00 * 2;- case HTP_TYPE_Q8_0: return (ne00 / 32) * 34;- case HTP_TYPE_I32: return ne00 * 4;- default: return 0;+ case HTP_TYPE_Q4_0: return HTP_TENSOR_TILE_SIZE_Q4_0;+ case HTP_TYPE_IQ4_NL: return HTP_TENSOR_TILE_SIZE_IQ4_NL;+ case HTP_TYPE_Q4_1: return HTP_TENSOR_TILE_SIZE_Q4_1;+ case HTP_TYPE_Q4_K: return HTP_TENSOR_TILE_SIZE_Q4_K;+ case HTP_TYPE_Q8_0: return HTP_TENSOR_TILE_SIZE_Q8_0;+ case HTP_TYPE_Q5_K: return HTP_TENSOR_TILE_SIZE_Q5_K;+ case HTP_TYPE_Q6_K: return HTP_TENSOR_TILE_SIZE_Q6_K;+ case HTP_TYPE_Q3_K: return HTP_TENSOR_TILE_SIZE_Q3_K;+ case HTP_TYPE_Q2_K: return HTP_TENSOR_TILE_SIZE_Q2_K;+ case HTP_TYPE_MXFP4: return HTP_TENSOR_TILE_SIZE_MXFP4;+ case HTP_TYPE_F16: return HTP_TENSOR_TILE_SIZE_F16;+ default: return 0;}}-struct htp_context;-void htp_flush_dirty_ranges(struct htp_context * ctx);-void htp_tensor_flush_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n);-void htp_tensor_dirty_all(struct htp_context * ctx, const struct htp_tensor * const * tensors, uint32_t n);+static inline uint32_t htp_tensor_get_tile_padded_size(int type) {+ return (htp_tensor_get_tile_size(type) + 127u) & ~127u;+}++static inline bool htp_tensor_is_tiled(const struct htp_tensor * t) {+ return t && (t->flags & HTP_TENSOR_TILED) != 0;+}++static inline uint32_t htp_tensor_get_row_size(int type, uint32_t ne00) {+ switch (type) {+ case HTP_TYPE_F32: return ne00 * 4;+ case HTP_TYPE_F16:+ case HTP_TYPE_BF16: return ne00 * 2;+ case HTP_TYPE_Q8_0: return (ne00 / 32) * 34;+ case HTP_TYPE_Q4_0:+ case HTP_TYPE_IQ4_NL: return (ne00 / 32) * 18;+ case HTP_TYPE_Q4_1: return (ne00 / 32) * 20;+ case HTP_TYPE_MXFP4: return (ne00 / 32) * 17;+ case HTP_TYPE_Q2_K: return (ne00 / 256) * 84;+ case HTP_TYPE_Q3_K: return (ne00 / 256) * 110;+ case HTP_TYPE_Q4_K: return (ne00 / 256) * 144;+ case HTP_TYPE_Q5_K: return (ne00 / 256) * 176;+ case HTP_TYPE_Q6_K: return (ne00 / 256) * 210;+ case HTP_TYPE_I32: return ne00 * 4;+ case HTP_TYPE_I64: return ne00 * 8;+ default: return 0;+ }+}#endif // HTP_TENSOR_Hdiff --git a/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-float.h b/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-float.hindex 605892aa77..d7488976bb 100644--- a/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-float.h+++ b/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-float.h@@ -76,7 +76,7 @@ static inline void quantize_f16_f16_kernel(#define HVX_OP_MUL_F32(a, b) Q6_Vsf_vmpy_VsfVsf(a, b)#endif-static inline void vec_dot_f32_f32_aa_1x1(const uint32_t n, float * restrict s, const void * restrict vx, const void * restrict vy) {+static inline void flat_vec_dot_f32_f32_aa_1x1(const uint32_t n, float * restrict s, const void * restrict vx, const void * restrict vy) {const HVX_Vector * restrict x = (const HVX_Vector *) vx;const HVX_Vector * restrict y = (const HVX_Vector *) vy;@@ -104,7 +104,7 @@ static inline void vec_dot_f32_f32_aa_1x1(const uint32_t n, float * restrict s,*s = hvx_vec_get_f32(hvx_vec_reduce_sum_f32(rsum));}-static inline void vec_dot_f32_f32_aa_2x1(const uint32_t n, float * restrict s0,+static inline void flat_vec_dot_f32_f32_aa_2x1(const uint32_t n, float * restrict s0,const void * restrict vx0, const void * restrict vx1,const void * restrict vy0) {const HVX_Vector * restrict x0 = (const HVX_Vector *) vx0;@@ -143,7 +143,7 @@ static inline void vec_dot_f32_f32_aa_2x1(const uint32_t n, float * restrict s0,hvx_vec_store_u(s0, 8, rsum);}-static inline void vec_dot_f32_f32_aa_2x2(const uint32_t n, float * restrict s0, float * restrict s1,+static inline void flat_vec_dot_f32_f32_aa_2x2(const uint32_t n, float * restrict s0, float * restrict s1,const void * restrict vx0, const void * restrict vx1,const void * restrict vy0, const void * restrict vy1) {const HVX_Vector * restrict x0 = (const HVX_Vector *) vx0;@@ -196,10 +196,203 @@ static inline void vec_dot_f32_f32_aa_2x2(const uint32_t n, float * restrict s0,hvx_vec_store_u(s1, 8, r0_r1_c1_sum);}+static inline void flat_vec_dot_bf16_f32_aa_1x1(const uint32_t n, float * restrict s, const void * restrict vx, const void * restrict vy) {+ const HVX_Vector * restrict x = (const HVX_Vector *) vx;+ const HVX_Vector * restrict y = (const HVX_Vector *) vy;++ uint32_t nvec = n / 64;+ uint32_t rem = n % 64;++ HVX_Vector rsum0 = Q6_V_vzero();+ HVX_Vector rsum1 = Q6_V_vzero();++ uint32_t i = 0;+ #pragma unroll(2)+ for (i = 0; i < nvec; i++) {+ HVX_Vector v_bf16 = x[i];+ HVX_VectorPair p = Q6_Ww_vunpack_Vh(v_bf16);+ HVX_Vector w0_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p), 16);+ HVX_Vector w1_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p), 16);++ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w0_f32, y[2 * i + 0]));+ rsum1 = HVX_OP_ADD_F32(rsum1, HVX_OP_MUL_F32(w1_f32, y[2 * i + 1]));+ }++ if (rem) {+ HVX_VectorPred bmask = Q6_Q_vsetq2_R(rem * 2);+ HVX_Vector v_bf16 = Q6_V_vand_QV(bmask, x[i]);+ HVX_VectorPair p = Q6_Ww_vunpack_Vh(v_bf16);+ HVX_Vector w0_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p), 16);+ HVX_Vector w1_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p), 16);++ if (rem <= 32) {+ HVX_VectorPred a_mask = Q6_Q_vsetq2_R(rem * 4);+ HVX_Vector y0 = Q6_V_vand_QV(a_mask, y[2 * i + 0]);+ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w0_f32, y0));+ } else {+ HVX_Vector y0 = y[2 * i + 0];+ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w0_f32, y0));++ HVX_VectorPred a_mask = Q6_Q_vsetq2_R((rem - 32) * 4);+ HVX_Vector y1 = Q6_V_vand_QV(a_mask, y[2 * i + 1]);+ rsum1 = HVX_OP_ADD_F32(rsum1, HVX_OP_MUL_F32(w1_f32, y1));+ }+ }++ HVX_Vector rsum = HVX_OP_ADD_F32(rsum0, rsum1);+ *s = hvx_vec_get_f32(hvx_vec_reduce_sum_f32(rsum));+}++static inline void flat_vec_dot_bf16_f32_aa_2x1(const uint32_t n, float * restrict s0,+ const void * restrict vx0, const void * restrict vx1,+ const void * restrict vy0) {+ const HVX_Vector * restrict x0 = (const HVX_Vector *) vx0;+ const HVX_Vector * restrict x1 = (const HVX_Vector *) vx1;+ const HVX_Vector * restrict y = (const HVX_Vector *) vy0;++ uint32_t nvec = n / 64;+ uint32_t rem = n % 64;++ HVX_Vector rsum0 = Q6_V_vzero();+ HVX_Vector rsum1 = Q6_V_vzero();++ uint32_t i = 0;+ #pragma unroll(2)+ for (i = 0; i < nvec; i++) {+ HVX_VectorPair p0 = Q6_Ww_vunpack_Vh(x0[i]);+ HVX_VectorPair p1 = Q6_Ww_vunpack_Vh(x1[i]);+ HVX_Vector w00_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p0), 16);+ HVX_Vector w01_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p0), 16);+ HVX_Vector w10_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p1), 16);+ HVX_Vector w11_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p1), 16);++ HVX_Vector y0 = y[2 * i + 0];+ HVX_Vector y1 = y[2 * i + 1];++ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w00_f32, y0));+ rsum1 = HVX_OP_ADD_F32(rsum1, HVX_OP_MUL_F32(w10_f32, y0));+ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w01_f32, y1));+ rsum1 = HVX_OP_ADD_F32(rsum1, HVX_OP_MUL_F32(w11_f32, y1));+ }++ if (rem) {+ HVX_VectorPred bmask = Q6_Q_vsetq2_R(rem * 2);+ HVX_VectorPair p0 = Q6_Ww_vunpack_Vh(Q6_V_vand_QV(bmask, x0[i]));+ HVX_VectorPair p1 = Q6_Ww_vunpack_Vh(Q6_V_vand_QV(bmask, x1[i]));+ HVX_Vector w00_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p0), 16);+ HVX_Vector w01_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p0), 16);+ HVX_Vector w10_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p1), 16);+ HVX_Vector w11_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p1), 16);++ if (rem <= 32) {+ HVX_VectorPred a_mask = Q6_Q_vsetq2_R(rem * 4);+ HVX_Vector y0 = Q6_V_vand_QV(a_mask, y[2 * i + 0]);+ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w00_f32, y0));+ rsum1 = HVX_OP_ADD_F32(rsum1, HVX_OP_MUL_F32(w10_f32, y0));+ } else {+ HVX_Vector y0 = y[2 * i + 0];+ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w00_f32, y0));+ rsum1 = HVX_OP_ADD_F32(rsum1, HVX_OP_MUL_F32(w10_f32, y0));++ HVX_VectorPred a_mask = Q6_Q_vsetq2_R((rem - 32) * 4);+ HVX_Vector y1 = Q6_V_vand_QV(a_mask, y[2 * i + 1]);+ rsum0 = HVX_OP_ADD_F32(rsum0, HVX_OP_MUL_F32(w01_f32, y1));+ rsum1 = HVX_OP_ADD_F32(rsum1, HVX_OP_MUL_F32(w11_f32, y1));+ }+ }++ HVX_Vector rsum = hvx_vec_reduce_sum_f32x2(rsum0, rsum1);+ hvx_vec_store_u(s0, 8, rsum);+}++static inline void flat_vec_dot_bf16_f32_aa_2x2(const uint32_t n, float * restrict s0, float * restrict s1,+ const void * restrict vx0, const void * restrict vx1,+ const void * restrict vy0, const void * restrict vy1) {+ const HVX_Vector * restrict x0 = (const HVX_Vector *) vx0;+ const HVX_Vector * restrict x1 = (const HVX_Vector *) vx1;+ const HVX_Vector * restrict y0 = (const HVX_Vector *) vy0;+ const HVX_Vector * restrict y1 = (const HVX_Vector *) vy1;++ uint32_t nvec = n / 64;+ uint32_t rem = n % 64;++ HVX_Vector r0_c0_sum = Q6_V_vzero();+ HVX_Vector r0_c1_sum = Q6_V_vzero();+ HVX_Vector r1_c0_sum = Q6_V_vzero();+ HVX_Vector r1_c1_sum = Q6_V_vzero();++ uint32_t i = 0;+ #pragma unroll(2)+ for (i = 0; i < nvec; i++) {+ HVX_VectorPair p0 = Q6_Ww_vunpack_Vh(x0[i]);+ HVX_VectorPair p1 = Q6_Ww_vunpack_Vh(x1[i]);+ HVX_Vector w00_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p0), 16);+ HVX_Vector w01_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p0), 16);+ HVX_Vector w10_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p1), 16);+ HVX_Vector w11_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p1), 16);++ HVX_Vector c0_0 = y0[2 * i + 0];+ HVX_Vector c0_1 = y0[2 * i + 1];+ HVX_Vector c1_0 = y1[2 * i + 0];+ HVX_Vector c1_1 = y1[2 * i + 1];++ r0_c0_sum = HVX_OP_ADD_F32(r0_c0_sum, HVX_OP_MUL_F32(w00_f32, c0_0));+ r0_c1_sum = HVX_OP_ADD_F32(r0_c1_sum, HVX_OP_MUL_F32(w00_f32, c1_0));+ r1_c0_sum = HVX_OP_ADD_F32(r1_c0_sum, HVX_OP_MUL_F32(w10_f32, c0_0));+ r1_c1_sum = HVX_OP_ADD_F32(r1_c1_sum, HVX_OP_MUL_F32(w10_f32, c1_0));++ r0_c0_sum = HVX_OP_ADD_F32(r0_c0_sum, HVX_OP_MUL_F32(w01_f32, c0_1));+ r0_c1_sum = HVX_OP_ADD_F32(r0_c1_sum, HVX_OP_MUL_F32(w01_f32, c1_1));+ r1_c0_sum = HVX_OP_ADD_F32(r1_c0_sum, HVX_OP_MUL_F32(w11_f32, c0_1));+ r1_c1_sum = HVX_OP_ADD_F32(r1_c1_sum, HVX_OP_MUL_F32(w11_f32, c1_1));+ }++ if (rem) {+ HVX_VectorPred bmask = Q6_Q_vsetq2_R(rem * 2);+ HVX_VectorPair p0 = Q6_Ww_vunpack_Vh(Q6_V_vand_QV(bmask, x0[i]));+ HVX_VectorPair p1 = Q6_Ww_vunpack_Vh(Q6_V_vand_QV(bmask, x1[i]));+ HVX_Vector w00_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p0), 16);+ HVX_Vector w01_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p0), 16);+ HVX_Vector w10_f32 = Q6_Vw_vasl_VwR(Q6_V_lo_W(p1), 16);+ HVX_Vector w11_f32 = Q6_Vw_vasl_VwR(Q6_V_hi_W(p1), 16);++ if (rem <= 32) {+ HVX_VectorPred a_mask = Q6_Q_vsetq2_R(rem * 4);+ HVX_Vector c0_0 = Q6_V_vand_QV(a_mask, y0[2 * i + 0]);+ HVX_Vector c1_0 = Q6_V_vand_QV(a_mask, y1[2 * i + 0]);+ r0_c0_sum = HVX_OP_ADD_F32(r0_c0_sum, HVX_OP_MUL_F32(w00_f32, c0_0));+ r0_c1_sum = HVX_OP_ADD_F32(r0_c1_sum, HVX_OP_MUL_F32(w00_f32, c1_0));+ r1_c0_sum = HVX_OP_ADD_F32(r1_c0_sum, HVX_OP_MUL_F32(w10_f32, c0_0));+ r1_c1_sum = HVX_OP_ADD_F32(r1_c1_sum, HVX_OP_MUL_F32(w10_f32, c1_0));+ } else {+ HVX_Vector c0_0 = y0[2 * i + 0];+ HVX_Vector c1_0 = y1[2 * i + 0];+ r0_c0_sum = HVX_OP_ADD_F32(r0_c0_sum, HVX_OP_MUL_F32(w00_f32, c0_0));+ r0_c1_sum = HVX_OP_ADD_F32(r0_c1_sum, HVX_OP_MUL_F32(w00_f32, c1_0));+ r1_c0_sum = HVX_OP_ADD_F32(r1_c0_sum, HVX_OP_MUL_F32(w10_f32, c0_0));+ r1_c1_sum = HVX_OP_ADD_F32(r1_c1_sum, HVX_OP_MUL_F32(w10_f32, c1_0));++ HVX_VectorPred a_mask = Q6_Q_vsetq2_R((rem - 32) * 4);+ HVX_Vector c0_1 = Q6_V_vand_QV(a_mask, y0[2 * i + 1]);+ HVX_Vector c1_1 = Q6_V_vand_QV(a_mask, y1[2 * i + 1]);+ r0_c0_sum = HVX_OP_ADD_F32(r0_c0_sum, HVX_OP_MUL_F32(w01_f32, c0_1));+ r0_c1_sum = HVX_OP_ADD_F32(r0_c1_sum, HVX_OP_MUL_F32(w01_f32, c1_1));+ r1_c0_sum = HVX_OP_ADD_F32(r1_c0_sum, HVX_OP_MUL_F32(w11_f32, c0_1));+ r1_c1_sum = HVX_OP_ADD_F32(r1_c1_sum, HVX_OP_MUL_F32(w11_f32, c1_1));+ }+ }++ HVX_Vector r0_r1_c0_sum = hvx_vec_reduce_sum_f32x2(r0_c0_sum, r1_c0_sum);+ HVX_Vector r0_r1_c1_sum = hvx_vec_reduce_sum_f32x2(r0_c1_sum, r1_c1_sum);++ hvx_vec_store_u(s0, 8, r0_r1_c0_sum);+ hvx_vec_store_u(s1, 8, r0_r1_c1_sum);+}+#undef HVX_OP_ADD_F32#undef HVX_OP_MUL_F32-static inline void vec_dot_f16_f16_aa_1x1(const uint32_t n, float * restrict s, const void * restrict vx, const void * restrict vy) {+static inline void flat_vec_dot_f16_f16_aa_1x1(const uint32_t n, float * restrict s, const void * restrict vx, const void * restrict vy) {const HVX_Vector * restrict x = (const HVX_Vector *) vx;const HVX_Vector * restrict y = (const HVX_Vector *) vy;@@ -226,7 +419,7 @@ static inline void vec_dot_f16_f16_aa_1x1(const uint32_t n, float * restrict s,hvx_vec_store_u(s, 4, hvx_vec_reduce_sum_f32(rsum));}-static inline void vec_dot_f16_f16_aa_2x1(const uint32_t n, float * restrict s0,+static inline void flat_vec_dot_f16_f16_aa_2x1(const uint32_t n, float * restrict s0,const void * restrict vx0, const void * restrict vx1,const void * restrict vy0) {const HVX_Vector * restrict x0 = (const HVX_Vector *) vx0;@@ -263,7 +456,7 @@ static inline void vec_dot_f16_f16_aa_2x1(const uint32_t n, float * restrict s0,hvx_vec_store_u(s0, 8, rsum);}-static inline void vec_dot_f16_f16_aa_2x2(const uint32_t n, float * restrict s0, float * restrict s1,+static inline void flat_vec_dot_f16_f16_aa_2x2(const uint32_t n, float * restrict s0, float * restrict s1,const void * restrict vx0, const void * restrict vx1,const void * restrict vy0, const void * restrict vy1) {const HVX_Vector * restrict x0 = (const HVX_Vector *) vx0;diff --git a/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-tiled.h b/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-tiled.hindex 0c86e407ef..91e5117255 100644--- a/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-tiled.h+++ b/ggml/src/ggml-hexagon/htp/hvx-mm-kernels-tiled.h@@ -274,6 +274,72 @@ static void quantize_row_f32_q8_1_s16_tiled(float * restrict x, uint8_t * restri}}+static inline void quantize_half_f16_tiled_splat(HVX_Vector v_src, HVX_Vector * restrict dst) {+ static const uint8_t __attribute__((aligned(128))) repl[128] = {+ 0x00, 0x00, 0x00, 0x00, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,+ 0x10, 0x10, 0x10, 0x10, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,+ 0x20, 0x20, 0x20, 0x20, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,+ 0x10, 0x10, 0x10, 0x10, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,+ 0x40, 0x40, 0x40, 0x40, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,+ 0x10, 0x10, 0x10, 0x10, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,+ 0x20, 0x20, 0x20, 0x20, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,+ 0x10, 0x10, 0x10, 0x10, 0x04, 0x04, 0x04, 0x04, 0x08, 0x08, 0x08, 0x08, 0x04, 0x04, 0x04, 0x04,+ };+ HVX_Vector v_repl_ctrl = *(const HVX_Vector *) repl;++ #pragma unroll(4)+ for (int cp = 0; cp < 32; cp++) {+ dst[cp] = Q6_V_vdelta_VV(cp == 0 ? v_src : Q6_V_vror_VR(v_src, cp * 4), v_repl_ctrl);+ }+}++static inline void quantize_block_f32_f16_tiled(const float * restrict x, uint8_t * restrict y_block) {+ assert((unsigned long) x % 128 == 0);+ assert((unsigned long) y_block % 128 == 0);++ const HVX_Vector * vx = (const HVX_Vector *) x;+ HVX_Vector * restrict dst = (HVX_Vector *) y_block;++ HVX_Vector vx01_hf = hvx_vec_f32_to_f16(vx[0], vx[1]);+ quantize_half_f16_tiled_splat(vx01_hf, dst);++ HVX_Vector vx23_hf = hvx_vec_f32_to_f16(vx[2], vx[3]);+ quantize_half_f16_tiled_splat(vx23_hf, dst + 32);+}++static inline void quantize_block_f16_f16_tiled(const __fp16 * restrict x, uint8_t * restrict y_block) {+ assert((unsigned long) x % 128 == 0);+ assert((unsigned long) y_block % 128 == 0);++ const HVX_Vector * vx = (const HVX_Vector *) x;+ HVX_Vector * restrict dst = (HVX_Vector *) y_block;++ quantize_half_f16_tiled_splat(vx[0], dst);+ quantize_half_f16_tiled_splat(vx[1], dst + 32);+}++static void quantize_row_f32_f16_tiled(const float * restrict x, uint8_t * restrict y, uint32_t k) {+ assert(k % 32 == 0);+ const uint32_t qk = QK_Q8_0_TILED;+ const uint32_t nb = (k + qk - 1) / qk;++ for (uint32_t i = 0; i < nb; i++) {+ uint8_t * restrict y_block = y + i * 4 * 2048;+ quantize_block_f32_f16_tiled(x + i * qk, y_block);+ }+}++static void quantize_row_f16_f16_tiled(const __fp16 * restrict x, uint8_t * restrict y, uint32_t k) {+ assert(k % 32 == 0);+ const uint32_t qk = QK_Q8_0_TILED;+ const uint32_t nb = (k + qk - 1) / qk;++ for (uint32_t i = 0; i < nb; i++) {+ uint8_t * restrict y_block = y + i * 4 * 2048;+ quantize_block_f16_f16_tiled(x + i * qk, y_block);+ }+}+// Dot kernels & helpers that consume tiled activationsstatic inline HVX_Vector hvx_vec_mul_f16_f16_to_f32_lower32(HVX_Vector v1, HVX_Vector v2) {@@ -454,7 +520,7 @@ static inline HVX_VectorPair accum_q8_0_32x2(return Q6_W_vcombine_VV(v_sum1, v_sum0);}-// Q5_K: OR 0x10 into every lane of v whose flag j is set in the plane (see HTP_MM_WEIGHT_TILE_SIZE_Q5_K)+// Q5_K: OR 0x10 into every lane of v whose flag j is set in the plane (see HTP_TENSOR_TILE_SIZE_Q5_K)static inline HVX_Vector hvx_q5k_or_hibit(HVX_Vector v, HVX_Vector v_plane, int j) {HVX_VectorPred q = Q6_Q_vand_VR(v_plane, 0x01010101u << j);return Q6_V_vandor_VQR(v, q, 0x10101010);@@ -526,7 +592,7 @@ static inline HVX_VectorPair accum_5bit_32x2(return Q6_W_vcombine_VV(v_sum1, v_sum0);}-// Q6_K weights are stored unsigned (0..63), see HTP_MM_WEIGHT_TILE_SIZE_Q6_K. Unpack k-group g of a tile to signed bytes (q - 32)+// Q6_K weights are stored unsigned (0..63), see HTP_TENSOR_TILE_SIZE_Q6_K. Unpack k-group g of a tile to signed bytes (q - 32)static inline HVX_Vector unpack_q6_k_group(const HVX_Vector * restrict vptr, int g, HVX_Vector mask_0f, HVX_Vector mask_03, HVX_Vector i32) {HVX_Vector v_lo = (g & 1) ? Q6_Vub_vlsr_VubR(vptr[g >> 1], 4) : Q6_V_vand_VV(vptr[g >> 1], mask_0f);HVX_Vector v_hi = (g & 3) ? Q6_Vub_vlsr_VubR(vptr[4 + (g >> 2)], 2 * (g & 3)) : vptr[4 + (g >> 2)];@@ -593,7 +659,7 @@ static inline HVX_Vector scale_q6_k_32x1(HVX_VectorPair v_sums, HVX_Vector v_scareturn hvx_vec_add_f32_f32(v_lo, v_hi);}-// Q3_K / Q2_K: low 2 bits of k-group g, see HTP_MM_WEIGHT_TILE_SIZE_Q3_K+// Q3_K / Q2_K: low 2 bits of k-group g, see HTP_TENSOR_TILE_SIZE_Q3_Kstatic inline HVX_Vector unpack_q3_k_low2(const HVX_Vector * restrict vptr, int g, HVX_Vector mask_03) {HVX_Vector v = vptr[g >> 2];if ((g & 3) == 3) {@@ -1482,6 +1548,59 @@ static void tiled_vec_dot_mxfp4_32x2(const uint32_t n, float * restrict s0, floa}}+static void tiled_vec_dot_f16_32x1(const uint32_t n, float * restrict s, const void * restrict vx, const void * restrict vy, uint32_t valid_rows, const float * restrict sz) {+ const HVX_Vector * restrict vptr = (const HVX_Vector *) vx;+ const HVX_Vector * restrict v_act = (const HVX_Vector *) vy;+ const uint32_t n_vecs = ((n + 31) / 32) * 16;++ HVX_VectorPair acc = Q6_W_vzero();++ #pragma unroll(4)+ for (uint32_t i = 0; i < n_vecs; i++) {+ acc = hvx_vec_mpyacc_f32_f16(acc, vptr[i], v_act[i]);+ }++ HVX_Vector v_sum = hvx_vec_add_f32_f32(Q6_V_lo_W(acc), Q6_V_hi_W(acc));++ if (sz) {+ hvx_vec_store_u(s, valid_rows * sizeof(float), hvx_vec_add_f32_f32(v_sum, hvx_vmemu(sz)));+ } else {+ hvx_vec_store_u(s, valid_rows * sizeof(float), v_sum);+ }+}++static void tiled_vec_dot_f16_32x2(const uint32_t n, float * restrict s0, float * restrict s1, const void * restrict vx, const void * restrict vy0, const void * restrict vy1, uint32_t valid_rows, const float * restrict sz0, const float * restrict sz1) {+ const HVX_Vector * restrict vptr = (const HVX_Vector *) vx;+ const HVX_Vector * restrict v_act0 = (const HVX_Vector *) vy0;+ const HVX_Vector * restrict v_act1 = (const HVX_Vector *) vy1;+ const uint32_t n_vecs = ((n + 31) / 32) * 16;++ HVX_VectorPair acc0 = Q6_W_vzero();+ HVX_VectorPair acc1 = Q6_W_vzero();++ #pragma unroll(4)+ for (uint32_t i = 0; i < n_vecs; i++) {+ HVX_Vector v_w = vptr[i];+ acc0 = hvx_vec_mpyacc_f32_f16(acc0, v_w, v_act0[i]);+ acc1 = hvx_vec_mpyacc_f32_f16(acc1, v_w, v_act1[i]);+ }++ HVX_Vector v_sum0 = hvx_vec_add_f32_f32(Q6_V_lo_W(acc0), Q6_V_hi_W(acc0));+ HVX_Vector v_sum1 = hvx_vec_add_f32_f32(Q6_V_lo_W(acc1), Q6_V_hi_W(acc1));++ if (sz0) {+ hvx_vec_store_u(s0, valid_rows * sizeof(float), hvx_vec_add_f32_f32(v_sum0, hvx_vmemu(sz0)));+ } else {+ hvx_vec_store_u(s0, valid_rows * sizeof(float), v_sum0);+ }+ if (sz1) {+ hvx_vec_store_u(s1, valid_rows * sizeof(float), hvx_vec_add_f32_f32(v_sum1, hvx_vmemu(sz1)));+ } else {+ hvx_vec_store_u(s1, valid_rows * sizeof(float), v_sum1);+ }+}++static inline void quantize_f32_q8_0_tiled_kernel(const uint8_t * restrict src_data,uint8_t * restrict dst_data,@@ -1533,6 +1652,40 @@ static inline void quantize_f32_q8_1_s16_tiled_kernel(}}+static inline void quantize_f32_f16_tiled_kernel(+ const uint8_t * restrict src_data,+ uint8_t * restrict dst_data,+ uint8_t * restrict tmp_data,+ uint32_t ne0,+ uint32_t nrows,+ size_t src_row_size,+ size_t dst_row_size+) {+ (void) tmp_data;+ for (uint32_t i = 0; i < nrows; ++i) {+ quantize_row_f32_f16_tiled((const float *) src_data, dst_data, ne0);+ dst_data += dst_row_size;+ src_data += src_row_size;+ }+}++static inline void quantize_f16_f16_tiled_kernel(+ const uint8_t * restrict src_data,+ uint8_t * restrict dst_data,+ uint8_t * restrict tmp_data,+ uint32_t ne0,+ uint32_t nrows,+ size_t src_row_size,+ size_t dst_row_size+) {+ (void) tmp_data;+ for (uint32_t i = 0; i < nrows; ++i) {+ quantize_row_f16_f16_tiled((const __fp16 *) src_data, dst_data, ne0);+ dst_data += dst_row_size;+ src_data += src_row_size;+ }+}+static inline void quantize_f32_q8_0_tiled_block_kernel(const float * restrict src,uint8_t * restrict dst,@@ -1622,3 +1775,63 @@ static inline void quantize_f32_q8_1_s16_tiled_block_kernel(}}}++static inline void quantize_f32_f16_tiled_block_kernel(+ const float * restrict src,+ uint8_t * restrict dst,+ uint8_t * restrict tmp_data,+ uint32_t ne0,+ uint32_t ib_first,+ uint32_t ib_last,+ size_t src_row_size,+ size_t dst_row_size,+ uint32_t r,+ uint32_t c+) {+ (void) tmp_data;+ const uint32_t qk = QK_Q8_0_TILED;+ const uint32_t nb = (ne0 + qk - 1) / qk;++ for (uint32_t ib = ib_first; ib < ib_last; ++ib) {+ const float * restrict src_ptr = (const float *) ((const uint8_t *) src + r * src_row_size + c * qk * sizeof(float));+ uint8_t * restrict dst_ptr = dst + r * dst_row_size + c * 4 * 2048;++ quantize_block_f32_f16_tiled(src_ptr, dst_ptr);++ c++;+ if (c == nb) {+ c = 0;+ r++;+ }+ }+}++static inline void quantize_f16_f16_tiled_block_kernel(+ const __fp16 * restrict src,+ uint8_t * restrict dst,+ uint8_t * restrict tmp_data,+ uint32_t ne0,+ uint32_t ib_first,+ uint32_t ib_last,+ size_t src_row_size,+ size_t dst_row_size,+ uint32_t r,+ uint32_t c+) {+ (void) tmp_data;+ const uint32_t qk = QK_Q8_0_TILED;+ const uint32_t nb = (ne0 + qk - 1) / qk;++ for (uint32_t ib = ib_first; ib < ib_last; ++ib) {+ const __fp16 * restrict src_ptr = (const __fp16 *) ((const uint8_t *) src + r * src_row_size + c * qk * sizeof(__fp16));+ uint8_t * restrict dst_ptr = dst + r * dst_row_size + c * 4 * 2048;++ quantize_block_f16_f16_tiled(src_ptr, dst_ptr);++ c++;+ if (c == nb) {+ c = 0;+ r++;+ }+ }+}diff --git a/ggml/src/ggml-hexagon/htp/hvx-quant.h b/ggml/src/ggml-hexagon/htp/hvx-quant.hindex 6b172cd63c..b7837967a8 100644--- a/ggml/src/ggml-hexagon/htp/hvx-quant.h+++ b/ggml/src/ggml-hexagon/htp/hvx-quant.h@@ -5,6 +5,7 @@#include <stdint.h>#include <string.h>+#include "htp-tensor.h"#include "hvx-arith.h"#include "hvx-base.h"#include "hvx-reduce.h"@@ -96,7 +97,7 @@ static inline void hvx_quantize_row_q8_0_f32(void * restrict dst_ptr, const floa}}-static inline void hvx_dequantize_row_q8_0_f32(float * restrict dst_ptr, const void * restrict src_ptr, int n) {+static inline void hvx_dequantize_row_flat_q8_0_f32(float * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {const int nb = n / QK8_0;const block_q8_0 * src = (const block_q8_0 *) src_ptr;@@ -120,7 +121,7 @@ static inline void hvx_dequantize_row_q8_0_f32(float * restrict dst_ptr, const v}}-static inline void hvx_dequantize_row_q8_0_f16(__fp16 * restrict dst_ptr, const void * restrict src_ptr, int n) {+static inline void hvx_dequantize_row_flat_q8_0_f16(__fp16 * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {const int nb = n / QK8_0;const block_q8_0 * src = (const block_q8_0 *) src_ptr;@@ -146,20 +147,460 @@ static inline void hvx_dequantize_row_q8_0_f16(__fp16 * restrict dst_ptr, const}}-static inline void hvx_dequantize_row_f16_f32(float * restrict dst_ptr, const void * restrict src_ptr, int n) {- const int nb = n / 32;+static inline void hvx_dequantize_row_flat_f16_f32(float * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {+ const uint32_t nvec = n / VLEN_FP32;+ const uint32_t nloe = n % VLEN_FP32;const _Float16 * src = (const _Float16 *) src_ptr;- for (int i = 0; i < nb; i++) {- HVX_Vector v_f16 = *(const HVX_UVector *) (src + i * 32);+ for (uint32_t i = 0; i < nvec; i++) {+ HVX_Vector v_f16 = *(const HVX_UVector *) (src + i * VLEN_FP32);HVX_VectorPair vp_f32 = hvx_vec_f16_to_f32(v_f16);HVX_Vector res = Q6_V_lo_W(vp_f32);- float * block_dst = dst_ptr + i * 32;- hvx_vmem(block_dst) = res;+ hvx_vmem(dst_ptr + i * VLEN_FP32) = res;+ }++ if (nloe) {+ HVX_Vector v_f16 = *(const HVX_UVector *) (src + nvec * VLEN_FP32);+ HVX_VectorPair vp_f32 = hvx_vec_f16_to_f32(v_f16);+ HVX_Vector res = Q6_V_lo_W(vp_f32);++ hvx_vec_store_a(dst_ptr + nvec * VLEN_FP32, nloe * sizeof(float), res);+ }+}++static inline void hvx_dequantize_row_flat_bf16_f32(float * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {+ const uint32_t nvec = n / VLEN_FP32;+ const uint32_t nloe = n % VLEN_FP32;+ const uint16_t * src = (const uint16_t *) src_ptr;++ for (uint32_t i = 0; i < nvec; i++) {+ HVX_Vector v_bf16 = *(const HVX_UVector *) (src + i * VLEN_FP32);+ HVX_VectorPair p = Q6_Ww_vunpack_Vh(v_bf16);+ HVX_Vector res = Q6_Vw_vasl_VwR(Q6_V_lo_W(p), 16);++ hvx_vmem(dst_ptr + i * VLEN_FP32) = res;+ }++ if (nloe) {+ HVX_Vector v_bf16 = *(const HVX_UVector *) (src + nvec * VLEN_FP32);+ HVX_VectorPair p = Q6_Ww_vunpack_Vh(v_bf16);+ HVX_Vector res = Q6_Vw_vasl_VwR(Q6_V_lo_W(p), 16);++ hvx_vec_store_a(dst_ptr + nvec * VLEN_FP32, nloe * sizeof(float), res);+ }+}++static inline void hvx_dequantize_row_flat_q4_0_f32(float * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {+ const int nb = n / QK4_0;+ const block_q4_0 * src = (const block_q4_0 *) src_ptr;++ const HVX_VectorPred q16 = Q6_Q_vsetq_R(16);+ const HVX_Vector v_eight = Q6_Vb_vsplat_R(8);+ const HVX_Vector v_zero = Q6_V_vzero();++ for (int i = 0; i < nb; i++) {+ HVX_Vector vd_f16 = Q6_Vh_vsplat_R(*(const int16_t *) &src[i].d);+ HVX_VectorPair vp_f32 = hvx_vec_f16_to_f32(vd_f16);+ HVX_Vector vd = Q6_V_lo_W(vp_f32);++ HVX_Vector vq = *(const HVX_UVector *) src[i].qs;+ HVX_Vector v_lo = Q6_V_vand_VV(vq, Q6_Vb_vsplat_R(0x0F));+ HVX_Vector v_hi = Q6_Vub_vlsr_VubR(vq, 4);++ HVX_Vector v_lo_s8 = Q6_Vb_vsub_VbVb(v_lo, v_eight);+ HVX_Vector v_hi_s8 = Q6_Vb_vsub_VbVb(v_hi, v_eight);++ HVX_Vector v32 = Q6_V_vmux_QVV(q16, v_lo_s8, Q6_V_vlalign_VVR(v_hi_s8, v_zero, 16));++ HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(v32);+ HVX_Vector v_i16 = Q6_V_lo_W(p16);+ HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(v_i16);+ HVX_Vector v_i32 = Q6_V_lo_W(p32);++ HVX_Vector v_f32 = Q6_Vsf_equals_Vw(v_i32);+ HVX_Vector res = hvx_vec_mul_f32_f32(v_f32, vd);++ hvx_vmem(dst_ptr + i * QK4_0) = res;+ }+}++static inline void hvx_dequantize_row_flat_q4_k_f32(float * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {+ const int nb = n / QK_K;+ const block_q4_K * src = (const block_q4_K *) src_ptr;++ for (int i = 0; i < nb; i++) {+ const float d = GGML_FP16_TO_FP32(src[i].d);+ const float min = GGML_FP16_TO_FP32(src[i].dmin);+ const uint8_t * sc = src[i].scales;++ float * y = dst_ptr + i * QK_K;++ for (int step = 0; step < 4; ++step) {+ uint8_t sc1, m1_u, sc2, m2_u;+ if (step < 2) {+ sc1 = sc[2 * step + 0] & 63;+ m1_u = sc[2 * step + 4] & 63;+ sc2 = sc[2 * step + 1] & 63;+ m2_u = sc[2 * step + 5] & 63;+ } else {+ const int s = step - 2;+ sc1 = (sc[8 + 2 * s + 0] & 0xF) | ((sc[2 * s + 0] >> 6) << 4);+ m1_u = (sc[8 + 2 * s + 0] >> 4) | ((sc[4 + 2 * s + 0] >> 6) << 4);+ sc2 = (sc[8 + 2 * s + 1] & 0xF) | ((sc[2 * s + 1] >> 6) << 4);+ m2_u = (sc[8 + 2 * s + 1] >> 4) | ((sc[4 + 2 * s + 1] >> 6) << 4);+ }++ HVX_Vector vq = *(const HVX_UVector *) (src[i].qs + step * 32);+ HVX_Vector v_lo = Q6_V_vand_VV(vq, Q6_Vb_vsplat_R(0x0F));+ HVX_Vector v_hi = Q6_Vub_vlsr_VubR(vq, 4);++ HVX_VectorPair p16_lo = Q6_Wuh_vunpack_Vub(v_lo);+ HVX_VectorPair p32_lo = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_lo));+ HVX_Vector v_i32_lo = Q6_V_lo_W(p32_lo);+ HVX_Vector v_f32_lo = Q6_Vsf_equals_Vw(v_i32_lo);++ HVX_Vector vd1 = hvx_vec_splat_f32(d * sc1);+ HVX_Vector vm1 = hvx_vec_splat_f32(min * m1_u);+ HVX_Vector v_out1 = hvx_vec_sub_f32_f32(hvx_vec_mul_f32_f32(v_f32_lo, vd1), vm1);+ hvx_vmem(y + 0) = v_out1;++ HVX_VectorPair p16_hi = Q6_Wuh_vunpack_Vub(v_hi);+ HVX_VectorPair p32_hi = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_hi));+ HVX_Vector v_i32_hi = Q6_V_lo_W(p32_hi);+ HVX_Vector v_f32_hi = Q6_Vsf_equals_Vw(v_i32_hi);++ HVX_Vector vd2 = hvx_vec_splat_f32(d * sc2);+ HVX_Vector vm2 = hvx_vec_splat_f32(min * m2_u);+ HVX_Vector v_out2 = hvx_vec_sub_f32_f32(hvx_vec_mul_f32_f32(v_f32_hi, vd2), vm2);+ hvx_vmem(y + 32) = v_out2;++ y += 64;+ }}}+static inline void hvx_dequantize_row_flat_q5_k_f32(float * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {+ const int nb = n / QK_K;+ const block_q5_K * src = (const block_q5_K *) src_ptr;+ for (int i = 0; i < nb; i++) {+ const float d = GGML_FP16_TO_FP32(src[i].d);+ const float min = GGML_FP16_TO_FP32(src[i].dmin);+ const uint8_t * sc = src[i].scales;++ HVX_Vector v_qh = *(const HVX_UVector *) src[i].qh;+ float * y = dst_ptr + i * QK_K;++ for (int step = 0; step < 4; ++step) {+ uint8_t sc1, m1_u, sc2, m2_u;+ if (step < 2) {+ sc1 = sc[2 * step + 0] & 63;+ m1_u = sc[2 * step + 4] & 63;+ sc2 = sc[2 * step + 1] & 63;+ m2_u = sc[2 * step + 5] & 63;+ } else {+ const int s = step - 2;+ sc1 = (sc[8 + 2 * s + 0] & 0xF) | ((sc[2 * s + 0] >> 6) << 4);+ m1_u = (sc[8 + 2 * s + 0] >> 4) | ((sc[4 + 2 * s + 0] >> 6) << 4);+ sc2 = (sc[8 + 2 * s + 1] & 0xF) | ((sc[2 * s + 1] >> 6) << 4);+ m2_u = (sc[8 + 2 * s + 1] >> 4) | ((sc[4 + 2 * s + 1] >> 6) << 4);+ }++ HVX_Vector v_ql = *(const HVX_UVector *) (src[i].qs + step * 32);+ HVX_Vector v_lo = Q6_V_vand_VV(v_ql, Q6_Vb_vsplat_R(0x0F));+ HVX_Vector v_hi = Q6_Vub_vlsr_VubR(v_ql, 4);++ HVX_Vector v_h1 = Q6_Vw_vasl_VwR(Q6_V_vand_VV(Q6_Vub_vlsr_VubR(v_qh, 2 * step), Q6_Vb_vsplat_R(1)), 4);+ HVX_Vector v_val1 = Q6_V_vor_VV(v_lo, v_h1);++ HVX_VectorPair p16_1 = Q6_Wuh_vunpack_Vub(v_val1);+ HVX_VectorPair p32_1 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_1));+ HVX_Vector v_i32_1 = Q6_V_lo_W(p32_1);+ HVX_Vector v_f32_1 = Q6_Vsf_equals_Vw(v_i32_1);++ HVX_Vector vd1 = hvx_vec_splat_f32(d * sc1);+ HVX_Vector vm1 = hvx_vec_splat_f32(min * m1_u);+ HVX_Vector v_out1 = hvx_vec_sub_f32_f32(hvx_vec_mul_f32_f32(v_f32_1, vd1), vm1);+ hvx_vmem(y + 0) = v_out1;++ HVX_Vector v_h2 = Q6_Vw_vasl_VwR(Q6_V_vand_VV(Q6_Vub_vlsr_VubR(v_qh, 2 * step + 1), Q6_Vb_vsplat_R(1)), 4);+ HVX_Vector v_val2 = Q6_V_vor_VV(v_hi, v_h2);++ HVX_VectorPair p16_2 = Q6_Wuh_vunpack_Vub(v_val2);+ HVX_VectorPair p32_2 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_2));+ HVX_Vector v_i32_2 = Q6_V_lo_W(p32_2);+ HVX_Vector v_f32_2 = Q6_Vsf_equals_Vw(v_i32_2);++ HVX_Vector vd2 = hvx_vec_splat_f32(d * sc2);+ HVX_Vector vm2 = hvx_vec_splat_f32(min * m2_u);+ HVX_Vector v_out2 = hvx_vec_sub_f32_f32(hvx_vec_mul_f32_f32(v_f32_2, vd2), vm2);+ hvx_vmem(y + 32) = v_out2;++ y += 64;+ }+ }+}++static inline void hvx_dequantize_row_flat_q6_k_f32(float * restrict dst_ptr, const void * restrict src_ptr, uint32_t n) {+ const int nb = n / QK_K;+ const block_q6_K * src = (const block_q6_K *) src_ptr;++ const HVX_VectorPred q64 = Q6_Q_vsetq_R(64);+ const HVX_Vector v32_b = Q6_Vb_vsplat_R(32);+ const HVX_Vector mask_0f = Q6_Vb_vsplat_R(0x0F);+ const HVX_Vector mask_03 = Q6_Vb_vsplat_R(0x03);++ for (int i = 0; i < nb; i++) {+ const float d = GGML_FP16_TO_FP32(src[i].d);++ const uint8_t * restrict ql = src[i].ql;+ const uint8_t * restrict qh = src[i].qh;+ const int8_t * restrict sc = src[i].scales;++ float * y = dst_ptr + i * QK_K;++ for (int half = 0; half < 2; ++half) {+ float s[8];+ for (int j = 0; j < 8; ++j) {+ s[j] = d * sc[j];+ }++ HVX_Vector vscale1 = Q6_V_vmux_QVV(q64, hvx_vec_splat_f32(s[0]), hvx_vec_splat_f32(s[1]));+ HVX_Vector vscale2 = Q6_V_vmux_QVV(q64, hvx_vec_splat_f32(s[2]), hvx_vec_splat_f32(s[3]));+ HVX_Vector vscale3 = Q6_V_vmux_QVV(q64, hvx_vec_splat_f32(s[4]), hvx_vec_splat_f32(s[5]));+ HVX_Vector vscale4 = Q6_V_vmux_QVV(q64, hvx_vec_splat_f32(s[6]), hvx_vec_splat_f32(s[7]));++ HVX_Vector v_ql0 = *(const HVX_UVector *) (ql);+ HVX_Vector v_ql1 = *(const HVX_UVector *) (ql + 32);+ HVX_Vector v_qh = *(const HVX_UVector *) (qh);++ HVX_Vector ql0_lo = Q6_V_vand_VV(v_ql0, mask_0f);+ HVX_Vector qh_0 = Q6_Vw_vasl_VwR(Q6_V_vand_VV(v_qh, mask_03), 4);+ HVX_Vector v_q1 = Q6_Vb_vsub_VbVb(Q6_V_vor_VV(ql0_lo, qh_0), v32_b);++ HVX_Vector ql1_lo = Q6_V_vand_VV(v_ql1, mask_0f);+ HVX_Vector qh_1 = Q6_Vw_vasl_VwR(Q6_V_vand_VV(Q6_Vub_vlsr_VubR(v_qh, 2), mask_03), 4);+ HVX_Vector v_q2 = Q6_Vb_vsub_VbVb(Q6_V_vor_VV(ql1_lo, qh_1), v32_b);++ HVX_Vector ql0_hi = Q6_Vub_vlsr_VubR(v_ql0, 4);+ HVX_Vector qh_2 = Q6_Vw_vasl_VwR(Q6_V_vand_VV(Q6_Vub_vlsr_VubR(v_qh, 4), mask_03), 4);+ HVX_Vector v_q3 = Q6_Vb_vsub_VbVb(Q6_V_vor_VV(ql0_hi, qh_2), v32_b);++ HVX_Vector ql1_hi = Q6_Vub_vlsr_VubR(v_ql1, 4);+ HVX_Vector qh_3 = Q6_Vw_vasl_VwR(Q6_V_vand_VV(Q6_Vub_vlsr_VubR(v_qh, 6), mask_03), 4);+ HVX_Vector v_q4 = Q6_Vb_vsub_VbVb(Q6_V_vor_VV(ql1_hi, qh_3), v32_b);++ HVX_VectorPair p16_1 = Q6_Wh_vunpack_Vb(v_q1);+ HVX_VectorPair p32_1 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_1));+ HVX_Vector v_f32_1 = Q6_Vsf_equals_Vw(Q6_V_lo_W(p32_1));+ hvx_vmem(y + 0) = hvx_vec_mul_f32_f32(v_f32_1, vscale1);++ HVX_VectorPair p16_2 = Q6_Wh_vunpack_Vb(v_q2);+ HVX_VectorPair p32_2 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_2));+ HVX_Vector v_f32_2 = Q6_Vsf_equals_Vw(Q6_V_lo_W(p32_2));+ hvx_vmem(y + 32) = hvx_vec_mul_f32_f32(v_f32_2, vscale2);++ HVX_VectorPair p16_3 = Q6_Wh_vunpack_Vb(v_q3);+ HVX_VectorPair p32_3 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_3));+ HVX_Vector v_f32_3 = Q6_Vsf_equals_Vw(Q6_V_lo_W(p32_3));+ hvx_vmem(y + 64) = hvx_vec_mul_f32_f32(v_f32_3, vscale3);++ HVX_VectorPair p16_4 = Q6_Wh_vunpack_Vb(v_q4);+ HVX_VectorPair p32_4 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16_4));+ HVX_Vector v_f32_4 = Q6_Vsf_equals_Vw(Q6_V_lo_W(p32_4));+ hvx_vmem(y + 96) = hvx_vec_mul_f32_f32(v_f32_4, vscale4);++ y += 128;+ ql += 64;+ qh += 32;+ sc += 8;+ }+ }+}++// bytes row, row + 32, row + 64 and row + 96 of v, to bytes 0..3+static inline HVX_Vector hvx_dequantize_gather4(HVX_Vector v, uint32_t row) {+ v = Q6_V_vror_VR(v, row);+ HVX_Vector packed = Q6_V_vmux_QVV(Q6_Q_vsetq_R(1), v, Q6_V_vror_VR(v, 31));+ packed = Q6_V_vmux_QVV(Q6_Q_vsetq_R(2), packed, Q6_V_vror_VR(v, 62));+ return Q6_V_vmux_QVV(Q6_Q_vsetq_R(3), packed, Q6_V_vror_VR(v, 93));+}++static inline void hvx_dequantize_row_tiled_q8_0_f32(float * dst_spad, const uint8_t * src_spad, uint32_t n_k_tiles, uint32_t tile_stride, uint32_t row) {+ const HVX_VectorPred first2 = Q6_Q_vsetq_R(2);+ const HVX_VectorPred first4 = Q6_Q_vsetq_R(4);++ for (uint32_t k_tile = 0; k_tile < n_k_tiles; ++k_tile) {+ const uint8_t * tile = src_spad + k_tile * tile_stride;+ float * dst = dst_spad + k_tile * HTP_TENSOR_TILE_N_COLS;++ HVX_Vector vq = Q6_V_vzero();+ for (int group = 7; group >= 0; --group) {+ const HVX_Vector v = Q6_V_vror_VR(hvx_vmem(tile + group * VLEN), 2 * row);+ const HVX_Vector packed = Q6_V_vmux_QVV(first2, v, Q6_V_vror_VR(v, 62));+ vq = Q6_V_vmux_QVV(first4, packed, Q6_V_vror_VR(vq, VLEN - 4));+ }++ const HVX_Vector scales = hvx_vmem(tile + 1024);+ const HVX_Vector scale_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 2 * row));+ const HVX_Vector scale = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_hf));+ const HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(vq);+ const HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16));+ HVX_Vector values = hvx_vec_mul_f32_f32(Q6_Vsf_equals_Vw(Q6_V_lo_W(p32)), scale);+ *(HVX_Vector *) dst = values;+ }+}++static inline void hvx_dequantize_row_tiled_q4_0_f32(float * dst_spad, const uint8_t * src_spad, uint32_t n_k_tiles, uint32_t tile_stride, uint32_t row) {+ const HVX_VectorPred first4 = Q6_Q_vsetq_R(4);++ for (uint32_t k_tile = 0; k_tile < n_k_tiles; ++k_tile) {+ const uint8_t * tile = src_spad + k_tile * tile_stride;+ float * dst = dst_spad + k_tile * HTP_TENSOR_TILE_N_COLS;++ HVX_Vector vq = Q6_V_vzero();+ for (int group = 3; group >= 0; --group) {+ vq = Q6_V_vmux_QVV(first4, hvx_dequantize_gather4(hvx_vmem(tile + group * VLEN), row), Q6_V_vror_VR(vq, VLEN - 4));+ }++ const HVX_Vector lo = Q6_V_vand_VV(vq, Q6_Vb_vsplat_R(0x0F));+ const HVX_Vector hi = Q6_Vub_vlsr_VubR(vq, 4);+ vq = Q6_V_lo_W(Q6_W_vshuff_VVR(hi, lo, -1));+ vq = Q6_Vb_vsub_VbVb(vq, Q6_Vb_vsplat_R(8));++ const HVX_Vector scales = hvx_vmem(tile + 512);+ const HVX_Vector scale_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 2 * row));+ const HVX_Vector scale = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_hf));+ const HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(vq);+ const HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16));+ HVX_Vector values = hvx_vec_mul_f32_f32(Q6_Vsf_equals_Vw(Q6_V_lo_W(p32)), scale);+ *(HVX_Vector *) dst = values;+ }+}++static inline void hvx_dequantize_row_tiled_q4_k_f32(float * dst_spad, const uint8_t * src_spad, uint32_t n_k_tiles, uint32_t tile_stride, uint32_t row) {+ const HVX_VectorPred first4 = Q6_Q_vsetq_R(4);++ for (uint32_t k_tile = 0; k_tile < n_k_tiles; ++k_tile) {+ const uint8_t * tile = src_spad + k_tile * tile_stride;+ float * dst = dst_spad + k_tile * HTP_TENSOR_TILE_N_COLS;++ HVX_Vector vq = Q6_V_vzero();+ for (int group = 3; group >= 0; --group) {+ vq = Q6_V_vmux_QVV(first4, hvx_dequantize_gather4(hvx_vmem(tile + group * VLEN), row), Q6_V_vror_VR(vq, VLEN - 4));+ }++ const HVX_Vector lo = Q6_V_vand_VV(vq, Q6_Vb_vsplat_R(0x0F));+ const HVX_Vector hi = Q6_Vub_vlsr_VubR(vq, 4);+ vq = Q6_V_lo_W(Q6_W_vshuff_VVR(hi, lo, -1));++ const HVX_Vector scales = hvx_vmem(tile + 512);+ const HVX_Vector scale_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 4 * row));+ const HVX_Vector scale = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_hf));+ const HVX_Vector offset_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 4 * row + 2));+ const HVX_Vector offset = Q6_V_lo_W(hvx_vec_f16_to_f32(offset_hf));++ const HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(vq);+ const HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16));+ HVX_Vector values = hvx_vec_mul_f32_f32(Q6_Vsf_equals_Vw(Q6_V_lo_W(p32)), scale);+ *(HVX_Vector *) dst = hvx_vec_add_f32_f32(values, offset);+ }+}++static inline void hvx_dequantize_row_tiled_q5_k_f32(float * dst_spad, const uint8_t * src_spad, uint32_t n_k_tiles, uint32_t tile_stride, uint32_t row) {+ const HVX_VectorPred first4 = Q6_Q_vsetq_R(4);+ const HVX_Vector one = Q6_Vb_vsplat_R(1);++ for (uint32_t k_tile = 0; k_tile < n_k_tiles; ++k_tile) {+ const uint8_t * tile = src_spad + k_tile * tile_stride;+ float * dst = dst_spad + k_tile * HTP_TENSOR_TILE_N_COLS;++ const HVX_Vector p5 = hvx_dequantize_gather4(hvx_vmem(tile + 640), row);+ HVX_Vector vq = Q6_V_vzero();+ HVX_Vector vh = Q6_V_vzero();+ for (int group = 3; group >= 0; --group) {+ vq = Q6_V_vmux_QVV(first4, hvx_dequantize_gather4(hvx_vmem(tile + group * VLEN), row), Q6_V_vror_VR(vq, VLEN - 4));+ vh = Q6_V_vmux_QVV(first4, Q6_Vub_vlsr_VubR(p5, 2 * group), Q6_V_vror_VR(vh, VLEN - 4));+ }++ const HVX_Vector lo = Q6_V_vor_VV(Q6_V_vand_VV(vq, Q6_Vb_vsplat_R(0x0F)), Q6_Vw_vasl_VwR(Q6_V_vand_VV(vh, one), 4));+ const HVX_Vector hi = Q6_V_vor_VV(Q6_Vub_vlsr_VubR(vq, 4), Q6_Vw_vasl_VwR(Q6_V_vand_VV(Q6_Vub_vlsr_VubR(vh, 1), one), 4));+ vq = Q6_V_lo_W(Q6_W_vshuff_VVR(hi, lo, -1));++ const HVX_Vector scales = hvx_vmem(tile + 512);+ const HVX_Vector scale_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 4 * row));+ const HVX_Vector scale = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_hf));+ const HVX_Vector offset_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 4 * row + 2));+ const HVX_Vector offset = Q6_V_lo_W(hvx_vec_f16_to_f32(offset_hf));++ const HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(vq);+ const HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16));+ HVX_Vector values = hvx_vec_mul_f32_f32(Q6_Vsf_equals_Vw(Q6_V_lo_W(p32)), scale);+ *(HVX_Vector *) dst = hvx_vec_add_f32_f32(values, offset);+ }+}++static inline void hvx_dequantize_row_tiled_q6_k_f32(float * dst_spad, const uint8_t * src_spad, uint32_t n_k_tiles, uint32_t tile_stride, uint32_t row) {+ const HVX_VectorPred first4 = Q6_Q_vsetq_R(4);+ const HVX_VectorPred first16 = Q6_Q_vsetq_R(16 * sizeof(float));+ const HVX_Vector mask_0f = Q6_Vb_vsplat_R(0x0F);+ const HVX_Vector mask_03 = Q6_Vb_vsplat_R(0x03);++ for (uint32_t k_tile = 0; k_tile < n_k_tiles; ++k_tile) {+ const uint8_t * tile = src_spad + k_tile * tile_stride;+ float * dst = dst_spad + k_tile * HTP_TENSOR_TILE_N_COLS;++ HVX_Vector vq = Q6_V_vzero();+ for (int group = 7; group >= 0; --group) {+ const HVX_Vector lo_plane = Q6_V_vror_VR(hvx_vmem(tile + (group >> 1) * VLEN), 4 * row);+ const HVX_Vector hi_plane = Q6_V_vror_VR(hvx_vmem(tile + 512 + (group >> 2) * VLEN), 4 * row);+ const HVX_Vector lo = (group & 1) ? Q6_Vub_vlsr_VubR(lo_plane, 4) : Q6_V_vand_VV(lo_plane, mask_0f);+ const HVX_Vector hi = Q6_Vub_vlsr_VubR(hi_plane, 2 * (group & 3));+ const HVX_Vector packed = Q6_V_vor_VV(lo, Q6_Vw_vasl_VwR(Q6_V_vand_VV(hi, mask_03), 4));+ vq = Q6_V_vmux_QVV(first4, packed, Q6_V_vror_VR(vq, VLEN - 4));+ }++ const HVX_Vector scales = hvx_vmem(tile + 768);+ const HVX_Vector scale_lo_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 2 * row));+ const HVX_Vector scale_hi_hf = hvx_vec_repl_f16(Q6_V_vror_VR(scales, 64 + 2 * row));+ const HVX_Vector scale_lo = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_lo_hf));+ const HVX_Vector scale_hi = Q6_V_lo_W(hvx_vec_f16_to_f32(scale_hi_hf));+ const HVX_Vector scale = Q6_V_vmux_QVV(first16, scale_lo, scale_hi);+ const HVX_VectorPair p16 = Q6_Wh_vunpack_Vb(Q6_Vb_vsub_VbVb(vq, Q6_Vb_vsplat_R(32)));+ const HVX_VectorPair p32 = Q6_Ww_vunpack_Vh(Q6_V_lo_W(p16));+ *(HVX_Vector *) dst = hvx_vec_mul_f32_f32(Q6_Vsf_equals_Vw(Q6_V_lo_W(p32)), scale);+ }+}++static inline HVX_Vector hvx_dequantize_tiled_f16_gather4(const uint8_t * p, uint32_t r_byte) {+ HVX_Vector w0 = Q6_V_vror_VR(hvx_vmem(p + 0 * VLEN), r_byte);+ HVX_Vector w1 = Q6_V_vror_VR(hvx_vmem(p + 1 * VLEN), r_byte);+ HVX_Vector w01 = Q6_V_lo_W(Q6_W_vshuff_VVR(w1, w0, 4));+ HVX_Vector w2 = Q6_V_vror_VR(hvx_vmem(p + 2 * VLEN), r_byte);+ HVX_Vector w3 = Q6_V_vror_VR(hvx_vmem(p + 3 * VLEN), r_byte);+ HVX_Vector w23 = Q6_V_lo_W(Q6_W_vshuff_VVR(w3, w2, 4));+ return Q6_V_lo_W(Q6_W_vshuff_VVR(w23, w01, 8));+}++static inline void hvx_dequantize_row_tiled_f16_f32(float * dst_spad, const uint8_t * src_spad, uint32_t n_k_tiles, uint32_t tile_stride, uint32_t row) {+ const uint32_t r_byte = 4 * row;++ for (uint32_t k_tile = 0; k_tile < n_k_tiles; ++k_tile) {+ const uint8_t * tile = src_spad + k_tile * tile_stride;+ float * dst = dst_spad + k_tile * HTP_TENSOR_TILE_N_COLS;++ HVX_Vector g0 = hvx_dequantize_tiled_f16_gather4(tile + 0 * VLEN, r_byte);+ HVX_Vector g1 = hvx_dequantize_tiled_f16_gather4(tile + 4 * VLEN, r_byte);+ HVX_Vector g01 = Q6_V_lo_W(Q6_W_vshuff_VVR(g1, g0, 16));++ HVX_Vector g2 = hvx_dequantize_tiled_f16_gather4(tile + 8 * VLEN, r_byte);+ HVX_Vector g3 = hvx_dequantize_tiled_f16_gather4(tile + 12 * VLEN, r_byte);+ HVX_Vector g23 = Q6_V_lo_W(Q6_W_vshuff_VVR(g3, g2, 16));++ HVX_Vector g = Q6_V_lo_W(Q6_W_vshuff_VVR(g23, g01, 32));+ *(HVX_Vector *) dst = Q6_V_lo_W(hvx_vec_f16_to_f32(g));+ }+}#endif // HVX_QUANT_Hdiff --git a/ggml/src/ggml-hexagon/htp/main.c b/ggml/src/ggml-hexagon/htp/main.cindex df7c4a5bd1..2374add3f5 100644--- a/ggml/src/ggml-hexagon/htp/main.c+++ b/ggml/src/ggml-hexagon/htp/main.c@@ -28,8 +28,8 @@#include "ggml-common.h"#include "hex-bitmap.h"#include "htp-ctx.h"+#include "htp-dcache.h"#include "htp-ops.h"-#include "htp-tensor.h"#include "htp_iface.h"#include "work-queue.h"#include "hex-profile.h"@@ -835,7 +835,7 @@ static int op_fence(struct htp_ops_context * octx) {atomic_uint * sync_fence = (atomic_uint *) (uintptr_t) sync->data;if (mode == 1) {- htp_flush_dirty_ranges(ctx);+ htp_dcache_flush_dirty_ranges(ctx);htp_mdev_group_barrier(octx);@@ -1211,7 +1211,7 @@ static int proc_op_req(struct htp_ops_context * octx, struct htp_buf_desc * bufssrc->ne[0], src->ne[1], src->ne[2], src->ne[3]);}- htp_tensor_flush_all(octx->ctx, octx->src, HTP_OP_MAX_INPUTS);+ htp_dcache_flush_dirty_tensors(octx->ctx, octx->src, HTP_OP_MAX_INPUTS);// Prep output tensorsfor (uint32_t i = 0; i < HTP_OP_MAX_OUTPUTS; i++) {@@ -1227,7 +1227,7 @@ static int proc_op_req(struct htp_ops_context * octx, struct htp_buf_desc * bufsdst->ne[0], dst->ne[1], dst->ne[2], dst->ne[3]);}- htp_tensor_dirty_all(octx->ctx, octx->dsts, HTP_OP_MAX_OUTPUTS);+ htp_dcache_mark_dirty_tensors(octx->ctx, octx->dsts, HTP_OP_MAX_OUTPUTS);htp_mdev_group_barrier(octx);diff --git a/ggml/src/ggml-hexagon/htp/matmul-ops.c b/ggml/src/ggml-hexagon/htp/matmul-ops.cindex 28aed4680d..3837c290d6 100644--- a/ggml/src/ggml-hexagon/htp/matmul-ops.c+++ b/ggml/src/ggml-hexagon/htp/matmul-ops.c@@ -38,8 +38,12 @@ typedef struct {int act_stride;uint32_t act_elem_size; // 4=F32 src1, 2=F16 src1int weight_stride;+ int weight_type;int dst_stride;+ int dst_cols;uint32_t bias_stride;+ int tile_size;+ int aligned_tile_size;int ne02;int ne03;int ne12;@@ -56,7 +60,7 @@ typedef struct {int r3;struct fastdiv_values div_r2;struct fastdiv_values div_r3;-} hmx_mm_f16_f32_batched_params_t;+} hmx_mm_batched_params_t;static bool htp_matmul_has_extended_weight(const struct htp_ops_context * octx, uint32_t n_weights) {for (uint32_t i = 0; i < n_weights; ++i) {@@ -89,6 +93,11 @@ struct htp_mm_context {const void * restrict vy, uint32_t valid_rows,const float * restrict sz);+ void (*vec_dot_32x2)(const uint32_t n, float * restrict s0, float * restrict s1,+ const void * restrict vx,+ const void * restrict vy0, const void * restrict vy1, uint32_t valid_rows,+ const float * restrict sz0, const float * restrict sz1);+// Precomputed valuesuint32_t src0_nrows_per_thread;uint32_t src0_row_start;@@ -238,314 +247,301 @@ static const uint8_t __attribute__((aligned(VLEN))) kvalues_mxfp4_lut[] = {#include "hvx-mm-kernels-float.h"#include "hmx-mm-kernels-tiled.h"-// Specialized repacked matmul macros-#define MATMUL_2D_REPACKED_IMPL(SUFFIX, TILE_SIZE, DOT_2X2, DOT_2X1) \-static void hvx_mm_2d_repacked_##SUFFIX(unsigned int nth, unsigned int ith, void * data) { \- htp_matmul_preamble; \- \- const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start; \- const uint32_t act_nrows = mmctx->cur_m_rows ? mmctx->cur_m_rows : (ne11 * ne12 * ne13); \- const uint32_t cur_m_start = mmctx->cur_m_start; \- \- const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith; \- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end); \- \- struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \- \- const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params; \- const uint32_t n_prefetch = kparams->n_prefetch; \- assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0); \- \- const size_t dst_row_size = nb1; \- const size_t act_row_size = nb11; \- const size_t act_stride = mmctx->vtcm_act_stride; \- const size_t src2_stride = src2 ? ((src2->ne[1] == 1) ? 0 : src2->nb[1]) : 0; \- \- uint8_t * restrict vtcm_dst_ptr = mmctx->vtcm_dst + mmctx->vtcm_dst_size_per_thread * ith; \- uint8_t * restrict vtcm_src0_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith; \- uint8_t * restrict act_data = mmctx->vtcm_act; \- \- const dma_addr_t src0_row = src0->data; \- \- const uint32_t tile_size = TILE_SIZE; \- const uint32_t aligned_tile_size = hex_align_up(tile_size, 128); \- \- uint32_t n_k_tiles_w = ne00 / 32; \- uint32_t n_k_tiles_a = ne10 / 32; \- uint32_t tile_row_stride = n_k_tiles_w * tile_size; \- uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size; \- \- uint32_t ct_start = src0_start_row / 32; \- uint32_t ct_end = (src0_end_row + 31) / 32; \- \- uint32_t push_ct = ct_start; \- if (src0_start_row < src0_end_row) { \- for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) { \- dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + d * tile_row_transfer_size_aligned, \- src0_row + push_ct * tile_row_stride), aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \- } \- } \- \- if (src0_start_row >= src0_end_row) { \- return; \- } \- \- for (uint32_t ct = ct_start; ct < ct_end; ct++) { \- const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst; \- \- int valid_rows = (int)ne0 - (int)(ct * 32); \- valid_rows = MIN(32, MAX(0, valid_rows)); \- \- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct); \- uint32_t ir1 = 0; \- for (; ir1 + 1 < act_nrows; ir1 += 2) { \- const uint8_t * restrict act_col0 = (const uint8_t *) (act_data + (ir1+0) * act_stride); \- const uint8_t * restrict act_col1 = (const uint8_t *) (act_data + (ir1+1) * act_stride); \- float * restrict dst_row0 = (float *) (dst->data + ((cur_m_start + ir1+0) * dst_row_size)); \- float * restrict dst_row1 = (float *) (dst->data + ((cur_m_start + ir1+1) * dst_row_size)); \- \- float * dst_ptr0 = &dst_row0[ct * 32]; \- float * dst_ptr1 = &dst_row1[ct * 32]; \- \- const float * src2_ptr0 = NULL; \- const float * src2_ptr1 = NULL; \- if (src2) { \- const float * restrict src2_row0 = (const float *) ((const uint8_t *) src2->data + ((cur_m_start + ir1+0) * src2_stride)); \- const float * restrict src2_row1 = (const float *) ((const uint8_t *) src2->data + ((cur_m_start + ir1+1) * src2_stride)); \- src2_ptr0 = &src2_row0[ct * 32]; \- src2_ptr1 = &src2_row1[ct * 32]; \- } \- DOT_2X2(ne10, dst_ptr0, dst_ptr1, w_tile, act_col0, act_col1, valid_rows, src2_ptr0, src2_ptr1); \- } \- \- for (; ir1 < act_nrows; ++ir1) { \- const uint8_t * restrict act_col = (const uint8_t *) (act_data + ir1 * act_stride); \- float * restrict dst_row = (float *) (dst->data + ((cur_m_start + ir1) * dst_row_size)); \- float * dst_ptr = &dst_row[ct * 32]; \- \- const float * src2_ptr = NULL; \- if (src2) { \- const float * restrict src2_row = (const float *) ((const uint8_t *) src2->data + ((cur_m_start + ir1) * src2_stride)); \- src2_ptr = &src2_row[ct * 32]; \- } \- DOT_2X1(ne10, dst_ptr, w_tile, act_col, valid_rows, src2_ptr); \- } \- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct); \- \- if (push_ct < ct_end) { \- dma_queue_push(dma_q, dma_make_data(w_tile, src0_row + push_ct * tile_row_stride), \- aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \- push_ct++; \- } \- } \-}+static void hvx_mm_2d_tiled(unsigned int nth, unsigned int ith, void * data) {+ htp_matmul_preamble;++ const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start;+ const uint32_t act_nrows = mmctx->cur_m_rows ? mmctx->cur_m_rows : (ne11 * ne12 * ne13);+ const uint32_t cur_m_start = mmctx->cur_m_start;++ const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith;+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end);++ struct htp_thread_trace * tr = &octx->ctx->trace[ith];++ const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;+ const uint32_t n_prefetch = kparams->n_prefetch;+ assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0);++ const size_t dst_row_size = nb1;+ const size_t act_row_size = nb11;+ const size_t act_stride = mmctx->vtcm_act_stride;+ const size_t src2_stride = src2 ? ((src2->ne[1] == 1) ? 0 : src2->nb[1]) : 0;++ uint8_t * restrict vtcm_dst_ptr = mmctx->vtcm_dst + mmctx->vtcm_dst_size_per_thread * ith;+ uint8_t * restrict vtcm_src0_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith;+ uint8_t * restrict act_data = mmctx->vtcm_act;++ const dma_addr_t src0_row = src0->data;++ const uint32_t tile_size = htp_tensor_get_tile_size(src0->type);+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src0->type);++ uint32_t n_k_tiles_w = ne00 / 32;+ uint32_t n_k_tiles_a = ne10 / 32;+ uint32_t tile_row_stride = n_k_tiles_w * tile_size;+ uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size;++ uint32_t ct_start = src0_start_row / 32;+ uint32_t ct_end = (src0_end_row + 31) / 32;++ uint32_t push_ct = ct_start;+ if (src0_start_row < src0_end_row) {+ for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) {+ dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + d * tile_row_transfer_size_aligned,+ src0_row + push_ct * tile_row_stride), aligned_tile_size, tile_size, tile_size, n_k_tiles_a);+ }+ }++ if (src0_start_row >= src0_end_row) {+ return;+ }++ for (uint32_t ct = ct_start; ct < ct_end; ct++) {+ const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;++ int valid_rows = (int)ne0 - (int)(ct * 32);+ valid_rows = MIN(32, MAX(0, valid_rows));++ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);+ uint32_t ir1 = 0;+ for (; ir1 + 1 < act_nrows; ir1 += 2) {+ const uint8_t * restrict act_col0 = (const uint8_t *) (act_data + (ir1+0) * act_stride);+ const uint8_t * restrict act_col1 = (const uint8_t *) (act_data + (ir1+1) * act_stride);+ float * restrict dst_row0 = (float *) (dst->data + ((cur_m_start + ir1+0) * dst_row_size));+ float * restrict dst_row1 = (float *) (dst->data + ((cur_m_start + ir1+1) * dst_row_size));++ float * dst_ptr0 = &dst_row0[ct * 32];+ float * dst_ptr1 = &dst_row1[ct * 32];-#define MATVEC_2D_REPACKED_IMPL(SUFFIX, TILE_SIZE, DOT_2X1) \-static void hvx_mv_2d_repacked_##SUFFIX(unsigned int nth, unsigned int ith, void * data) { \- htp_matmul_preamble; \- \- const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start; \- \- const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith; \- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end); \- \- struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \- \- const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params; \- const uint32_t n_prefetch = kparams->n_prefetch; \- assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0); \- \- const size_t dst_row_size = nb1; \- const size_t act_row_size = nb11; \- const size_t act_stride = mmctx->vtcm_act_stride; \- \- uint8_t * vtcm_dst_ptr = mmctx->vtcm_dst + mmctx->vtcm_dst_size_per_thread * ith; \- uint8_t * vtcm_src0_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith; \- uint8_t * act_data = mmctx->vtcm_act; \- \- float * tmp = (float *) vtcm_dst_ptr; \- \- const dma_addr_t src0_row = src0->data; \- \- const uint8_t * restrict act_col = (const uint8_t *) act_data; \- float * restrict dst_col = (float *) dst->data; \- \- const uint32_t tile_size = TILE_SIZE; \- const uint32_t aligned_tile_size = hex_align_up(tile_size, 128); \- \- uint32_t n_k_tiles_w = ne00 / 32; \- uint32_t n_k_tiles_a = ne10 / 32; \- uint32_t tile_row_stride = n_k_tiles_w * tile_size; \- uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size; \- \- uint32_t ct_start = src0_start_row / 32; \- uint32_t ct_end = (src0_end_row + 31) / 32; \- \- uint32_t push_ct = ct_start; \- if (src0_start_row < src0_end_row) { \- if (src2) { \- float * vtcm_bias_ptr = (float *) mmctx->vtcm_bias + src0_start_row; \- const dma_addr_t src2_addr = src2->data + src0_start_row * sizeof(float); \- int slice_size = (int)MIN(src0_end_row, ne0) - (int)src0_start_row; \- if (slice_size > 0) { \- dma_queue_push(dma_q, dma_make_data(vtcm_bias_ptr, src2_addr), \- slice_size * sizeof(float), slice_size * sizeof(float), slice_size * sizeof(float), 1); \- dma_queue_pop_nowait(dma_q); \- } \- } \- for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) { \- dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + d * tile_row_transfer_size_aligned, \- src0_row + push_ct * tile_row_stride), aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \- } \- } \- \- if (src0_start_row >= src0_end_row) { \- return; \- } \- \- for (uint32_t ct = ct_start; ct < ct_end; ct++) { \- const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst; \- \- float * dst_ptr = &tmp[ct * 32 - src0_start_row]; \- int valid_rows = (int)ne0 - (int)(ct * 32); \- valid_rows = MIN(32, MAX(0, valid_rows)); \- \- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct); \- DOT_2X1(ne10, dst_ptr, w_tile, act_col, valid_rows, NULL); \- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct); \- \- if (push_ct < ct_end) { \- dma_queue_push(dma_q, dma_make_data(w_tile, src0_row + push_ct * tile_row_stride), \- aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \- push_ct++; \- } \- } \- \- int copy_cnt = (int)MIN(src0_end_row, ne0) - (int)src0_start_row; \- if (copy_cnt > 0) { \- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct_end); \- if (src2) { \- hvx_add_f32_uuu((uint8_t *) &dst_col[src0_start_row], \- (const uint8_t *) tmp, \- (const uint8_t *) ((const float *) mmctx->vtcm_bias + src0_start_row), \- copy_cnt); \- } else { \- hvx_copy_f32_ua((uint8_t *) &dst_col[src0_start_row], (uint8_t *) tmp, copy_cnt); \- } \- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct_end); \- } \+ const float * src2_ptr0 = NULL;+ const float * src2_ptr1 = NULL;+ if (src2) {+ const float * restrict src2_row0 = (const float *) ((const uint8_t *) src2->data + ((cur_m_start + ir1+0) * src2_stride));+ const float * restrict src2_row1 = (const float *) ((const uint8_t *) src2->data + ((cur_m_start + ir1+1) * src2_stride));+ src2_ptr0 = &src2_row0[ct * 32];+ src2_ptr1 = &src2_row1[ct * 32];+ }+ mmctx->vec_dot_32x2(ne10, dst_ptr0, dst_ptr1, w_tile, act_col0, act_col1, valid_rows, src2_ptr0, src2_ptr1);+ }++ for (; ir1 < act_nrows; ++ir1) {+ const uint8_t * restrict act_col = (const uint8_t *) (act_data + ir1 * act_stride);+ float * restrict dst_row = (float *) (dst->data + ((cur_m_start + ir1) * dst_row_size));+ float * dst_ptr = &dst_row[ct * 32];++ const float * src2_ptr = NULL;+ if (src2) {+ const float * restrict src2_row = (const float *) ((const uint8_t *) src2->data + ((cur_m_start + ir1) * src2_stride));+ src2_ptr = &src2_row[ct * 32];+ }+ mmctx->vec_dot_32x1(ne10, dst_ptr, w_tile, act_col, valid_rows, src2_ptr);+ }+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct);++ if (push_ct < ct_end) {+ dma_queue_push(dma_q, dma_make_data(w_tile, src0_row + push_ct * tile_row_stride),+ aligned_tile_size, tile_size, tile_size, n_k_tiles_a);+ push_ct++;+ }+ }}-#define MATMUL_NX_2D_REPACKED_IMPL(SUFFIX, TILE_SIZE, DOT_2X2, DOT_2X1) \-static void hvx_mm_nx_2d_repacked_##SUFFIX(unsigned int nth, unsigned int ith, void * data) { \- struct htp_mm_context * mmctx = data; \- struct htp_ops_context * octx = mmctx->octx; \- const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params; \- const uint32_t n_weights = kparams->n_weights; \- \- const struct htp_tensor * restrict act = octx->src[n_weights]; /* x */ \- const uint32_t ne10 = act->ne[0]; \- const uint32_t act_nrows = act->ne[1] * act->ne[2] * act->ne[3]; \- const size_t act_stride = mmctx->vtcm_act_stride; \- \- uint8_t * restrict vtcm_weight_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith; \- uint8_t * restrict act_data = mmctx->vtcm_act; \- \- struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \- const uint32_t n_prefetch = kparams->n_prefetch; \- assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0); \- \- const uint32_t tile_size = TILE_SIZE; \- const uint32_t aligned_tile_size = hex_align_up(tile_size, 128); \- uint32_t n_k_tiles_a = ne10 / 32; \- uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size; \- \- for (uint32_t widx = 0; widx < n_weights; widx++) { \- const struct htp_tensor * restrict src_w = octx->src[widx]; \- const struct htp_tensor * restrict dst = octx->dsts[widx]; \- if (!src_w || !dst) continue; \- dma_queue * dma_q = octx->ctx->dma[ith]; \- \- const uint32_t ne00 = src_w->ne[0]; \- const uint32_t ne01 = src_w->ne[1]; \- const size_t dst_row_size = dst->nb[1]; \- const dma_addr_t src_w_row = src_w->data; \- \- uint32_t n_k_tiles_w = ne00 / 32; \- uint32_t tile_row_stride = n_k_tiles_w * tile_size; \- \- uint32_t src0_start_row = 0; \- uint32_t src0_end_row = ne01; \- if (octx->ctx->mdev.count > 1) { \- const bool can_split = htp_tensor_can_row_partition(dst, sizeof(float)); \- const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(ne01, can_split ? 32 : 0, \- octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div); \- src0_start_row = range.start; \- src0_end_row = range.start + range.count; \- } \- \- const uint32_t nrows = src0_end_row - src0_start_row; \- uint32_t src0_nrows_per_thread = fastdiv(nrows + nth - 1, &octx->n_threads_div); \- src0_nrows_per_thread = hex_round_up(src0_nrows_per_thread, 32); \- \- const uint32_t start_row = src0_start_row + src0_nrows_per_thread * ith; \- const uint32_t end_row = MIN(start_row + src0_nrows_per_thread, src0_end_row); \- if (start_row >= end_row) continue; \- \- uint32_t ct_start = start_row / 32; \- uint32_t ct_end = (end_row + 31) / 32; \- \- uint32_t push_ct = ct_start; \- for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) { \- dma_queue_push(dma_q, dma_make_data(vtcm_weight_ptr + d * tile_row_transfer_size_aligned, \- src_w_row + push_ct * tile_row_stride), aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \- } \- \- for (uint32_t ct = ct_start; ct < ct_end; ct++) { \- const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst; \- int valid_rows = (int)ne01 - (int)(ct * 32); \- valid_rows = MIN(32, MAX(0, valid_rows)); \- \- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct); \- uint32_t ir1 = 0; \- for (; ir1 + 1 < act_nrows; ir1 += 2) { \- const uint8_t * restrict act_col0 = (const uint8_t *) (act_data + (ir1+0) * act_stride); \- const uint8_t * restrict act_col1 = (const uint8_t *) (act_data + (ir1+1) * act_stride); \- \- float * restrict dst_row0 = (float *) (dst->data + ((ir1+0) * dst_row_size)); \- float * restrict dst_row1 = (float *) (dst->data + ((ir1+1) * dst_row_size)); \- float * dst_ptr0 = &dst_row0[ct * 32]; \- float * dst_ptr1 = &dst_row1[ct * 32]; \- \- DOT_2X2(ne10, dst_ptr0, dst_ptr1, w_tile, act_col0, act_col1, valid_rows, NULL, NULL); \- } \- \- for (; ir1 < act_nrows; ++ir1) { \- const uint8_t * restrict act_col = (const uint8_t *) (act_data + ir1 * act_stride); \- float * restrict dst_row = (float *) (dst->data + (ir1 * dst_row_size)); \- float * dst_ptr = &dst_row[ct * 32]; \- DOT_2X1(ne10, dst_ptr, w_tile, act_col, valid_rows, NULL); \- } \- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct); \- \- if (push_ct < ct_end) { \- dma_queue_push(dma_q, dma_make_data(w_tile, src_w_row + push_ct * tile_row_stride), \- aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \- push_ct++; \- } \- } \- } \+static void hvx_mv_2d_tiled(unsigned int nth, unsigned int ith, void * data) {+ htp_matmul_preamble;++ const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start;++ const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith;+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end);++ struct htp_thread_trace * tr = &octx->ctx->trace[ith];++ const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;+ const uint32_t n_prefetch = kparams->n_prefetch;+ assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0);++ const size_t dst_row_size = nb1;+ const size_t act_row_size = nb11;+ const size_t act_stride = mmctx->vtcm_act_stride;++ uint8_t * vtcm_dst_ptr = mmctx->vtcm_dst + mmctx->vtcm_dst_size_per_thread * ith;+ uint8_t * vtcm_src0_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith;+ uint8_t * act_data = mmctx->vtcm_act;++ float * tmp = (float *) vtcm_dst_ptr;++ const dma_addr_t src0_row = src0->data;++ const uint8_t * restrict act_col = (const uint8_t *) act_data;+ float * restrict dst_col = (float *) dst->data;++ const uint32_t tile_size = htp_tensor_get_tile_size(src0->type);+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src0->type);++ uint32_t n_k_tiles_w = ne00 / 32;+ uint32_t n_k_tiles_a = ne10 / 32;+ uint32_t tile_row_stride = n_k_tiles_w * tile_size;+ uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size;++ uint32_t ct_start = src0_start_row / 32;+ uint32_t ct_end = (src0_end_row + 31) / 32;++ uint32_t push_ct = ct_start;+ if (src0_start_row < src0_end_row) {+ if (src2) {+ float * vtcm_bias_ptr = (float *) mmctx->vtcm_bias + src0_start_row;+ const dma_addr_t src2_addr = src2->data + src0_start_row * sizeof(float);+ int slice_size = (int)MIN(src0_end_row, ne0) - (int)src0_start_row;+ if (slice_size > 0) {+ dma_queue_push(dma_q, dma_make_data(vtcm_bias_ptr, src2_addr),+ slice_size * sizeof(float), slice_size * sizeof(float), slice_size * sizeof(float), 1);+ dma_queue_pop_nowait(dma_q);+ }+ }+ for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) {+ dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + d * tile_row_transfer_size_aligned,+ src0_row + push_ct * tile_row_stride), aligned_tile_size, tile_size, tile_size, n_k_tiles_a);+ }+ }++ if (src0_start_row >= src0_end_row) {+ return;+ }++ for (uint32_t ct = ct_start; ct < ct_end; ct++) {+ const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;++ float * dst_ptr = &tmp[ct * 32 - src0_start_row];+ int valid_rows = (int)ne0 - (int)(ct * 32);+ valid_rows = MIN(32, MAX(0, valid_rows));++ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);+ mmctx->vec_dot_32x1(ne10, dst_ptr, w_tile, act_col, valid_rows, NULL);+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct);++ if (push_ct < ct_end) {+ dma_queue_push(dma_q, dma_make_data(w_tile, src0_row + push_ct * tile_row_stride),+ aligned_tile_size, tile_size, tile_size, n_k_tiles_a);+ push_ct++;+ }+ }++ int copy_cnt = (int)MIN(src0_end_row, ne0) - (int)src0_start_row;+ if (copy_cnt > 0) {+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct_end);+ if (src2) {+ hvx_add_f32_uuu((uint8_t *) &dst_col[src0_start_row],+ (const uint8_t *) tmp,+ (const uint8_t *) ((const float *) mmctx->vtcm_bias + src0_start_row),+ copy_cnt);+ } else {+ hvx_copy_f32_ua((uint8_t *) &dst_col[src0_start_row], (uint8_t *) tmp, copy_cnt);+ }+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct_end);+ }}-MATMUL_2D_REPACKED_IMPL(q4_0, 576, tiled_vec_dot_q4_0_32x2, tiled_vec_dot_q4_0_32x1)-MATMUL_2D_REPACKED_IMPL(q4_1, 640, tiled_vec_dot_q4_1_32x2, tiled_vec_dot_q4_1_32x1)-MATMUL_2D_REPACKED_IMPL(q8_0, 1088, tiled_vec_dot_q8_0_32x2, tiled_vec_dot_q8_0_32x1)-MATMUL_2D_REPACKED_IMPL(q6_k, 896, tiled_vec_dot_q6_k_32x2, tiled_vec_dot_q6_k_32x1)-MATMUL_2D_REPACKED_IMPL(q5_k, 768, tiled_vec_dot_q5_k_32x2, tiled_vec_dot_q5_k_32x1)-MATMUL_2D_REPACKED_IMPL(q3_k, 512, tiled_vec_dot_q3_k_32x2, tiled_vec_dot_q3_k_32x1)-MATMUL_2D_REPACKED_IMPL(q2_k, 336, tiled_vec_dot_q2_k_32x2, tiled_vec_dot_q2_k_32x1)-MATMUL_2D_REPACKED_IMPL(iq4nl, 576, tiled_vec_dot_iq4nl_32x2, tiled_vec_dot_iq4nl_32x1)-MATMUL_2D_REPACKED_IMPL(mxfp4, 544, tiled_vec_dot_mxfp4_32x2, tiled_vec_dot_mxfp4_32x1)+static void hvx_mm_nx_2d_tiled(unsigned int nth, unsigned int ith, void * data) {+ struct htp_mm_context * mmctx = data;+ struct htp_ops_context * octx = mmctx->octx;+ const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;+ const uint32_t n_weights = kparams->n_weights;++ const struct htp_tensor * restrict act = octx->src[n_weights]; /* x */+ const uint32_t ne10 = act->ne[0];+ const uint32_t act_nrows = act->ne[1] * act->ne[2] * act->ne[3];+ const size_t act_stride = mmctx->vtcm_act_stride;++ uint8_t * restrict vtcm_weight_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith;+ uint8_t * restrict act_data = mmctx->vtcm_act;++ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ const uint32_t n_prefetch = kparams->n_prefetch;+ assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0);++ uint32_t n_k_tiles_a = ne10 / 32;++ for (uint32_t widx = 0; widx < n_weights; widx++) {+ const struct htp_tensor * restrict src_w = octx->src[widx];+ const struct htp_tensor * restrict dst = octx->dsts[widx];+ if (!src_w || !dst) continue;+ dma_queue * dma_q = octx->ctx->dma[ith];++ const uint32_t tile_size = htp_tensor_get_tile_size(src_w->type);+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src_w->type);+ const uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size;++ const uint32_t ne00 = src_w->ne[0];+ const uint32_t ne01 = src_w->ne[1];+ const size_t dst_row_size = dst->nb[1];+ const dma_addr_t src_w_row = src_w->data;++ uint32_t n_k_tiles_w = ne00 / 32;+ uint32_t tile_row_stride = n_k_tiles_w * tile_size;++ uint32_t src0_start_row = 0;+ uint32_t src0_end_row = ne01;+ if (octx->ctx->mdev.count > 1) {+ const bool can_split = htp_tensor_can_row_partition(dst, sizeof(float));+ const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(ne01, can_split ? 32 : 0,+ octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);+ src0_start_row = range.start;+ src0_end_row = range.start + range.count;+ }++ const uint32_t nrows = src0_end_row - src0_start_row;+ uint32_t src0_nrows_per_thread = fastdiv(nrows + nth - 1, &octx->n_threads_div);+ src0_nrows_per_thread = hex_round_up(src0_nrows_per_thread, 32);++ const uint32_t start_row = src0_start_row + src0_nrows_per_thread * ith;+ const uint32_t end_row = MIN(start_row + src0_nrows_per_thread, src0_end_row);+ if (start_row >= end_row) continue;++ uint32_t ct_start = start_row / 32;+ uint32_t ct_end = (end_row + 31) / 32;++ uint32_t push_ct = ct_start;+ for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) {+ dma_queue_push(dma_q, dma_make_data(vtcm_weight_ptr + d * tile_row_transfer_size_aligned,+ src_w_row + push_ct * tile_row_stride), aligned_tile_size, tile_size, tile_size, n_k_tiles_a);+ }++ for (uint32_t ct = ct_start; ct < ct_end; ct++) {+ const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;+ int valid_rows = (int)dst->ne[0] - (int)(ct * 32);+ valid_rows = MIN(32, MAX(0, valid_rows));++ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);+ uint32_t ir1 = 0;+ for (; ir1 + 1 < act_nrows; ir1 += 2) {+ const uint8_t * restrict act_col0 = (const uint8_t *) (act_data + (ir1+0) * act_stride);+ const uint8_t * restrict act_col1 = (const uint8_t *) (act_data + (ir1+1) * act_stride);++ float * restrict dst_row0 = (float *) (dst->data + ((ir1+0) * dst_row_size));+ float * restrict dst_row1 = (float *) (dst->data + ((ir1+1) * dst_row_size));+ float * dst_ptr0 = &dst_row0[ct * 32];+ float * dst_ptr1 = &dst_row1[ct * 32];++ mmctx->vec_dot_32x2(ne10, dst_ptr0, dst_ptr1, w_tile, act_col0, act_col1, valid_rows, NULL, NULL);+ }++ for (; ir1 < act_nrows; ++ir1) {+ const uint8_t * restrict act_col = (const uint8_t *) (act_data + ir1 * act_stride);+ float * restrict dst_row = (float *) (dst->data + (ir1 * dst_row_size));+ float * dst_ptr = &dst_row[ct * 32];+ mmctx->vec_dot_32x1(ne10, dst_ptr, w_tile, act_col, valid_rows, NULL);+ }+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct);++ if (push_ct < ct_end) {+ dma_queue_push(dma_q, dma_make_data(w_tile, src_w_row + push_ct * tile_row_stride),+ aligned_tile_size, tile_size, tile_size, n_k_tiles_a);+ push_ct++;+ }+ }+ }+}static void hvx_mm_transfer_act_dma(struct htp_ops_context * octx,@@ -649,12 +645,14 @@ static void name(unsigned int nth, unsigned int ith, void * data) {htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_A_QUANT, ir_first); \}-QUANTIZE_IMPL(quantize_f32_q8_0_tiled, "quantize-f32-q8_0_tiled", quantize_f32_q8_0_tiled_kernel, htp_mm_q8_0_tiled_row_size(ne0))-QUANTIZE_IMPL(quantize_f32_q8_1_tiled, "quantize-f32-q8_1_tiled", quantize_f32_q8_1_tiled_kernel, htp_mm_q8_1_tiled_row_size(ne0))+QUANTIZE_IMPL(quantize_f32_q8_0_tiled, "quantize-f32-q8_0_tiled", quantize_f32_q8_0_tiled_kernel, htp_mm_q8_0_tiled_row_size(ne0))+QUANTIZE_IMPL(quantize_f32_q8_1_tiled, "quantize-f32-q8_1_tiled", quantize_f32_q8_1_tiled_kernel, htp_mm_q8_1_tiled_row_size(ne0))QUANTIZE_IMPL(quantize_f32_q8_1_s16_tiled, "quantize-f32-q8_1_s16_tiled", quantize_f32_q8_1_s16_tiled_kernel, htp_mm_q8_1_tiled_row_size(ne0))-QUANTIZE_IMPL(quantize_f32_f32, "quantize-f32-f32", quantize_f32_f32_kernel, mmctx->vtcm_act_stride)-QUANTIZE_IMPL(quantize_f32_f16, "quantize-f32-f16", quantize_f32_f16_kernel, mmctx->vtcm_act_stride)-QUANTIZE_IMPL(quantize_f16_f16, "quantize-f16-f16", quantize_f16_f16_kernel, mmctx->vtcm_act_stride)+QUANTIZE_IMPL(quantize_f32_f16_tiled, "quantize-f32-f16_tiled", quantize_f32_f16_tiled_kernel, htp_mm_f16_tiled_row_size(ne0))+QUANTIZE_IMPL(quantize_f16_f16_tiled, "quantize-f16-f16_tiled", quantize_f16_f16_tiled_kernel, htp_mm_f16_tiled_row_size(ne0))+QUANTIZE_IMPL(quantize_f32_f32, "quantize-f32-f32", quantize_f32_f32_kernel, mmctx->vtcm_act_stride)+QUANTIZE_IMPL(quantize_f32_f16, "quantize-f32-f16", quantize_f32_f16_kernel, mmctx->vtcm_act_stride)+QUANTIZE_IMPL(quantize_f16_f16, "quantize-f16-f16", quantize_f16_f16_kernel, mmctx->vtcm_act_stride)static void quantize_f32_q8_0_tiled_block(unsigned int nth, unsigned int ith, void * data) {(void) nth;@@ -740,178 +738,209 @@ static void quantize_f32_q8_1_s16_tiled_block(unsigned int nth, unsigned int ithhtp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_A_QUANT, mmctx->quant_ib_first[ith]);}-// q8_1 for weight types with offsets (q8_1_s16 for Q2_K), otherwise q8_0-static inline worker_callback_t htp_mm_act_quant_row_func(int weight_type) {+static void quantize_f32_f16_tiled_block(unsigned int nth, unsigned int ith, void * data) {+ (void) nth;+ struct htp_mm_context * mmctx = data;+ if (mmctx->quant_ib_first[ith] >= mmctx->quant_ib_last[ith]) {+ return;+ }+ struct htp_ops_context * octx = mmctx->octx;+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_A_QUANT, mmctx->quant_ib_first[ith]);++ const struct htp_tensor * src = mmctx->act;++ quantize_f32_f16_tiled_block_kernel(+ (const float *) mmctx->vtcm_act_raw,+ mmctx->vtcm_act,+ NULL,+ src->ne[0],+ mmctx->quant_ib_first[ith],+ mmctx->quant_ib_last[ith],+ mmctx->vtcm_act_raw_stride,+ htp_mm_f16_tiled_row_size(src->ne[0]),+ mmctx->quant_r[ith],+ mmctx->quant_c[ith]+ );++ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_A_QUANT, mmctx->quant_ib_first[ith]);+}++static void quantize_f16_f16_tiled_block(unsigned int nth, unsigned int ith, void * data) {+ (void) nth;+ struct htp_mm_context * mmctx = data;+ if (mmctx->quant_ib_first[ith] >= mmctx->quant_ib_last[ith]) {+ return;+ }+ struct htp_ops_context * octx = mmctx->octx;+ struct htp_thread_trace * tr = &octx->ctx->trace[ith];+ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_A_QUANT, mmctx->quant_ib_first[ith]);++ const struct htp_tensor * src = mmctx->act;++ quantize_f16_f16_tiled_block_kernel(+ (const __fp16 *) mmctx->vtcm_act_raw,+ mmctx->vtcm_act,+ NULL,+ src->ne[0],+ mmctx->quant_ib_first[ith],+ mmctx->quant_ib_last[ith],+ mmctx->vtcm_act_raw_stride,+ htp_mm_f16_tiled_row_size(src->ne[0]),+ mmctx->quant_r[ith],+ mmctx->quant_c[ith]+ );++ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_A_QUANT, mmctx->quant_ib_first[ith]);+}++// q8_1 for weight types with offsets (q8_1_s16 for Q2_K), f16_tiled for F16, otherwise q8_0+static inline worker_callback_t htp_mm_act_quant_row_func(int weight_type, int act_type) {+ if (weight_type == HTP_TYPE_F16) {+ return (act_type == HTP_TYPE_F16) ? quantize_f16_f16_tiled : quantize_f32_f16_tiled;+ }if (weight_type == HTP_TYPE_Q2_K) {return quantize_f32_q8_1_s16_tiled;}return htp_mm_weight_has_offset(weight_type) ? quantize_f32_q8_1_tiled : quantize_f32_q8_0_tiled;}-static inline worker_callback_t htp_mm_act_quant_block_func(int weight_type) {+static inline worker_callback_t htp_mm_act_quant_block_func(int weight_type, int act_type) {+ if (weight_type == HTP_TYPE_F16) {+ return (act_type == HTP_TYPE_F16) ? quantize_f16_f16_tiled_block : quantize_f32_f16_tiled_block;+ }if (weight_type == HTP_TYPE_Q2_K) {return quantize_f32_q8_1_s16_tiled_block;}return htp_mm_weight_has_offset(weight_type) ? quantize_f32_q8_1_tiled_block : quantize_f32_q8_0_tiled_block;}-MATVEC_2D_REPACKED_IMPL(q4_0, 576, tiled_vec_dot_q4_0_32x1)-MATVEC_2D_REPACKED_IMPL(q4_1, 640, tiled_vec_dot_q4_1_32x1)-MATVEC_2D_REPACKED_IMPL(q8_0, 1088, tiled_vec_dot_q8_0_32x1)-MATVEC_2D_REPACKED_IMPL(q5_k, 768, tiled_vec_dot_q5_k_32x1)-MATVEC_2D_REPACKED_IMPL(q6_k, 896, tiled_vec_dot_q6_k_32x1)-MATVEC_2D_REPACKED_IMPL(q3_k, 512, tiled_vec_dot_q3_k_32x1)-MATVEC_2D_REPACKED_IMPL(q2_k, 336, tiled_vec_dot_q2_k_32x1)-MATVEC_2D_REPACKED_IMPL(iq4nl, 576, tiled_vec_dot_iq4nl_32x1)-MATVEC_2D_REPACKED_IMPL(mxfp4, 544, tiled_vec_dot_mxfp4_32x1)--MATMUL_NX_2D_REPACKED_IMPL(q4_0, 576, tiled_vec_dot_q4_0_32x2, tiled_vec_dot_q4_0_32x1)-MATMUL_NX_2D_REPACKED_IMPL(q4_1, 640, tiled_vec_dot_q4_1_32x2, tiled_vec_dot_q4_1_32x1)-MATMUL_NX_2D_REPACKED_IMPL(q8_0, 1088, tiled_vec_dot_q8_0_32x2, tiled_vec_dot_q8_0_32x1)-MATMUL_NX_2D_REPACKED_IMPL(iq4nl, 576, tiled_vec_dot_iq4nl_32x2, tiled_vec_dot_iq4nl_32x1)-MATMUL_NX_2D_REPACKED_IMPL(mxfp4, 544, tiled_vec_dot_mxfp4_32x2, tiled_vec_dot_mxfp4_32x1)-MATMUL_NX_2D_REPACKED_IMPL(q5_k, 768, tiled_vec_dot_q5_k_32x2, tiled_vec_dot_q5_k_32x1)-MATMUL_NX_2D_REPACKED_IMPL(q3_k, 512, tiled_vec_dot_q3_k_32x2, tiled_vec_dot_q3_k_32x1)-MATMUL_NX_2D_REPACKED_IMPL(q2_k, 336, tiled_vec_dot_q2_k_32x2, tiled_vec_dot_q2_k_32x1)--#define MATMUL_4D_REPACKED_IMPL(SUFFIX, TILE_SIZE, DOT_2X2, DOT_2X1) \-static void hvx_mm_4d_repacked_##SUFFIX(unsigned int nth, unsigned int ith, void * data) { \- htp_matmul_preamble; \- \- const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start; \- const uint32_t cur_m_rows = mmctx->cur_m_rows ? mmctx->cur_m_rows : (ne11 * ne12 * ne13); \- const uint32_t cur_m_start = mmctx->cur_m_start; \- \- const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith; \- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end); \- \- struct htp_thread_trace * tr = &octx->ctx->trace[ith]; \- \- const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params; \- const uint32_t n_prefetch = kparams->n_prefetch; \- assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0); \- \- const size_t dst_row_size = nb1; \- const size_t act_stride = mmctx->vtcm_act_stride; \- \- uint8_t * restrict vtcm_src0_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith; \- uint8_t * restrict act_data = mmctx->vtcm_act; \- \- const uint32_t tile_size = TILE_SIZE; \- const uint32_t aligned_tile_size = hex_align_up(tile_size, 128); \- \- const uint32_t n_k_tiles_w = ne00 / 32; \- const uint32_t n_k_tiles_a = ne10 / 32; \- const uint32_t tile_row_stride = n_k_tiles_w * tile_size; \- const uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size; \- const uint32_t src0_slice_stride = ((ne01 + 31) / 32) * tile_row_stride; \- \- const uint32_t ct_start = src0_start_row / 32; \- const uint32_t ct_end = (src0_end_row + 31) / 32; \- \- \- if (src0_start_row >= src0_end_row || cur_m_rows == 0) { \- return; \- } \- \- const uint32_t total_batches = ne12 * ne13; \- const uint32_t b_start = fastdiv(cur_m_start, &kparams->div_ne1); \- uint32_t b_end = fastdiv(cur_m_start + cur_m_rows + ne11 - 1, &kparams->div_ne1); \- b_end = MIN(b_end, total_batches); \- \- uint32_t b_grp_start = b_start; \- while (b_grp_start < b_end) { \- const uint32_t b3 = fastdiv(b_grp_start, &kparams->div_ne12); \- const uint32_t b2 = b_grp_start - b3 * ne12; \- const uint32_t i02 = fastdiv(b2, &kparams->div_r2); \- const uint32_t i03 = fastdiv(b3, &kparams->div_r3); \- \- uint32_t b_grp_end = b_grp_start + 1; \- while (b_grp_end < b_end) { \- const uint32_t cur_b3 = fastdiv(b_grp_end, &kparams->div_ne12); \- const uint32_t cur_b2 = b_grp_end - cur_b3 * ne12; \- const uint32_t cur_i02 = fastdiv(cur_b2, &kparams->div_r2); \- const uint32_t cur_i03 = fastdiv(cur_b3, &kparams->div_r3); \- if (cur_i02 != i02 || cur_i03 != i03) { \- break; \- } \- b_grp_end++; \- } \- \- const uint32_t slice_idx = i03 * ne02 + i02; \- const dma_addr_t src0_slice = src0->data + (size_t) slice_idx * src0_slice_stride; \- \- uint32_t push_ct = ct_start; \- for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) { \- dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + d * tile_row_transfer_size_aligned, \- src0_slice + (size_t) push_ct * tile_row_stride), \- aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \- } \- \- for (uint32_t ct = ct_start; ct < ct_end; ct++) { \- const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst; \- \- int valid_rows = (int)ne0 - (int)(ct * 32); \- valid_rows = MIN(32, MAX(0, valid_rows)); \- \- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct); \- for (uint32_t b = b_grp_start; b < b_grp_end; b++) { \- const uint32_t b_m_start = b * ne11; \- const uint32_t m_first = MAX(cur_m_start, b_m_start); \- const uint32_t m_last = MIN(cur_m_start + cur_m_rows, b_m_start + ne11); \- if (m_first >= m_last) continue; \- \- const uint32_t cur_b3 = fastdiv(b, &kparams->div_ne12); \- const uint32_t cur_b2 = b - cur_b3 * ne12; \- uint8_t * dst_batch_base = (uint8_t *) dst->data + (size_t) cur_b2 * nb2 + (size_t) cur_b3 * nb3; \- \- const uint32_t chunk_m_offset = m_first - cur_m_start; \- const uint32_t dst_m_offset = m_first - b_m_start; \- const uint32_t batch_nrows = m_last - m_first; \- \- uint32_t ir1 = 0; \- for (; ir1 + 1 < batch_nrows; ir1 += 2) { \- const uint8_t * restrict act_col0 = (const uint8_t *) (act_data + (chunk_m_offset + ir1 + 0) * act_stride); \- const uint8_t * restrict act_col1 = (const uint8_t *) (act_data + (chunk_m_offset + ir1 + 1) * act_stride); \- float * restrict dst_row0 = (float *) (dst_batch_base + (dst_m_offset + ir1 + 0) * dst_row_size); \- float * restrict dst_row1 = (float *) (dst_batch_base + (dst_m_offset + ir1 + 1) * dst_row_size); \- float * dst_ptr0 = &dst_row0[ct * 32]; \- float * dst_ptr1 = &dst_row1[ct * 32]; \- DOT_2X2(ne10, dst_ptr0, dst_ptr1, w_tile, act_col0, act_col1, valid_rows, NULL, NULL); \- } \- for (; ir1 < batch_nrows; ++ir1) { \- const uint8_t * restrict act_col = (const uint8_t *) (act_data + (chunk_m_offset + ir1) * act_stride); \- float * restrict dst_row = (float *) (dst_batch_base + (dst_m_offset + ir1) * dst_row_size); \- float * dst_ptr = &dst_row[ct * 32]; \- DOT_2X1(ne10, dst_ptr, w_tile, act_col, valid_rows, NULL); \- } \- } \- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct); \- \- if (push_ct < ct_end) { \- dma_queue_push(dma_q, dma_make_data(w_tile, src0_slice + (size_t) push_ct * tile_row_stride), \- aligned_tile_size, tile_size, tile_size, n_k_tiles_a); \- push_ct++; \- } \- } \- b_grp_start = b_grp_end; \- } \- if (src2) { \- hvx_tensor_add_f32_grid(dst, src2, cur_m_start, cur_m_start + cur_m_rows, src0_start_row, src0_end_row, &kparams->div_ne12_ne1, &kparams->div_ne1); \- } \+static void hvx_mm_4d_tiled(unsigned int nth, unsigned int ith, void * data) {+ htp_matmul_preamble;++ const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start;+ const uint32_t cur_m_rows = mmctx->cur_m_rows ? mmctx->cur_m_rows : (ne11 * ne12 * ne13);+ const uint32_t cur_m_start = mmctx->cur_m_start;++ const uint32_t src0_start_row = mmctx->src0_row_start + src0_nrows_per_thread * ith;+ const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, mmctx->src0_row_end);++ struct htp_thread_trace * tr = &octx->ctx->trace[ith];++ const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;+ const uint32_t n_prefetch = kparams->n_prefetch;+ assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0);++ const size_t dst_row_size = nb1;+ const size_t act_stride = mmctx->vtcm_act_stride;++ uint8_t * restrict vtcm_src0_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith;+ uint8_t * restrict act_data = mmctx->vtcm_act;++ const uint32_t tile_size = htp_tensor_get_tile_size(src0->type);+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src0->type);++ const uint32_t n_k_tiles_w = ne00 / 32;+ const uint32_t n_k_tiles_a = ne10 / 32;+ const uint32_t tile_row_stride = n_k_tiles_w * tile_size;+ const uint32_t tile_row_transfer_size_aligned = n_k_tiles_a * aligned_tile_size;+ const uint32_t src0_slice_stride = ((ne01 + 31) / 32) * tile_row_stride;++ const uint32_t ct_start = src0_start_row / 32;+ const uint32_t ct_end = (src0_end_row + 31) / 32;++ if (src0_start_row >= src0_end_row || cur_m_rows == 0) {+ return;+ }++ const uint32_t total_batches = ne12 * ne13;+ const uint32_t b_start = fastdiv(cur_m_start, &kparams->div_ne1);+ uint32_t b_end = fastdiv(cur_m_start + cur_m_rows + ne11 - 1, &kparams->div_ne1);+ b_end = MIN(b_end, total_batches);++ uint32_t b_grp_start = b_start;+ while (b_grp_start < b_end) {+ const uint32_t b3 = fastdiv(b_grp_start, &kparams->div_ne12);+ const uint32_t b2 = b_grp_start - b3 * ne12;+ const uint32_t i02 = fastdiv(b2, &kparams->div_r2);+ const uint32_t i03 = fastdiv(b3, &kparams->div_r3);++ uint32_t b_grp_end = b_grp_start + 1;+ while (b_grp_end < b_end) {+ const uint32_t cur_b3 = fastdiv(b_grp_end, &kparams->div_ne12);+ const uint32_t cur_b2 = b_grp_end - cur_b3 * ne12;+ const uint32_t cur_i02 = fastdiv(cur_b2, &kparams->div_r2);+ const uint32_t cur_i03 = fastdiv(cur_b3, &kparams->div_r3);+ if (cur_i02 != i02 || cur_i03 != i03) {+ break;+ }+ b_grp_end++;+ }++ const uint32_t slice_idx = i03 * ne02 + i02;+ const dma_addr_t src0_slice = src0->data + (size_t) slice_idx * src0_slice_stride;++ uint32_t push_ct = ct_start;+ for (uint32_t d = 0; d < n_prefetch && push_ct < ct_end; d++, push_ct++) {+ dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + d * tile_row_transfer_size_aligned,+ src0_slice + (size_t) push_ct * tile_row_stride),+ aligned_tile_size, tile_size, tile_size, n_k_tiles_a);+ }++ for (uint32_t ct = ct_start; ct < ct_end; ct++) {+ const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;++ int valid_rows = (int)ne0 - (int)(ct * 32);+ valid_rows = MIN(32, MAX(0, valid_rows));++ htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);+ for (uint32_t b = b_grp_start; b < b_grp_end; b++) {+ const uint32_t b_m_start = b * ne11;+ const uint32_t m_first = MAX(cur_m_start, b_m_start);+ const uint32_t m_last = MIN(cur_m_start + cur_m_rows, b_m_start + ne11);+ if (m_first >= m_last) continue;++ const uint32_t cur_b3 = fastdiv(b, &kparams->div_ne12);+ const uint32_t cur_b2 = b - cur_b3 * ne12;+ uint8_t * dst_batch_base = (uint8_t *) dst->data + (size_t) cur_b2 * nb2 + (size_t) cur_b3 * nb3;++ const uint32_t chunk_m_offset = m_first - cur_m_start;+ const uint32_t dst_m_offset = m_first - b_m_start;+ const uint32_t batch_nrows = m_last - m_first;++ uint32_t ir1 = 0;+ for (; ir1 + 1 < batch_nrows; ir1 += 2) {+ const uint8_t * restrict act_col0 = (const uint8_t *) (act_data + (chunk_m_offset + ir1 + 0) * act_stride);+ const uint8_t * restrict act_col1 = (const uint8_t *) (act_data + (chunk_m_offset + ir1 + 1) * act_stride);+ float * restrict dst_row0 = (float *) (dst_batch_base + (dst_m_offset + ir1 + 0) * dst_row_size);+ float * restrict dst_row1 = (float *) (dst_batch_base + (dst_m_offset + ir1 + 1) * dst_row_size);+ float * dst_ptr0 = &dst_row0[ct * 32];+ float * dst_ptr1 = &dst_row1[ct * 32];+ mmctx->vec_dot_32x2(ne10, dst_ptr0, dst_ptr1, w_tile, act_col0, act_col1, valid_rows, NULL, NULL);+ }+ for (; ir1 < batch_nrows; ++ir1) {+ const uint8_t * restrict act_col = (const uint8_t *) (act_data + (chunk_m_offset + ir1) * act_stride);+ float * restrict dst_row = (float *) (dst_batch_base + (dst_m_offset + ir1) * dst_row_size);+ float * dst_ptr = &dst_row[ct * 32];+ mmctx->vec_dot_32x1(ne10, dst_ptr, w_tile, act_col, valid_rows, NULL);+ }+ }+ htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ct);++ if (push_ct < ct_end) {+ dma_queue_push(dma_q, dma_make_data(w_tile, src0_slice + (size_t) push_ct * tile_row_stride),+ aligned_tile_size, tile_size, tile_size, n_k_tiles_a);+ push_ct++;+ }+ }+ b_grp_start = b_grp_end;+ }+ if (src2) {+ hvx_tensor_add_f32_grid(dst, src2, cur_m_start, cur_m_start + cur_m_rows, src0_start_row, src0_end_row, &kparams->div_ne12_ne1, &kparams->div_ne1);+ }}-MATMUL_4D_REPACKED_IMPL(q4_0, 576, tiled_vec_dot_q4_0_32x2, tiled_vec_dot_q4_0_32x1)-MATMUL_4D_REPACKED_IMPL(q4_1, 640, tiled_vec_dot_q4_1_32x2, tiled_vec_dot_q4_1_32x1)-MATMUL_4D_REPACKED_IMPL(q8_0, 1088, tiled_vec_dot_q8_0_32x2, tiled_vec_dot_q8_0_32x1)-MATMUL_4D_REPACKED_IMPL(q6_k, 896, tiled_vec_dot_q6_k_32x2, tiled_vec_dot_q6_k_32x1)-MATMUL_4D_REPACKED_IMPL(q5_k, 768, tiled_vec_dot_q5_k_32x2, tiled_vec_dot_q5_k_32x1)-MATMUL_4D_REPACKED_IMPL(q3_k, 512, tiled_vec_dot_q3_k_32x2, tiled_vec_dot_q3_k_32x1)-MATMUL_4D_REPACKED_IMPL(q2_k, 336, tiled_vec_dot_q2_k_32x2, tiled_vec_dot_q2_k_32x1)-MATMUL_4D_REPACKED_IMPL(iq4nl, 576, tiled_vec_dot_iq4nl_32x2, tiled_vec_dot_iq4nl_32x1)-MATMUL_4D_REPACKED_IMPL(mxfp4, 544, tiled_vec_dot_mxfp4_32x2, tiled_vec_dot_mxfp4_32x1)--static void hvx_mm_2d(unsigned int nth, unsigned int ith, void * data) {+static void hvx_mm_2d_flat(unsigned int nth, unsigned int ith, void * data) {htp_matmul_preamble;const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;@@ -1014,7 +1043,7 @@ static void hvx_mm_2d(unsigned int nth, unsigned int ith, void * data) {}}-static void hvx_mv_2d(unsigned int nth, unsigned int ith, void * data) {+static void hvx_mv_2d_flat(unsigned int nth, unsigned int ith, void * data) {htp_matmul_preamble;const uint32_t src0_nrows = mmctx->src0_row_end - mmctx->src0_row_start;@@ -1119,7 +1148,7 @@ static void hvx_mv_2d(unsigned int nth, unsigned int ith, void * data) {}}-static void hvx_mm_4d(unsigned int nth, unsigned int ith, void * data) {+static void hvx_mm_4d_flat(unsigned int nth, unsigned int ith, void * data) {htp_matmul_preamble;const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;@@ -1268,7 +1297,7 @@ static void hvx_mm_4d(unsigned int nth, unsigned int ith, void * data) {#define MMID_MATRIX_ROW(row_id, i1) matrix_rows[(row_id) * mmctx->mapping_stride + (i1)]-static void hvx_mm_id(unsigned int nth, unsigned int ith, void * data) {+static void hvx_mm_id_tiled(unsigned int nth, unsigned int ith, void * data) {htp_matmul_preamble;const struct htp_tensor * restrict ids = octx->src[2];@@ -1312,8 +1341,8 @@ static void hvx_mm_id(unsigned int nth, unsigned int ith, void * data) {const dma_addr_t src0_row = src0->data + cur_a * nb02;- const uint32_t tile_size = htp_mm_get_weight_tile_size(src0->type);- const uint32_t aligned_tile_size = htp_mm_get_weight_aligned_tile_size(src0->type);+ const uint32_t tile_size = htp_tensor_get_tile_size(src0->type);+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src0->type);const uint32_t n_k_tiles_w = ne00 / 32;const uint32_t n_k_tiles_a = ne10 / 32;const uint32_t tile_row_stride = n_k_tiles_w * tile_size;@@ -1331,7 +1360,7 @@ static void hvx_mm_id(unsigned int nth, unsigned int ith, void * data) {for (uint32_t ct = ct_start; ct < ct_end; ct++) {const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;- int valid_rows = (int)ne01 - (int)(ct * 32);+ int valid_rows = (int)ne0 - (int)(ct * 32);valid_rows = MIN(32, MAX(0, valid_rows));htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);@@ -1357,7 +1386,7 @@ static void hvx_mm_id(unsigned int nth, unsigned int ith, void * data) {}}-static void hvx_mv_id(unsigned int nth, unsigned int ith, void * data) {+static void hvx_mv_id_tiled(unsigned int nth, unsigned int ith, void * data) {htp_matmul_preamble;const struct htp_tensor * restrict ids = octx->src[2];@@ -1400,8 +1429,8 @@ static void hvx_mv_id(unsigned int nth, unsigned int ith, void * data) {const uint8_t * restrict act_col = (const uint8_t *) act_data;float * restrict dst_row = (float *) (dst->data + ie1 * nb1);- const uint32_t tile_size = htp_mm_get_weight_tile_size(src0->type);- const uint32_t aligned_tile_size = htp_mm_get_weight_aligned_tile_size(src0->type);+ const uint32_t tile_size = htp_tensor_get_tile_size(src0->type);+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src0->type);const uint32_t n_k_tiles_w = ne00 / 32;const uint32_t n_k_tiles_a = ne10 / 32;const uint32_t tile_row_stride = n_k_tiles_w * tile_size;@@ -1419,7 +1448,7 @@ static void hvx_mv_id(unsigned int nth, unsigned int ith, void * data) {for (uint32_t ct = ct_start; ct < ct_end; ct++) {const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;- int valid_rows = (int)ne01 - (int)(ct * 32);+ int valid_rows = (int)ne0 - (int)(ct * 32);valid_rows = MIN(32, MAX(0, valid_rows));htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);@@ -1435,7 +1464,7 @@ static void hvx_mv_id(unsigned int nth, unsigned int ith, void * data) {}}-static void hvx_mv_id_nx(unsigned int nth, unsigned int ith, void * data) {+static void hvx_mv_id_nx_tiled(unsigned int nth, unsigned int ith, void * data) {struct htp_mm_context * mmctx = (struct htp_mm_context *) data;struct htp_ops_context * octx = mmctx->octx;const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;@@ -1489,8 +1518,8 @@ static void hvx_mv_id_nx(unsigned int nth, unsigned int ith, void * data) {const uint8_t * restrict act_col = (const uint8_t *) act_data;float * restrict dst_row = (float *) (dst->data + ie1 * dst->nb[1]);- const uint32_t tile_size = htp_mm_get_weight_tile_size(src_w->type);- const uint32_t aligned_tile_size = htp_mm_get_weight_aligned_tile_size(src_w->type);+ const uint32_t tile_size = htp_tensor_get_tile_size(src_w->type);+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src_w->type);const uint32_t n_k_tiles_w = src_w->ne[0] / 32;const uint32_t n_k_tiles_a = act->ne[0] / 32;const uint32_t tile_row_stride = n_k_tiles_w * tile_size;@@ -1508,7 +1537,7 @@ static void hvx_mv_id_nx(unsigned int nth, unsigned int ith, void * data) {for (uint32_t ct = ct_start; ct < ct_end; ct++) {const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;- int valid_rows = (int)src_w->ne[1] - (int)(ct * 32);+ int valid_rows = (int)dst->ne[0] - (int)(ct * 32);valid_rows = MIN(32, MAX(0, valid_rows));htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);@@ -1525,7 +1554,7 @@ static void hvx_mv_id_nx(unsigned int nth, unsigned int ith, void * data) {}}-static void hvx_mm_id_nx(unsigned int nth, unsigned int ith, void * data) {+static void hvx_mm_id_nx_tiled(unsigned int nth, unsigned int ith, void * data) {struct htp_mm_context * mmctx = (struct htp_mm_context *) data;struct htp_ops_context * octx = mmctx->octx;const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;@@ -1580,8 +1609,8 @@ static void hvx_mm_id_nx(unsigned int nth, unsigned int ith, void * data) {const dma_addr_t src0_row = src_w->data + cur_a * src_w->nb[2];- const uint32_t tile_size = htp_mm_get_weight_tile_size(src_w->type);- const uint32_t aligned_tile_size = htp_mm_get_weight_aligned_tile_size(src_w->type);+ const uint32_t tile_size = htp_tensor_get_tile_size(src_w->type);+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(src_w->type);const uint32_t n_k_tiles_w = src_w->ne[0] / 32;const uint32_t n_k_tiles_a = act->ne[0] / 32;const uint32_t tile_row_stride = n_k_tiles_w * tile_size;@@ -1599,7 +1628,7 @@ static void hvx_mm_id_nx(unsigned int nth, unsigned int ith, void * data) {for (uint32_t ct = ct_start; ct < ct_end; ct++) {const uint8_t * w_tile = (void *) dma_queue_pop(dma_q).dst;- int valid_rows = (int)src_w->ne[1] - (int)(ct * 32);+ int valid_rows = (int)dst->ne[0] - (int)(ct * 32);valid_rows = MIN(32, MAX(0, valid_rows));htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ct);@@ -1626,44 +1655,58 @@ static void hvx_mm_id_nx(unsigned int nth, unsigned int ith, void * data) {}}-static int hvx_mm_init_vec_dot(struct htp_mm_context * mmctx, enum htp_data_type type) {+static int hvx_mm_init_tiled_vec_dot(struct htp_mm_context * mmctx, enum htp_data_type type) {switch (type) {case HTP_TYPE_Q4_0:mmctx->type = "q4_0_tiled-f32";mmctx->vec_dot_32x1 = tiled_vec_dot_q4_0_32x1;+ mmctx->vec_dot_32x2 = tiled_vec_dot_q4_0_32x2;return 0;case HTP_TYPE_Q4_1:case HTP_TYPE_Q4_K:mmctx->type = "q4_1_tiled-f32";mmctx->vec_dot_32x1 = tiled_vec_dot_q4_1_32x1;+ mmctx->vec_dot_32x2 = tiled_vec_dot_q4_1_32x2;return 0;case HTP_TYPE_Q8_0:mmctx->type = "q8_0_tiled-f32";mmctx->vec_dot_32x1 = tiled_vec_dot_q8_0_32x1;+ mmctx->vec_dot_32x2 = tiled_vec_dot_q8_0_32x2;return 0;case HTP_TYPE_Q5_K:mmctx->type = "q5_k_tiled-f32";mmctx->vec_dot_32x1 = tiled_vec_dot_q5_k_32x1;+ mmctx->vec_dot_32x2 = tiled_vec_dot_q5_k_32x2;return 0;case HTP_TYPE_Q6_K:mmctx->type = "q6_k_tiled-f32";mmctx->vec_dot_32x1 = tiled_vec_dot_q6_k_32x1;+ mmctx->vec_dot_32x2 = tiled_vec_dot_q6_k_32x2;return 0;case HTP_TYPE_Q3_K:mmctx->type = "q3_k_tiled-f32";mmctx->vec_dot_32x1 = tiled_vec_dot_q3_k_32x1;+ mmctx->vec_dot_32x2 = tiled_vec_dot_q3_k_32x2;return 0;case HTP_TYPE_Q2_K:mmctx->type = "q2_k_tiled-f32";mmctx->vec_dot_32x1 = tiled_vec_dot_q2_k_32x1;+ mmctx->vec_dot_32x2 = tiled_vec_dot_q2_k_32x2;return 0;case HTP_TYPE_IQ4_NL:mmctx->type = "iq4nl_tiled-f32";mmctx->vec_dot_32x1 = tiled_vec_dot_iq4nl_32x1;+ mmctx->vec_dot_32x2 = tiled_vec_dot_iq4nl_32x2;return 0;case HTP_TYPE_MXFP4:mmctx->type = "mxfp4_tiled-f32";mmctx->vec_dot_32x1 = tiled_vec_dot_mxfp4_32x1;+ mmctx->vec_dot_32x2 = tiled_vec_dot_mxfp4_32x2;+ return 0;+ case HTP_TYPE_F16:+ mmctx->type = "f16_tiled-f16";+ mmctx->vec_dot_32x1 = tiled_vec_dot_f16_32x1;+ mmctx->vec_dot_32x2 = tiled_vec_dot_f16_32x2;return 0;default:return -1;@@ -1705,15 +1748,11 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {mmctx->src0_row_start = src0_row_start;mmctx->src0_row_end = src0_row_end;- bool is_repacked = (src0->type == HTP_TYPE_Q4_0 || src0->type == HTP_TYPE_Q4_1 ||- src0->type == HTP_TYPE_Q8_0 || src0->type == HTP_TYPE_IQ4_NL ||- src0->type == HTP_TYPE_MXFP4 || src0->type == HTP_TYPE_Q6_K ||- src0->type == HTP_TYPE_Q4_K || src0->type == HTP_TYPE_Q5_K ||- src0->type == HTP_TYPE_Q3_K || src0->type == HTP_TYPE_Q2_K);+ const bool is_tiled = htp_tensor_is_tiled(src0);// Compute src0_nrows_per_threadmmctx->src0_nrows_per_thread = fastdiv(nrows + octx->n_threads - 1, &octx->n_threads_div);- if (is_repacked) {+ if (is_tiled) {mmctx->src0_nrows_per_thread = hex_round_up(mmctx->src0_nrows_per_thread, 32);} else {mmctx->src0_nrows_per_thread += (mmctx->src0_nrows_per_thread & 1); // round up to even@@ -1730,59 +1769,11 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {uint32_t n_quant_tasks = 1;const bool is_batched = (ne12 > 1 || ne13 > 1 || ne02 > 1 || ne03 > 1);if (is_batched) {- if (is_repacked) {- switch (src0->type) {- case HTP_TYPE_Q4_0: matmul_job_func = hvx_mm_4d_repacked_q4_0; break;- case HTP_TYPE_Q4_1:- case HTP_TYPE_Q4_K: matmul_job_func = hvx_mm_4d_repacked_q4_1; break;- case HTP_TYPE_Q8_0: matmul_job_func = hvx_mm_4d_repacked_q8_0; break;- case HTP_TYPE_Q6_K: matmul_job_func = hvx_mm_4d_repacked_q6_k; break;- case HTP_TYPE_Q5_K: matmul_job_func = hvx_mm_4d_repacked_q5_k; break;- case HTP_TYPE_Q3_K: matmul_job_func = hvx_mm_4d_repacked_q3_k; break;- case HTP_TYPE_Q2_K: matmul_job_func = hvx_mm_4d_repacked_q2_k; break;- case HTP_TYPE_IQ4_NL: matmul_job_func = hvx_mm_4d_repacked_iq4nl; break;- case HTP_TYPE_MXFP4: matmul_job_func = hvx_mm_4d_repacked_mxfp4; break;- default: return HTP_STATUS_NO_SUPPORT;- }- } else {- matmul_job_func = hvx_mm_4d;- }+ matmul_job_func = is_tiled ? hvx_mm_4d_tiled : hvx_mm_4d_flat;} else if (act_nrows > 1) {- if (is_repacked) {- switch (src0->type) {- case HTP_TYPE_Q4_0: matmul_job_func = hvx_mm_2d_repacked_q4_0; break;- case HTP_TYPE_Q4_1:- case HTP_TYPE_Q4_K: matmul_job_func = hvx_mm_2d_repacked_q4_1; break;- case HTP_TYPE_Q8_0: matmul_job_func = hvx_mm_2d_repacked_q8_0; break;- case HTP_TYPE_Q6_K: matmul_job_func = hvx_mm_2d_repacked_q6_k; break;- case HTP_TYPE_Q5_K: matmul_job_func = hvx_mm_2d_repacked_q5_k; break;- case HTP_TYPE_Q3_K: matmul_job_func = hvx_mm_2d_repacked_q3_k; break;- case HTP_TYPE_Q2_K: matmul_job_func = hvx_mm_2d_repacked_q2_k; break;- case HTP_TYPE_IQ4_NL: matmul_job_func = hvx_mm_2d_repacked_iq4nl; break;- case HTP_TYPE_MXFP4: matmul_job_func = hvx_mm_2d_repacked_mxfp4; break;- default: return HTP_STATUS_NO_SUPPORT;- }- } else {- matmul_job_func = hvx_mm_2d;- }+ matmul_job_func = is_tiled ? hvx_mm_2d_tiled : hvx_mm_2d_flat;} else {- if (is_repacked) {- switch (src0->type) {- case HTP_TYPE_Q4_0: matmul_job_func = hvx_mv_2d_repacked_q4_0; break;- case HTP_TYPE_Q4_1:- case HTP_TYPE_Q4_K: matmul_job_func = hvx_mv_2d_repacked_q4_1; break;- case HTP_TYPE_Q8_0: matmul_job_func = hvx_mv_2d_repacked_q8_0; break;- case HTP_TYPE_Q5_K: matmul_job_func = hvx_mv_2d_repacked_q5_k; break;- case HTP_TYPE_Q6_K: matmul_job_func = hvx_mv_2d_repacked_q6_k; break;- case HTP_TYPE_Q3_K: matmul_job_func = hvx_mv_2d_repacked_q3_k; break;- case HTP_TYPE_Q2_K: matmul_job_func = hvx_mv_2d_repacked_q2_k; break;- case HTP_TYPE_IQ4_NL: matmul_job_func = hvx_mv_2d_repacked_iq4nl; break;- case HTP_TYPE_MXFP4: matmul_job_func = hvx_mv_2d_repacked_mxfp4; break;- default: return HTP_STATUS_NO_SUPPORT;- }- } else {- matmul_job_func = hvx_mv_2d;- }+ matmul_job_func = is_tiled ? hvx_mv_2d_tiled : hvx_mv_2d_flat;}bool need_quant = true;@@ -1792,9 +1783,9 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {quant_task_func = (act->type == HTP_TYPE_F32) ? quantize_f32_f16 : quantize_f16_f16;need_quant = (act->type == HTP_TYPE_F32);mmctx->type = (act->type == HTP_TYPE_F32) ? "f32-f16" : "f16-f16";- mmctx->vec_dot_1x1 = vec_dot_f16_f16_aa_1x1;- mmctx->vec_dot_2x1 = vec_dot_f16_f16_aa_2x1;- mmctx->vec_dot_2x2 = vec_dot_f16_f16_aa_2x2;+ mmctx->vec_dot_1x1 = flat_vec_dot_f16_f16_aa_1x1;+ mmctx->vec_dot_2x1 = flat_vec_dot_f16_f16_aa_2x1;+ mmctx->vec_dot_2x2 = flat_vec_dot_f16_f16_aa_2x2;act_row_size = kparams->act_row_size;break;@@ -1802,16 +1793,26 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {quant_task_func = NULL;need_quant = false;mmctx->type = "f32-f32";- mmctx->vec_dot_1x1 = vec_dot_f32_f32_aa_1x1;- mmctx->vec_dot_2x1 = vec_dot_f32_f32_aa_2x1;- mmctx->vec_dot_2x2 = vec_dot_f32_f32_aa_2x2;+ mmctx->vec_dot_1x1 = flat_vec_dot_f32_f32_aa_1x1;+ mmctx->vec_dot_2x1 = flat_vec_dot_f32_f32_aa_2x1;+ mmctx->vec_dot_2x2 = flat_vec_dot_f32_f32_aa_2x2;+ act_row_size = kparams->act_row_size;+ break;++ case HTP_MM_KERNEL_HVX_BF16_F32_VTCM:+ quant_task_func = NULL;+ need_quant = false;+ mmctx->type = "bf16-f32";+ mmctx->vec_dot_1x1 = flat_vec_dot_bf16_f32_aa_1x1;+ mmctx->vec_dot_2x1 = flat_vec_dot_bf16_f32_aa_2x1;+ mmctx->vec_dot_2x2 = flat_vec_dot_bf16_f32_aa_2x2;act_row_size = kparams->act_row_size;break;case HTP_MM_KERNEL_HVX_QUANT_BLOCK:case HTP_MM_KERNEL_HVX_QUANT_ROW:- default:- if (hvx_mm_init_vec_dot(mmctx, src0->type) != 0) {+ default: {+ if (hvx_mm_init_tiled_vec_dot(mmctx, src0->type) != 0) {return HTP_STATUS_NO_SUPPORT;}@@ -1821,7 +1822,7 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {if (act_nrows < octx->n_threads && !is_batched) {n_quant_tasks = MIN(total_nb, octx->n_threads);- quant_task_func = htp_mm_act_quant_block_func(src0->type);+ quant_task_func = htp_mm_act_quant_block_func(src0->type, act->type);for (uint32_t ith = 0; ith < n_quant_tasks; ++ith) {uint32_t ib_first = (total_nb * ith) / n_quant_tasks;uint32_t ib_last = (total_nb * (ith + 1)) / n_quant_tasks;@@ -1832,10 +1833,11 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {}} else {n_quant_tasks = MIN(act_nrows, octx->n_threads);- quant_task_func = htp_mm_act_quant_row_func(src0->type);+ quant_task_func = htp_mm_act_quant_row_func(src0->type, act->type);}act_row_size = kparams->act_row_size;break;+ }}const uint32_t m_chunk = (kparams->m_chunk > 0 && (uint32_t) kparams->m_chunk < act_nrows)@@ -1848,6 +1850,7 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {if (kparams->kernel_type == HTP_MM_KERNEL_HVX_F16_F16_VTCM ||kparams->kernel_type == HTP_MM_KERNEL_HVX_F32_F32_VTCM ||+ kparams->kernel_type == HTP_MM_KERNEL_HVX_BF16_F32_VTCM ||kparams->kernel_type == HTP_MM_KERNEL_HVX_QUANT_ROW ||kparams->kernel_type == HTP_MM_KERNEL_HVX_QUANT_BLOCK) {mmctx->vtcm_act_size_per_thread = L.act_bytes;@@ -1908,7 +1911,7 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {work_queue_func_t q_func;if (cur_m_rows < octx->n_threads && (kparams->kernel_type == HTP_MM_KERNEL_HVX_QUANT_BLOCK || kparams->kernel_type == HTP_MM_KERNEL_HVX_QUANT_ROW)) {quant_tasks = MIN(total_nb, octx->n_threads);- q_func = htp_mm_act_quant_block_func(src0->type);+ q_func = htp_mm_act_quant_block_func(src0->type, act->type);for (uint32_t ith = 0; ith < quant_tasks; ++ith) {uint32_t ib_first = (total_nb * ith) / quant_tasks;uint32_t ib_last = (total_nb * (ith + 1)) / quant_tasks;@@ -1949,158 +1952,18 @@ static int hvx_mm_matmul(struct htp_ops_context * octx) {return HTP_STATUS_OK;}-static void hvx_mm_nx_2d(unsigned int nth, unsigned int ith, void * data) {- struct htp_mm_context * mmctx = data;- struct htp_ops_context * octx = mmctx->octx;- const struct htp_mm_kernel_params * kparams = (const struct htp_mm_kernel_params *) octx->kernel_params;- const uint32_t n_weights = kparams->n_weights;-- const struct htp_tensor * restrict act = octx->src[n_weights];- const uint32_t act_nrows = act->ne[1] * act->ne[2] * act->ne[3];- const size_t act_stride = mmctx->vtcm_act_stride;-- uint8_t * restrict vtcm_src0_ptr = mmctx->vtcm_src0 + mmctx->vtcm_src0_size_per_thread * ith;- uint8_t * restrict act_data = mmctx->vtcm_act;-- const uint32_t n_prefetch = kparams->n_prefetch;- assert(n_prefetch >= 2 && n_prefetch <= HTP_MM_MAX_PREFETCH && (n_prefetch & (n_prefetch - 1)) == 0);- const uint32_t prefetch_mask = n_prefetch - 1;-- struct htp_thread_trace * tr = &octx->ctx->trace[ith];--- for (uint32_t widx = 0; widx < n_weights; widx++) {- const struct htp_tensor * restrict src_w = octx->src[widx];- const struct htp_tensor * restrict dst = octx->dsts[widx];- if (!src_w || !dst) continue;- dma_queue * dma_q = octx->ctx->dma[ith];-- const uint32_t ne00 = src_w->ne[0];- const uint32_t ne01 = src_w->ne[1];- uint32_t start_row = 0;- uint32_t end_row = ne01;- if (octx->ctx->mdev.count > 1) {- const bool can_split = htp_tensor_can_row_partition(dst, sizeof(float));- const struct htp_tensor_mdev_range range = htp_tensor_mdev_partition(ne01, can_split ? 32 : 0, octx->ctx->mdev.idx, octx->ctx->mdev.count, &octx->ctx->mdev.count_div);- start_row = range.start;- end_row = range.start + range.count;- }-- const uint32_t nrows = end_row - start_row;- uint32_t src0_nrows_per_thread = fastdiv(nrows + nth - 1, &octx->n_threads_div);- src0_nrows_per_thread += (src0_nrows_per_thread & 1);-- const uint32_t src0_start_row = start_row + src0_nrows_per_thread * ith;- const uint32_t src0_end_row = MIN(src0_start_row + src0_nrows_per_thread, end_row);- const uint32_t src0_end_row_x2 = src0_start_row + ((src0_end_row - src0_start_row) & ~1U);- if (src0_start_row >= src0_end_row) continue;-- const size_t dst_row_size = dst->nb[1];- const size_t src0_row_size = src_w->nb[1];- const size_t src0_stride = hex_round_up(src0_row_size, 128);-- const dma_addr_t src0_row = src_w->data;-- for (uint32_t ir0 = src0_start_row; ir0 < src0_end_row_x2; ir0 += 2) {- const int is0 = (ir0 - src0_start_row);- if (is0 >= (int)n_prefetch) break;- dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + is0 * src0_stride, src0_row + ir0 * src0_row_size),- src0_stride, src0_row_size, src0_row_size, 2);- }-- for (uint32_t ir0 = src0_start_row; ir0 < src0_end_row_x2; ir0 += 2) {- const uint8_t * ss0 = (void *) dma_queue_pop(dma_q).dst;- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ir0);- uint32_t ir1 = 0;- for (; ir1 + 1 < act_nrows; ir1 += 2) {- const uint8_t * restrict act_col0 = (const uint8_t *) (act_data + (ir1+0) * act_stride);- const uint8_t * restrict act_col1 = (const uint8_t *) (act_data + (ir1+1) * act_stride);- float * restrict dst_row0 = (float *) (dst->data + ((ir1+0) * dst_row_size));- float * restrict dst_row1 = (float *) (dst->data + ((ir1+1) * dst_row_size));- mmctx->vec_dot_2x2(ne00, &dst_row0[ir0], &dst_row1[ir0], ss0, ss0 + src0_stride, act_col0, act_col1);- }- for (; ir1 < act_nrows; ++ir1) {- const uint8_t * restrict act_col = (const uint8_t *) (act_data + ir1 * act_stride);- float * restrict dst_row = (float *) (dst->data + (ir1 * dst_row_size));- mmctx->vec_dot_2x1(ne00, &dst_row[ir0], ss0, ss0 + src0_stride, act_col);- }- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ir0);-- const int pr0 = (ir0 + n_prefetch);- const int is0 = (pr0 - src0_start_row) & prefetch_mask;- if (pr0 < src0_end_row_x2) {- dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + is0 * src0_stride, src0_row + pr0 * src0_row_size),- src0_stride, src0_row_size, src0_row_size, 2);- }- }-- if (src0_end_row != src0_end_row_x2) {- uint32_t ir0 = src0_end_row_x2;- const int is0 = (ir0 - src0_start_row) & prefetch_mask;- dma_queue_push(dma_q, dma_make_data(vtcm_src0_ptr + is0 * src0_stride, src0_row + ir0 * src0_row_size),- src0_stride, src0_row_size, src0_row_size, 1);- const uint8_t * ss0 = (void *) dma_queue_pop(dma_q).dst;- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_COMP, ir0);- for (uint32_t ir1 = 0; ir1 < act_nrows; ++ir1) {- const uint8_t * restrict act_col = (const uint8_t *) (act_data + ir1 * act_stride);- float * restrict dst_row = (float *) (dst->data + (ir1 * dst_row_size));- mmctx->vec_dot_1x1(ne00, &dst_row[ir0], ss0, act_col);- }- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_COMP, ir0);- }- }-}--#define DEQUANTIZE_WORKER_LOOP_IMPL(SUFFIX) \-static void dequantize_tiled_worker_loop_##SUFFIX(unsigned int n, unsigned int i, void *data) { \- tiled_dequantize_state_t *state = (tiled_dequantize_state_t *)data; \- struct htp_thread_trace * tr = &state->traces[i]; \- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_W_DEQUANT, i); \- for (unsigned int task_id = i; task_id < (unsigned int)state->n_tasks; task_id += n) { \- int start = task_id * state->n_tiles_per_task; \- int end = hex_smin(start + state->n_tiles_per_task, state->n_tot_tiles); \- dequantize_tiled_weight_to_fp16_task_##SUFFIX(state, start, end); \- } \- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_W_DEQUANT, i); \-}--DEQUANTIZE_WORKER_LOOP_IMPL(q4_0)-DEQUANTIZE_WORKER_LOOP_IMPL(q4_1)-DEQUANTIZE_WORKER_LOOP_IMPL(iq4_nl)-DEQUANTIZE_WORKER_LOOP_IMPL(mxfp4)-DEQUANTIZE_WORKER_LOOP_IMPL(q8_0)-DEQUANTIZE_WORKER_LOOP_IMPL(q6_k)-DEQUANTIZE_WORKER_LOOP_IMPL(q5_k)-DEQUANTIZE_WORKER_LOOP_IMPL(q3_k)-DEQUANTIZE_WORKER_LOOP_IMPL(q2_k)--static void convert_f16_worker_loop(unsigned int n, unsigned int i, void *data) {- tiled_dequantize_state_t *state = (tiled_dequantize_state_t *)data;+static void dequantize_tiled_worker_loop(unsigned int n, unsigned int i, void * data) {+ tiled_dequantize_state_t * state = (tiled_dequantize_state_t *) data;struct htp_thread_trace * tr = &state->traces[i];htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_W_DEQUANT, i);- for (unsigned int task_id = i; task_id < (unsigned int)state->n_tasks; task_id += n) {+ for (unsigned int task_id = i; task_id < (unsigned int) state->n_tasks; task_id += n) {int start = task_id * state->n_tiles_per_task;int end = hex_smin(start + state->n_tiles_per_task, state->n_tot_tiles);- convert_f16_weight_to_fp16_tiles_task(state, start, end);+ state->task_fn(state, start, end);}htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_W_DEQUANT, i);}-static void quantize_f32_worker_loop(unsigned int n, unsigned int i, void *data) {- tiled_dequantize_state_t *state = (tiled_dequantize_state_t *)data;-- struct htp_thread_trace * tr = &state->traces[i];- htp_trace_event_start(tr, HTP_TRACE_EVT_HVX_A_QUANT, i);-- for (unsigned int task_id = i; task_id < (unsigned int)state->n_tasks; task_id += n) {- int start = task_id * state->n_tiles_per_task;- int end = hex_smin(start + state->n_tiles_per_task, state->n_tot_tiles);- quantize_f32_weight_to_fp16_tiles_task(state, start, end);- }-- htp_trace_event_stop(tr, HTP_TRACE_EVT_HVX_A_QUANT, i);-}-static void transfer_output_chunk_worker_fn(unsigned int n, unsigned int i, void *data) {output_transfer_task_state_t *st = (output_transfer_task_state_t *) data;@@ -2450,13 +2313,36 @@ static void transfer_output_chunk_scattered_worker_fn(unsigned int n, unsigned i// --- HMX Dispatchers & Entry Points ---+static inline tiled_dequant_task_fn_t hmx_get_dequant_worker_fn(int weight_type, bool is_tiled) {+ switch (weight_type) {+ case HTP_TYPE_Q4_0: return dequantize_tiled_weight_to_fp16_task_q4_0;+ case HTP_TYPE_IQ4_NL: return dequantize_tiled_weight_to_fp16_task_iq4_nl;+ case HTP_TYPE_Q4_1:+ case HTP_TYPE_Q4_K: return dequantize_tiled_weight_to_fp16_task_q4_1;+ case HTP_TYPE_MXFP4: return dequantize_tiled_weight_to_fp16_task_mxfp4;+ case HTP_TYPE_Q8_0: return dequantize_tiled_weight_to_fp16_task_q8_0;+ case HTP_TYPE_Q5_K: return dequantize_tiled_weight_to_fp16_task_q5_k;+ case HTP_TYPE_Q6_K: return dequantize_tiled_weight_to_fp16_task_q6_k;+ case HTP_TYPE_Q3_K: return dequantize_tiled_weight_to_fp16_task_q3_k;+ case HTP_TYPE_Q2_K: return dequantize_tiled_weight_to_fp16_task_q2_k;+ case HTP_TYPE_F16: return is_tiled ? NULL : convert_f16_weight_to_fp16_tiles_task;+ case HTP_TYPE_BF16: return convert_bf16_weight_to_fp16_tiles_task;+ case HTP_TYPE_F32: return quantize_f32_weight_to_fp16_tiles_task;+ default: return NULL;+ }+}+static void dequantize_tiled_weight_chunk_to_fp16_tiles(struct htp_context *ctx, __fp16 *vtcm_dst,const void *weight_src_ddr,int n_cols, int k_block,size_t row_stride, int weight_type,int n_k_tiles, struct fastdiv_values n_k_tiles_div,- worker_callback_t dequant_worker_fn, int n_threads) {+ tiled_dequant_task_fn_t dequant_worker_fn, int n_threads) {++ if (!dequant_worker_fn) {+ return;+ }assert(k_block % HTP_MM_HMX_TILE_N_COLS == 0);@@ -2471,6 +2357,7 @@ static void dequantize_tiled_weight_chunk_to_fp16_tiles(state.n_tiles_per_task = n_tiles_per_task;state.dst = vtcm_dst;state.src = (const uint8_t *)weight_src_ddr;+ state.task_fn = dequant_worker_fn;state.n_cols = n_cols;state.k_block = k_block;state.row_stride = row_stride;@@ -2480,14 +2367,14 @@ static void dequantize_tiled_weight_chunk_to_fp16_tiles(state.traces = ctx->trace;state.ctx = ctx;- state.tile_size = htp_mm_get_weight_tile_size(weight_type);- state.aligned_tile_size = htp_mm_get_weight_aligned_tile_size(weight_type);+ state.tile_size = htp_tensor_get_tile_size(weight_type);+ state.aligned_tile_size = htp_tensor_get_tile_padded_size(weight_type);if (state.n_tasks == 1 || n_threads == 1) {- dequant_worker_fn(1, 0, &state);+ dequantize_tiled_worker_loop(1, 0, &state);} else {int n_tasks = hex_smin((int) state.n_tasks, n_threads);- worker_pool_run_func(ctx->worker_pool, dequant_worker_fn, &state, n_tasks);+ worker_pool_run_func(ctx->worker_pool, dequantize_tiled_worker_loop, &state, n_tasks);}}@@ -2764,53 +2651,35 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,struct htp_thread_trace * tr = &ctx->trace[0];htp_trace_event_start(tr, HTP_TRACE_EVT_INIT, 0);- // Quantized weights are repacked and padded to 32, so we it has to be 32-aligned.- // Only F16/F32 weights can be non-32-aligned, they will be padded in the following kernel.- const bool wtype_is_quant = (weight_type != HTP_TYPE_F16 && weight_type != HTP_TYPE_F32);- if (k % 32 != 0 || (wtype_is_quant && n % 32 != 0)) {+ const bool is_tiled = (tile_size > 0);++ if (k % 32 != 0) {return -1;}- if (!hex_is_aligned(dst, VLEN) || (act_dma_addr & (VLEN - 1)) != 0) { return -1; }+ if (!hex_is_aligned(dst, sizeof(float))) { return -1; }- size_t row_stride = htp_mm_get_tiled_row_stride(weight_type, k);+ size_t row_stride = htp_tensor_get_row_size(weight_type, k);if (row_stride == 0) {return -1;}- worker_callback_t dequant_worker_fn = NULL;- switch (weight_type) {- case HTP_TYPE_Q4_0: dequant_worker_fn = dequantize_tiled_worker_loop_q4_0; break;- case HTP_TYPE_IQ4_NL: dequant_worker_fn = dequantize_tiled_worker_loop_iq4_nl; break;- case HTP_TYPE_Q4_1:- case HTP_TYPE_Q4_K: dequant_worker_fn = dequantize_tiled_worker_loop_q4_1; break;- case HTP_TYPE_MXFP4: dequant_worker_fn = dequantize_tiled_worker_loop_mxfp4; break;- case HTP_TYPE_Q8_0: dequant_worker_fn = dequantize_tiled_worker_loop_q8_0; break;- case HTP_TYPE_Q5_K: dequant_worker_fn = dequantize_tiled_worker_loop_q5_k; break;- case HTP_TYPE_Q6_K: dequant_worker_fn = dequantize_tiled_worker_loop_q6_k; break;- case HTP_TYPE_Q3_K: dequant_worker_fn = dequantize_tiled_worker_loop_q3_k; break;- case HTP_TYPE_Q2_K: dequant_worker_fn = dequantize_tiled_worker_loop_q2_k; break;- case HTP_TYPE_F16: dequant_worker_fn = convert_f16_worker_loop; break;- case HTP_TYPE_F32: dequant_worker_fn = quantize_f32_worker_loop; break;- default:- return -1;- }+ tiled_dequant_task_fn_t dequant_worker_fn = hmx_get_dequant_worker_fn(weight_type, is_tiled);const int n_k_tiles = k / HTP_MM_HMX_TILE_N_COLS;const struct fastdiv_values n_k_tiles_div = init_fastdiv_values(n_k_tiles);- const bool is_quant = (weight_type != HTP_TYPE_F16 && weight_type != HTP_TYPE_F32);const size_t vec_dot_size = k * sizeof(__fp16);const size_t vtcm_budget = ctx->vtcm_size;- const uint32_t dma_dst_stride = is_quant ? aligned_tile_size : row_stride;- const uint32_t dma_src_stride = is_quant ? tile_size : weight_stride;- const uint32_t dma_width_bytes = is_quant ? tile_size : row_stride;+ const uint32_t dma_dst_stride = is_tiled ? aligned_tile_size : row_stride;+ const uint32_t dma_src_stride = is_tiled ? tile_size : weight_stride;+ const uint32_t dma_width_bytes = is_tiled ? tile_size : row_stride;size_t m_chunk_n_rows = m_chunk;size_t n_chunk_n_cols = n_chunk;size_t vtcm_used = vtcm_size;- const size_t qweight_row_stride = is_quant ? (size_t)(n_k_tiles * aligned_tile_size) / 32 : 0;+ const size_t qweight_row_stride = is_tiled ? (size_t)(n_k_tiles * aligned_tile_size) / 32 : 0;struct htp_mm_hmx_vtcm_layout L;htp_mm_hmx_vtcm_layout_build(&L, HTP_MM_KERNEL_HMX_2D, weight_type, k, m_chunk_n_rows, n_chunk_n_cols, 1, pipeline, act_threads, aligned_tile_size, src2_bytes);@@ -2827,7 +2696,6 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,VTCM_LAYOUT_PTR(__fp16, base, L.off_weight[0]),VTCM_LAYOUT_PTR_OPTIONAL(__fp16, base, L.off_weight[1], pipeline)};-__fp16 *vtcm_f16_act = VTCM_LAYOUT_PTR(__fp16, base, L.off_act);float *vtcm_f32_act = VTCM_LAYOUT_PTR(float, base, L.off_act_f32);__fp16 *vtcm_output = VTCM_LAYOUT_PTR(__fp16, base, L.off_dst[0]);@@ -2855,13 +2723,23 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,if (pipeline) {// --- Asynchronous Pipelined Loop ---hmx_matmul_job_t job_slots[2]; // persistent double-buffered job descriptors+ void *vtcm_raw_bufs[4] = {+ vtcm_weight_raw[0],+ vtcm_weight_raw[1],+ dequant_worker_fn ? vtcm_weight_raw[0] : vtcm_scratch0,+ dequant_worker_fn ? vtcm_weight_raw[1] : vtcm_scratch1,+ };+ void *vtcm_weight_bufs[4] = {+ dequant_worker_fn ? vtcm_scratch0 : vtcm_weight_raw[0],+ dequant_worker_fn ? vtcm_scratch1 : vtcm_weight_raw[1],+ vtcm_scratch0,+ vtcm_scratch1,+ };+ void *vtcm_output_bufs[2] = { vtcm_output, vtcm_scratch2 };for (size_t mr = 0; mr < m; mr += m_chunk_n_rows) {const size_t n_rows = hex_smin(m - mr, m_chunk_n_rows);- void *vtcm_weight_bufs[2] = { vtcm_scratch0, vtcm_scratch1 };- void *vtcm_output_bufs[2] = { vtcm_output, vtcm_scratch2 };-struct activation_transfer_params act_params = {.ctx = ctx,.dst = vtcm_f16_act,@@ -2881,14 +2759,14 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,// Prologue: push A0 and optionally A1 (if n_chunk_cnt > 1)const size_t n_cols_A0 = hex_smin(n - 0 * n_chunk_n_cols, n_chunk_n_cols);- const uint32_t height_A0 = is_quant ? hmx_ceil_div(n_cols_A0, 32) * n_k_tiles : n_cols_A0;- dma_queue_push(weight_dma, dma_make_data(vtcm_weight_raw[0], weight),+ const uint32_t height_A0 = is_tiled ? hmx_ceil_div(n_cols_A0, 32) * n_k_tiles : n_cols_A0;+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[0], weight),dma_dst_stride, dma_src_stride, dma_width_bytes, height_A0);if (1 < n_chunk_cnt) {const size_t n_cols_A1 = hex_smin(n - 1 * n_chunk_n_cols, n_chunk_n_cols);- const uint32_t height_A1 = is_quant ? hmx_ceil_div(n_cols_A1, 32) * n_k_tiles : n_cols_A1;- dma_queue_push(weight_dma, dma_make_data(vtcm_weight_raw[1], weight + n_chunk_n_cols * weight_stride),+ const uint32_t height_A1 = is_tiled ? hmx_ceil_div(n_cols_A1, 32) * n_k_tiles : n_cols_A1;+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[1], weight + n_chunk_n_cols * weight_stride),dma_dst_stride, dma_src_stride, dma_width_bytes, height_A1);}@@ -2905,20 +2783,20 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,// 2. dequantize A_idequantize_tiled_weight_chunk_to_fp16_tiles(- ctx, vtcm_weight_bufs[i % 2], curr_raw,+ ctx, vtcm_weight_bufs[i % 4], curr_raw,n_cols, k, row_stride, weight_type,n_k_tiles, n_k_tiles_div, dequant_worker_fn, n_threads);// 3. push A_{i+2} (if i+2 < n_chunk_cnt)if (i + 2 < n_chunk_cnt) {- const uint32_t height_p2 = is_quant ? hmx_ceil_div(n_cols_p2, 32) * n_k_tiles : n_cols_p2;- dma_queue_push(weight_dma, dma_make_data(curr_raw, weight + nc_p2 * weight_stride),+ const uint32_t height_p2 = is_tiled ? hmx_ceil_div(n_cols_p2, 32) * n_k_tiles : n_cols_p2;+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[(i + 2) % 4], weight + nc_p2 * weight_stride),dma_dst_stride, dma_src_stride, dma_width_bytes, height_p2);}// 4. submit C_ihmx_matmul_job_init(&job_slots[i % 2], (__fp16 *) vtcm_output_bufs[i % 2],- (__fp16 *) vtcm_f16_act, (__fp16 *) vtcm_weight_bufs[i % 2],+ (__fp16 *) vtcm_f16_act, (__fp16 *) vtcm_weight_bufs[i % 4],vtcm_scales, hmx_ceil_div(n_rows, HTP_MM_HMX_TILE_N_ROWS),hmx_ceil_div(n_cols, HTP_MM_HMX_TILE_N_COLS), k / HTP_MM_HMX_TILE_N_ROWS);hmx_queue_push(ctx->hmx_queue, hmx_queue_make_desc(hmx_matmul_worker_fn, &job_slots[i % 2]));@@ -2951,6 +2829,9 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,} else {// --- Synchronous loop (m <= 32 or fallback) ---hmx_matmul_job_t job;+ void *vtcm_raw_bufs[2] = { vtcm_weight_raw[0], dequant_worker_fn ? vtcm_weight_raw[0] : vtcm_scratch0 };+ void *vtcm_weight_bufs[2] = { dequant_worker_fn ? vtcm_scratch0 : vtcm_weight_raw[0], vtcm_scratch0 };+for (size_t mr = 0; mr < m; mr += m_chunk_n_rows) {const size_t n_rows = hex_smin(m - mr, m_chunk_n_rows);@@ -2974,11 +2855,12 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,// A0: Pre-fetch the first weight chunk (nc = 0)if (n > 0) {const size_t n_cols = hex_smin(n, n_chunk_n_cols);- const uint32_t height = is_quant ? hmx_ceil_div(n_cols, 32) * n_k_tiles : n_cols;- dma_queue_push(weight_dma, dma_make_data(vtcm_weight_raw[0], weight), dma_dst_stride, dma_src_stride, dma_width_bytes, height);+ const uint32_t height = is_tiled ? hmx_ceil_div(n_cols, 32) * n_k_tiles : n_cols;+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[0], weight), dma_dst_stride, dma_src_stride, dma_width_bytes, height);}- for (size_t nc = 0; nc < n; nc += n_chunk_n_cols) {+ for (int i = 0; i < n_chunk_cnt; ++i) {+ const size_t nc = (size_t) i * n_chunk_n_cols;const size_t n_cols = hex_smin(n - nc, n_chunk_n_cols);const size_t n_row_tiles = hmx_ceil_div(n_rows, HTP_MM_HMX_TILE_N_ROWS);const size_t n_col_tiles = hmx_ceil_div(n_cols, HTP_MM_HMX_TILE_N_COLS);@@ -2988,7 +2870,7 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,// B: Weight Dequantize (Threaded)dequantize_tiled_weight_chunk_to_fp16_tiles(- ctx, vtcm_scratch0, curr_raw,+ ctx, vtcm_weight_bufs[i % 2], curr_raw,n_cols, k, row_stride, weight_type,n_k_tiles, n_k_tiles_div, dequant_worker_fn, n_threads);@@ -2996,12 +2878,12 @@ static int hmx_mm_2d_f32(struct htp_context *ctx,const size_t nc_next = nc + n_chunk_n_cols;if (nc_next < n) {const size_t n_cols_next = hex_smin(n - nc_next, n_chunk_n_cols);- const uint32_t height_next = is_quant ? hmx_ceil_div(n_cols_next, 32) * n_k_tiles : n_cols_next;- dma_queue_push(weight_dma, dma_make_data(curr_raw, weight + nc_next * weight_stride), dma_dst_stride, dma_src_stride, dma_width_bytes, height_next);+ const uint32_t height_next = is_tiled ? hmx_ceil_div(n_cols_next, 32) * n_k_tiles : n_cols_next;+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[(i + 1) % 2], weight + nc_next * weight_stride), dma_dst_stride, dma_src_stride, dma_width_bytes, height_next);}// C: HMX Compute (Queue-based)- hmx_matmul_job_init(&job, vtcm_output, vtcm_f16_act, vtcm_scratch0, vtcm_scales, n_row_tiles, n_col_tiles, k / HTP_MM_HMX_TILE_N_ROWS);+ hmx_matmul_job_init(&job, vtcm_output, vtcm_f16_act, (__fp16 *) vtcm_weight_bufs[i % 2], vtcm_scales, n_row_tiles, n_col_tiles, k / HTP_MM_HMX_TILE_N_ROWS);hmx_queue_push(ctx->hmx_queue, hmx_queue_make_desc(hmx_matmul_worker_fn, &job));hmx_queue_pop(ctx->hmx_queue);@@ -3041,35 +2923,20 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_kconst dma_addr_t act_dma_addr = act->data;if (k % 32 != 0) { return HTP_STATUS_NO_SUPPORT; }- if ((act_dma_addr & (VLEN - 1)) != 0) { return HTP_STATUS_NO_SUPPORT; }- size_t row_stride = htp_mm_get_tiled_row_stride(weight_type, k);+ size_t row_stride = htp_tensor_get_row_size(weight_type, k);if (row_stride == 0) {return HTP_STATUS_NO_SUPPORT;}- worker_callback_t dequant_worker_fn = NULL;- switch (weight_type) {- case HTP_TYPE_Q4_0: dequant_worker_fn = dequantize_tiled_worker_loop_q4_0; break;- case HTP_TYPE_IQ4_NL: dequant_worker_fn = dequantize_tiled_worker_loop_iq4_nl; break;- case HTP_TYPE_Q4_1:- case HTP_TYPE_Q4_K: dequant_worker_fn = dequantize_tiled_worker_loop_q4_1; break;- case HTP_TYPE_MXFP4: dequant_worker_fn = dequantize_tiled_worker_loop_mxfp4; break;- case HTP_TYPE_Q8_0: dequant_worker_fn = dequantize_tiled_worker_loop_q8_0; break;- case HTP_TYPE_Q5_K: dequant_worker_fn = dequantize_tiled_worker_loop_q5_k; break;- case HTP_TYPE_Q6_K: dequant_worker_fn = dequantize_tiled_worker_loop_q6_k; break;- case HTP_TYPE_Q3_K: dequant_worker_fn = dequantize_tiled_worker_loop_q3_k; break;- case HTP_TYPE_Q2_K: dequant_worker_fn = dequantize_tiled_worker_loop_q2_k; break;- case HTP_TYPE_F16: dequant_worker_fn = convert_f16_worker_loop; break;- case HTP_TYPE_F32: dequant_worker_fn = quantize_f32_worker_loop; break;- default:- return HTP_STATUS_NO_SUPPORT;- }-const int n_k_tiles = k / HTP_MM_HMX_TILE_N_COLS;const struct fastdiv_values n_k_tiles_div = init_fastdiv_values(n_k_tiles);- const bool is_quant = (weight_type != HTP_TYPE_F16 && weight_type != HTP_TYPE_F32);+ const int tile_size = kparams->tile_size;+ const int aligned_tile_size = kparams->aligned_tile_size;++ const bool is_tiled = (tile_size > 0);+ tiled_dequant_task_fn_t dequant_worker_fn = hmx_get_dequant_worker_fn(weight_type, is_tiled);const size_t vtcm_budget = ctx->vtcm_size;const int m_chunk_n_rows = kparams->m_chunk;@@ -3079,11 +2946,9 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_kconst int act_threads = kparams->n_act_threads;const struct fastdiv_values * act_threads_div = &kparams->div_n_act_threads;const struct fastdiv_values * k_div = &kparams->div_ne00_padded;- const int tile_size = kparams->tile_size;- const int aligned_tile_size = kparams->aligned_tile_size;- const uint32_t dma_dst_stride = is_quant ? aligned_tile_size : row_stride;- const uint32_t dma_width_bytes = is_quant ? tile_size : row_stride;+ const uint32_t dma_dst_stride = is_tiled ? aligned_tile_size : row_stride;+ const uint32_t dma_width_bytes = is_tiled ? tile_size : row_stride;struct htp_mm_hmx_vtcm_layout L;htp_mm_hmx_vtcm_layout_build(&L, HTP_MM_KERNEL_HMX_2D, weight_type, k, m_chunk_n_rows, n_chunk_n_cols, 1, pipeline, act_threads, aligned_tile_size, 0);@@ -3099,7 +2964,6 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_kVTCM_LAYOUT_PTR(__fp16, base, L.off_weight[0]),VTCM_LAYOUT_PTR_OPTIONAL(__fp16, base, L.off_weight[1], pipeline)};-__fp16 *vtcm_f16_act = VTCM_LAYOUT_PTR(__fp16, base, L.off_act);float *vtcm_f32_act = VTCM_LAYOUT_PTR(float, base, L.off_act_f32);__fp16 *vtcm_output = VTCM_LAYOUT_PTR(__fp16, base, L.off_dst[0]);@@ -3141,13 +3005,23 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_kif (pipeline) {hmx_matmul_job_t job_slots[2];+ void *vtcm_raw_bufs[4] = {+ vtcm_weight_raw[0],+ vtcm_weight_raw[1],+ dequant_worker_fn ? vtcm_weight_raw[0] : vtcm_scratch0,+ dequant_worker_fn ? vtcm_weight_raw[1] : vtcm_scratch1,+ };+ void *vtcm_weight_bufs[4] = {+ dequant_worker_fn ? vtcm_scratch0 : vtcm_weight_raw[0],+ dequant_worker_fn ? vtcm_scratch1 : vtcm_weight_raw[1],+ vtcm_scratch0,+ vtcm_scratch1,+ };+ void *vtcm_output_bufs[2] = { vtcm_output, vtcm_scratch2 };for (size_t mr = (size_t) m_start; mr < mr_end; mr += m_chunk_n_rows) {const size_t n_rows = hex_smin(mr_end - mr, m_chunk_n_rows);- void *vtcm_weight_bufs[2] = { vtcm_scratch0, vtcm_scratch1 };- void *vtcm_output_bufs[2] = { vtcm_output, vtcm_scratch2 };-struct activation_transfer_params act_params = {.ctx = ctx,.dst = vtcm_f16_act,@@ -3180,17 +3054,17 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_kconst int dst_cols = (int) dst->ne[0];const int n_chunk_cnt = hmx_ceil_div(n, n_chunk_n_cols);- const uint32_t dma_src_stride = is_quant ? tile_size : weight_stride;+ const uint32_t dma_src_stride = is_tiled ? tile_size : weight_stride;const size_t n_cols_A0 = hex_smin(n - 0 * n_chunk_n_cols, n_chunk_n_cols);- const uint32_t height_A0 = is_quant ? hmx_ceil_div(n_cols_A0, 32) * n_k_tiles : n_cols_A0;- dma_queue_push(weight_dma, dma_make_data(vtcm_weight_raw[0], weight),+ const uint32_t height_A0 = is_tiled ? hmx_ceil_div(n_cols_A0, 32) * n_k_tiles : n_cols_A0;+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[0], weight),dma_dst_stride, dma_src_stride, dma_width_bytes, height_A0);if (1 < n_chunk_cnt) {const size_t n_cols_A1 = hex_smin(n - 1 * n_chunk_n_cols, n_chunk_n_cols);- const uint32_t height_A1 = is_quant ? hmx_ceil_div(n_cols_A1, 32) * n_k_tiles : n_cols_A1;- dma_queue_push(weight_dma, dma_make_data(vtcm_weight_raw[1], weight + n_chunk_n_cols * weight_stride),+ const uint32_t height_A1 = is_tiled ? hmx_ceil_div(n_cols_A1, 32) * n_k_tiles : n_cols_A1;+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[1], weight + n_chunk_n_cols * weight_stride),dma_dst_stride, dma_src_stride, dma_width_bytes, height_A1);}@@ -3204,18 +3078,18 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_kvoid * curr_raw = (void *) dma_queue_pop(weight_dma).dst;dequantize_tiled_weight_chunk_to_fp16_tiles(- ctx, vtcm_weight_bufs[i % 2], curr_raw,+ ctx, vtcm_weight_bufs[i % 4], curr_raw,n_cols, k, row_stride, weight_type,n_k_tiles, n_k_tiles_div, dequant_worker_fn, n_threads);if (i + 2 < n_chunk_cnt) {- const uint32_t height_p2 = is_quant ? hmx_ceil_div(n_cols_p2, 32) * n_k_tiles : n_cols_p2;- dma_queue_push(weight_dma, dma_make_data(curr_raw, weight + nc_p2 * weight_stride),+ const uint32_t height_p2 = is_tiled ? hmx_ceil_div(n_cols_p2, 32) * n_k_tiles : n_cols_p2;+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[(i + 2) % 4], weight + nc_p2 * weight_stride),dma_dst_stride, dma_src_stride, dma_width_bytes, height_p2);}hmx_matmul_job_init(&job_slots[i % 2], (__fp16 *) vtcm_output_bufs[i % 2],- (__fp16 *) vtcm_f16_act, (__fp16 *) vtcm_weight_bufs[i % 2],+ (__fp16 *) vtcm_f16_act, (__fp16 *) vtcm_weight_bufs[i % 4],vtcm_scales, hmx_ceil_div(n_rows, HTP_MM_HMX_TILE_N_ROWS),hmx_ceil_div(n_cols, HTP_MM_HMX_TILE_N_COLS), k / HTP_MM_HMX_TILE_N_ROWS);hmx_queue_push(ctx->hmx_queue, hmx_queue_make_desc(hmx_matmul_worker_fn, &job_slots[i % 2]));@@ -3244,6 +3118,9 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_k}} else {hmx_matmul_job_t job;+ void *vtcm_raw_bufs[2] = { vtcm_weight_raw[0], dequant_worker_fn ? vtcm_weight_raw[0] : vtcm_scratch0 };+ void *vtcm_weight_bufs[2] = { dequant_worker_fn ? vtcm_scratch0 : vtcm_weight_raw[0], vtcm_scratch0 };+for (size_t mr = (size_t) m_start; mr < mr_end; mr += m_chunk_n_rows) {const size_t n_rows = hex_smin(mr_end - mr, m_chunk_n_rows);@@ -3277,16 +3154,18 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_kconst size_t weight_stride = src_w->nb[1];const size_t dst_stride = dst->nb[1] / sizeof(float);const int dst_cols = (int) dst->ne[0];+ const int n_chunk_cnt = hmx_ceil_div(n, n_chunk_n_cols);- const uint32_t dma_src_stride = is_quant ? tile_size : weight_stride;+ const uint32_t dma_src_stride = is_tiled ? tile_size : weight_stride;if (n > 0) {const size_t n_cols = hex_smin(n, n_chunk_n_cols);- const uint32_t height = is_quant ? hmx_ceil_div(n_cols, 32) * n_k_tiles : n_cols;- dma_queue_push(weight_dma, dma_make_data(vtcm_weight_raw[0], weight), dma_dst_stride, dma_src_stride, dma_width_bytes, height);+ const uint32_t height = is_tiled ? hmx_ceil_div(n_cols, 32) * n_k_tiles : n_cols;+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[0], weight), dma_dst_stride, dma_src_stride, dma_width_bytes, height);}- for (size_t nc = 0; nc < n; nc += n_chunk_n_cols) {+ for (int i = 0; i < n_chunk_cnt; ++i) {+ const size_t nc = (size_t) i * n_chunk_n_cols;const size_t n_cols = hex_smin(n - nc, n_chunk_n_cols);const size_t n_row_tiles = hmx_ceil_div(n_rows, HTP_MM_HMX_TILE_N_ROWS);const size_t n_col_tiles = hmx_ceil_div(n_cols, HTP_MM_HMX_TILE_N_COLS);@@ -3294,18 +3173,18 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_kvoid * curr_raw = (void *) dma_queue_pop(weight_dma).dst;dequantize_tiled_weight_chunk_to_fp16_tiles(- ctx, vtcm_scratch0, curr_raw,+ ctx, vtcm_weight_bufs[i % 2], curr_raw,n_cols, k, row_stride, weight_type,n_k_tiles, n_k_tiles_div, dequant_worker_fn, n_threads);const size_t nc_next = nc + n_chunk_n_cols;if (nc_next < n) {const size_t n_cols_next = hex_smin(n - nc_next, n_chunk_n_cols);- const uint32_t height_next = is_quant ? hmx_ceil_div(n_cols_next, 32) * n_k_tiles : n_cols_next;- dma_queue_push(weight_dma, dma_make_data(curr_raw, weight + nc_next * weight_stride), dma_dst_stride, dma_src_stride, dma_width_bytes, height_next);+ const uint32_t height_next = is_tiled ? hmx_ceil_div(n_cols_next, 32) * n_k_tiles : n_cols_next;+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[(i + 1) % 2], weight + nc_next * weight_stride), dma_dst_stride, dma_src_stride, dma_width_bytes, height_next);}- hmx_matmul_job_init(&job, vtcm_output, vtcm_f16_act, vtcm_scratch0, vtcm_scales, n_row_tiles, n_col_tiles, k / HTP_MM_HMX_TILE_N_ROWS);+ hmx_matmul_job_init(&job, vtcm_output, vtcm_f16_act, (__fp16 *) vtcm_weight_bufs[i % 2], vtcm_scales, n_row_tiles, n_col_tiles, k / HTP_MM_HMX_TILE_N_ROWS);hmx_queue_push(ctx->hmx_queue, hmx_queue_make_desc(hmx_matmul_worker_fn, &job));hmx_queue_pop(ctx->hmx_queue);@@ -3322,30 +3201,30 @@ static int hmx_mm_nx_2d_f32(struct htp_ops_context * octx, const struct htp_mm_kreturn HTP_STATUS_OK;}-static inline dma_addr_t hmx_mm_weight_batch_data(const hmx_mm_f16_f32_batched_params_t *params,+static inline dma_addr_t hmx_mm_weight_batch_data(const hmx_mm_batched_params_t *params,int dst_b2, int dst_b3) {const size_t b2_idx = (params->r2 <= 1) ? (size_t) dst_b2 : (size_t) fastdiv((uint32_t) dst_b2, ¶ms->div_r2);const size_t b3_idx = (params->r3 <= 1) ? (size_t) dst_b3 : (size_t) fastdiv((uint32_t) dst_b3, ¶ms->div_r3);return params->weight + b2_idx * params->src0_nb2 + b3_idx * params->src0_nb3;}-static inline dma_addr_t hmx_mm_act_batch_addr(const hmx_mm_f16_f32_batched_params_t *params,+static inline dma_addr_t hmx_mm_act_batch_addr(const hmx_mm_batched_params_t *params,int dst_b2, int dst_b3) {return params->act_dma_addr + dst_b2 * params->act_nb2 +dst_b3 * params->act_nb3;}-static inline float *hmx_mm_dst_batch_ptr(const hmx_mm_f16_f32_batched_params_t *params,+static inline float *hmx_mm_dst_batch_ptr(const hmx_mm_batched_params_t *params,int dst_b2, int dst_b3) {return (float *) ((uint8_t *) params->dst +(size_t) dst_b2 * params->dst_nb2 +(size_t) dst_b3 * params->dst_nb3);}-static int hmx_mm_f16_f32_batched_simple(struct htp_context *ctx,- const hmx_mm_f16_f32_batched_params_t *params,- int m_chunk, int n_chunk, int pipeline, int n_threads, int act_threads, int vtcm_size,- const struct fastdiv_values * act_threads_div, const struct fastdiv_values * k_div) {+static int hmx_mm_batched_simple(struct htp_context *ctx,+ const hmx_mm_batched_params_t *params,+ int m_chunk, int n_chunk, int pipeline, int n_threads, int act_threads, int vtcm_size,+ const struct fastdiv_values * act_threads_div, const struct fastdiv_values * k_div) {int ret = 0;for (int b3 = 0; b3 < params->ne13 && ret == 0; ++b3) {for (int b2 = 0; b2 < params->ne12 && ret == 0; ++b2) {@@ -3357,54 +3236,53 @@ static int hmx_mm_f16_f32_batched_simple(struct htp_context *ctx,hmx_mm_act_batch_addr(params, b2, b3),hmx_mm_weight_batch_data(params, b2, b3),params->m, params->k, params->n,- params->act_stride, params->act_elem_size, params->weight_stride * (int)sizeof(__fp16),- HTP_TYPE_F16, params->k, params->dst_stride, params->bias_stride, params->n,+ params->act_stride, params->act_elem_size, params->weight_stride,+ params->weight_type, params->k, params->dst_stride, params->bias_stride, params->dst_cols,m_chunk, n_chunk, pipeline, n_threads, act_threads,- act_threads_div, k_div, 0, 0, vtcm_size);+ act_threads_div, k_div, params->tile_size, params->aligned_tile_size, vtcm_size);}}return ret;}-static int hmx_mm_f16_f32_batched(struct htp_context *ctx, const hmx_mm_f16_f32_batched_params_t *params,- int m_chunk, int n_chunk, int pipeline, int n_threads, int act_threads,- const struct fastdiv_values * act_threads_div,- const struct fastdiv_values * k_div,- int vtcm_size) {- if (params->act_stride < params->k || params->weight_stride < params->k || params->dst_stride < params->n) { return -1; }+static int hmx_mm_batched(struct htp_context *ctx, const hmx_mm_batched_params_t *params,+ int m_chunk, int n_chunk, int pipeline, int n_threads, int act_threads,+ const struct fastdiv_values * act_threads_div,+ const struct fastdiv_values * k_div,+ int vtcm_size) {+ const size_t row_stride = htp_tensor_get_row_size(params->weight_type, params->k);+ if (row_stride == 0) { return -1; }+ if (params->act_stride < params->k || (params->tile_size == 0 && (size_t) params->weight_stride < row_stride) || params->dst_stride < params->dst_cols) { return -1; }if (params->ne02 <= 0 || params->ne03 <= 0 || params->ne12 <= 0 || params->ne13 <= 0) { return -1; }if (params->ne12 % params->ne02 != 0 || params->ne13 % params->ne03 != 0) { return -1; }// N (the weight row count) does not have to be 32-aligned:if (params->k % 32 != 0) { return -1; }- if (!hex_is_aligned(params->dst, VLEN) || (params->act_dma_addr & (VLEN - 1)) != 0) { return -1; }+ if (!hex_is_aligned(params->dst, sizeof(float))) { return -1; }const int group_size = params->r2;const size_t vtcm_budget = ctx->vtcm_size;// Check if the precomputed parameters are grouped or simple.// If simple, or if group_size <= 1, we use simple fallback loop.- // Grouped path is only valid if group_size > 1 and it fits within VTCM budget.- bool run_grouped = (group_size > 1 && (size_t)vtcm_size <= vtcm_budget);+ // Grouped path is only valid for flat F16 if group_size > 1 and it fits within VTCM budget.+ bool run_grouped = (params->weight_type == HTP_TYPE_F16 && params->tile_size == 0 && group_size > 1 && (size_t)vtcm_size <= vtcm_budget);if (!run_grouped) {- return hmx_mm_f16_f32_batched_simple(ctx, params, m_chunk, n_chunk, pipeline, n_threads, act_threads, vtcm_size, act_threads_div, k_div);+ return hmx_mm_batched_simple(ctx, params, m_chunk, n_chunk, pipeline, n_threads, act_threads, vtcm_size, act_threads_div, k_div);}struct htp_thread_trace * tr = &ctx->trace[0];htp_trace_event_start(tr, HTP_TRACE_EVT_INIT, 0);- const size_t vec_dot_size = params->k * sizeof(__fp16);-size_t m_chunk_n_rows = m_chunk;size_t n_chunk_n_cols = n_chunk;- size_t vtcm_used = vtcm_size;struct htp_mm_hmx_vtcm_layout L;- htp_mm_hmx_vtcm_layout_build(&L, HTP_MM_KERNEL_HMX_F16_BATCHED, HTP_TYPE_F16, params->k, m_chunk_n_rows, n_chunk_n_cols, group_size, false, act_threads, 0, params->bias_bytes);+ htp_mm_hmx_vtcm_layout_build(&L, HTP_MM_KERNEL_HMX_BATCHED, HTP_TYPE_F16, params->k, m_chunk_n_rows, n_chunk_n_cols, group_size, false, act_threads, 0, params->bias_bytes);if (L.total_bytes > vtcm_budget) {FARF(HIGH, "%s: grouped layout overflowed VTCM, falling back to simple batched loop", __func__);htp_trace_event_stop(tr, HTP_TRACE_EVT_INIT, 0);- return hmx_mm_f16_f32_batched_simple(ctx, params, m_chunk, n_chunk, pipeline, n_threads, act_threads, vtcm_size, act_threads_div, k_div);+ return hmx_mm_batched_simple(ctx, params, m_chunk, n_chunk, pipeline, n_threads, act_threads, vtcm_size, act_threads_div, k_div);}uint8_t * const base = (uint8_t *) ctx->vtcm_base;@@ -3431,7 +3309,7 @@ static int hmx_mm_f16_f32_batched(struct htp_context *ctx, const hmx_mm_f16_f32_L.total_bytes, vtcm_budget);const size_t fp16_row_bytes = (size_t) params->k * sizeof(__fp16);- const size_t weight_row_bytes = (size_t) params->weight_stride * sizeof(__fp16);+ const size_t weight_row_bytes = (size_t) params->weight_stride;htp_trace_event_stop(tr, HTP_TRACE_EVT_INIT, 0);@@ -3481,7 +3359,7 @@ static int hmx_mm_f16_f32_batched(struct htp_context *ctx, const hmx_mm_f16_f32_}if (n_chunk_n_cols < (size_t) params->n) {const size_t n_cols_second = hex_smin((size_t) params->n - n_chunk_n_cols, n_chunk_n_cols);- const dma_addr_t second_weight_chunk = weight_group + n_chunk_n_cols * params->weight_stride * sizeof(__fp16);+ const dma_addr_t second_weight_chunk = weight_group + n_chunk_n_cols * weight_row_bytes;dma_queue_push(weight_dma, dma_make_data(vtcm_scratch1, second_weight_chunk),fp16_row_bytes, weight_row_bytes, fp16_row_bytes, n_cols_second);}@@ -3500,7 +3378,7 @@ static int hmx_mm_f16_f32_batched(struct htp_context *ctx, const hmx_mm_f16_f32_const size_t nc_next = nc + n_chunk_n_cols * 2;if (nc_next < (size_t) params->n) {const size_t n_cols_next = hex_smin((size_t) params->n - nc_next, n_chunk_n_cols);- const dma_addr_t next_weight_chunk = weight_group + nc_next * params->weight_stride * sizeof(__fp16);+ const dma_addr_t next_weight_chunk = weight_group + nc_next * weight_row_bytes;dma_queue_push(weight_dma, dma_make_data(curr_raw, next_weight_chunk),fp16_row_bytes, weight_row_bytes, fp16_row_bytes, n_cols_next);@@ -3519,7 +3397,7 @@ static int hmx_mm_f16_f32_batched(struct htp_context *ctx, const hmx_mm_f16_f32_{float *output = hmx_mm_dst_batch_ptr(params, b2_base + g, b3) + mr * params->dst_stride + nc;const float *bias_chunk = has_bias ? (vtcm_bias + mr * params->bias_stride + nc) : NULL;- int chunk_dst_cols = params->n - (int)nc;+ int chunk_dst_cols = params->dst_cols - (int)nc;if (chunk_dst_cols > 0) {transfer_output_chunk_threaded(ctx, output, bias_chunk, vtcm_output, (int) n_rows, (int) n_cols,params->dst_stride, params->bias_stride, chunk_dst_cols, n_threads);@@ -3642,6 +3520,8 @@ static int hmx_mm_id_2d_f32(struct htp_context *ctx,size_t dst_nb1, size_t dst_nb2,int weight_stride,int weight_type,+ int tile_size,+ int aligned_tile_size,const struct mmid_row_mapping *matrix_rows,int cur_a,int mapping_stride,@@ -3658,47 +3538,27 @@ static int hmx_mm_id_2d_f32(struct htp_context *ctx,if (k % 32 != 0 || n % 32 != 0) { return -1; }if (!hex_is_aligned(dst, VLEN) || !hex_is_aligned(activation, VLEN)) { return -1; }- size_t row_stride = htp_mm_get_tiled_row_stride(weight_type, k);+ size_t row_stride = htp_tensor_get_row_size(weight_type, k);if (row_stride == 0) {return -1;}- worker_callback_t dequant_worker_fn = NULL;- switch (weight_type) {- case HTP_TYPE_Q4_0: dequant_worker_fn = dequantize_tiled_worker_loop_q4_0; break;- case HTP_TYPE_IQ4_NL: dequant_worker_fn = dequantize_tiled_worker_loop_iq4_nl; break;- case HTP_TYPE_Q4_1:- case HTP_TYPE_Q4_K: dequant_worker_fn = dequantize_tiled_worker_loop_q4_1; break;- case HTP_TYPE_MXFP4: dequant_worker_fn = dequantize_tiled_worker_loop_mxfp4; break;- case HTP_TYPE_Q8_0: dequant_worker_fn = dequantize_tiled_worker_loop_q8_0; break;- case HTP_TYPE_Q5_K: dequant_worker_fn = dequantize_tiled_worker_loop_q5_k; break;- case HTP_TYPE_Q6_K: dequant_worker_fn = dequantize_tiled_worker_loop_q6_k; break;- case HTP_TYPE_Q3_K: dequant_worker_fn = dequantize_tiled_worker_loop_q3_k; break;- case HTP_TYPE_Q2_K: dequant_worker_fn = dequantize_tiled_worker_loop_q2_k; break;- case HTP_TYPE_F16: dequant_worker_fn = convert_f16_worker_loop; break;- case HTP_TYPE_F32: dequant_worker_fn = quantize_f32_worker_loop; break;- default:- return -1;- }-const int n_k_tiles = k / HTP_MM_HMX_TILE_N_COLS;const struct fastdiv_values n_k_tiles_div = init_fastdiv_values(n_k_tiles);- const bool is_quant = (weight_type != HTP_TYPE_F16 && weight_type != HTP_TYPE_F32);-const size_t vec_dot_size = k * sizeof(__fp16);const size_t vtcm_budget = ctx->vtcm_size;size_t vtcm_used = 0;- int tile_size = htp_mm_get_weight_tile_size(weight_type);- int aligned_tile_size = htp_mm_get_weight_aligned_tile_size(weight_type);+ const bool is_tiled = (tile_size > 0);+ tiled_dequant_task_fn_t dequant_worker_fn = hmx_get_dequant_worker_fn(weight_type, is_tiled);- const uint32_t dma_dst_stride = is_quant ? aligned_tile_size : row_stride;- const uint32_t dma_src_stride = is_quant ? tile_size : weight_stride;- const uint32_t dma_width_bytes = is_quant ? tile_size : row_stride;+ const uint32_t dma_dst_stride = is_tiled ? aligned_tile_size : row_stride;+ const uint32_t dma_src_stride = is_tiled ? tile_size : weight_stride;+ const uint32_t dma_width_bytes = is_tiled ? tile_size : row_stride;- const size_t qweight_row_stride = is_quant ? (size_t)(n_k_tiles * aligned_tile_size) / 32 : 0;- const size_t weight_row_stride = is_quant ? qweight_row_stride : row_stride;+ const size_t qweight_row_stride = is_tiled ? (size_t)(n_k_tiles * aligned_tile_size) / 32 : 0;+ const size_t weight_row_stride = is_tiled ? qweight_row_stride : row_stride;size_t size_per_n = 0, size_per_m = 0, size_per_mn = 0;htp_mm_hmx_get_2d_chunk_costs(weight_type, k, /*pipeline=*/false, aligned_tile_size,@@ -3738,6 +3598,9 @@ static int hmx_mm_id_2d_f32(struct htp_context *ctx,htp_trace_event_stop(tr, HTP_TRACE_EVT_INIT, 0);hmx_matmul_job_t job;+ void *vtcm_raw_bufs[2] = { vtcm_weight, dequant_worker_fn ? vtcm_weight : vtcm_scratch0 };+ void *vtcm_weight_bufs[2] = { dequant_worker_fn ? vtcm_scratch0 : vtcm_weight, vtcm_scratch0 };+ const int n_chunk_cnt = hmx_ceil_div(n, n_chunk_n_cols);for (size_t mr = (size_t) m_start; mr < (size_t) m_end; mr += m_chunk_n_rows) {const size_t n_rows = hex_smin((size_t) m_end - mr, m_chunk_n_rows);@@ -3750,12 +3613,13 @@ static int hmx_mm_id_2d_f32(struct htp_context *ctx,// A0: Pre-fetch the first weight chunk (nc = 0)if (n > 0) {const size_t n_cols = hex_smin((size_t) n, n_chunk_n_cols);- const uint32_t height = is_quant ? hmx_ceil_div(n_cols, 32) * n_k_tiles : n_cols;- dma_queue_push(weight_dma, dma_make_data(vtcm_weight, weight),+ const uint32_t height = is_tiled ? hmx_ceil_div(n_cols, 32) * n_k_tiles : n_cols;+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[0], weight),dma_dst_stride, dma_src_stride, dma_width_bytes, height);}- for (size_t nc = 0; nc < (size_t) n; nc += n_chunk_n_cols) {+ for (int i = 0; i < n_chunk_cnt; ++i) {+ const size_t nc = (size_t) i * n_chunk_n_cols;const size_t n_cols = hex_smin((size_t) n - nc, n_chunk_n_cols);const size_t n_col_tiles = hmx_ceil_div(n_cols, HTP_MM_HMX_TILE_N_COLS);@@ -3764,7 +3628,7 @@ static int hmx_mm_id_2d_f32(struct htp_context *ctx,// B: Weight Dequantize (Threaded)dequantize_tiled_weight_chunk_to_fp16_tiles(- ctx, vtcm_scratch0, curr_raw,+ ctx, vtcm_weight_bufs[i % 2], curr_raw,n_cols, k, row_stride, weight_type,n_k_tiles, n_k_tiles_div, dequant_worker_fn, n_threads);@@ -3773,13 +3637,13 @@ static int hmx_mm_id_2d_f32(struct htp_context *ctx,const size_t nc_next = nc + n_chunk_n_cols;if (nc_next < (size_t) n) {const size_t n_cols_next = hex_smin((size_t) n - nc_next, n_chunk_n_cols);- const uint32_t height_next = is_quant ? hmx_ceil_div(n_cols_next, 32) * n_k_tiles : n_cols_next;- dma_queue_push(weight_dma, dma_make_data(curr_raw, weight + nc_next * weight_stride),+ const uint32_t height_next = is_tiled ? hmx_ceil_div(n_cols_next, 32) * n_k_tiles : n_cols_next;+ dma_queue_push(weight_dma, dma_make_data(vtcm_raw_bufs[(i + 1) % 2], weight + nc_next * weight_stride),dma_dst_stride, dma_src_stride, dma_width_bytes, height_next);}// C: HMX Compute (Queue-based)- hmx_matmul_job_init(&job, vtcm_output, vtcm_f16_act, vtcm_scratch0, vtcm_scales, n_row_tiles, n_col_tiles, k / HTP_MM_HMX_TILE_N_ROWS);+ hmx_matmul_job_init(&job, vtcm_output, vtcm_f16_act, (__fp16 *) vtcm_weight_bufs[i % 2], vtcm_scales, n_row_tiles, n_col_tiles, k / HTP_MM_HMX_TILE_N_ROWS);hmx_queue_push(ctx->hmx_queue, hmx_queue_make_desc(hmx_matmul_worker_fn, &job));hmx_queue_pop(ctx->hmx_queue);@@ -3803,7 +3667,6 @@ static int hmx_mm_op_matmul(struct htp_ops_context * octx, const struct htp_mm_kconst int m_total = (int) act->ne[1];const uint32_t act_elem_size = (act->type == HTP_TYPE_F16) ? sizeof(__fp16) : sizeof(float);const int act_stride = (int) (act->nb[1] / act_elem_size);- const int wgt_stride = (int)(src0->nb[1] / sizeof(__fp16));int m_start = 0;int m_rows = m_total;@@ -3842,46 +3705,50 @@ static int hmx_mm_op_matmul(struct htp_ops_context * octx, const struct htp_mm_kint ret = -1;const int n_threads = kparams->n_threads;- if (kparams->kernel_type == HTP_MM_KERNEL_HMX_F16_BATCHED) {- hmx_mm_f16_f32_batched_params_t batch_params = {- .dst = dst_ptr,- .bias_addr = src2_addr,- .bias_bytes = src2_bytes,- .act_dma_addr = act_addr,- .weight = src0->data,- .weight_dma = octx->ctx->dma[0],- .m = m_rows,- .k = k,- .n = n,- .act_stride = act_stride,- .act_elem_size = act_elem_size,- .weight_stride = wgt_stride,- .dst_stride = dst_stride,- .bias_stride = src2_stride,- .ne02 = ne02,- .ne03 = ne03,- .ne12 = ne12,- .ne13 = ne13,- .src0_nb2 = src0->nb[2],- .src0_nb3 = src0->nb[3],- .act_nb2 = act->nb[2],- .act_nb3 = act->nb[3],- .dst_nb2 = dst->nb[2],- .dst_nb3 = dst->nb[3],- .bias_nb2 = src2_nb2,- .bias_nb3 = src2_nb3,- .r2 = (ne02 > 0) ? (ne12 / ne02) : 1,- .r3 = (ne03 > 0) ? (ne13 / ne03) : 1,- .div_r2 = kparams->div_r2,- .div_r3 = kparams->div_r3,+ if (kparams->kernel_type == HTP_MM_KERNEL_HMX_BATCHED) {+ hmx_mm_batched_params_t batch_params = {+ .dst = dst_ptr,+ .bias_addr = src2_addr,+ .bias_bytes = src2_bytes,+ .act_dma_addr = act_addr,+ .weight = src0->data,+ .weight_dma = octx->ctx->dma[0],+ .m = m_rows,+ .k = k,+ .n = n,+ .act_stride = act_stride,+ .act_elem_size = act_elem_size,+ .weight_stride = (int) src0->nb[1],+ .weight_type = (int) src0->type,+ .dst_stride = dst_stride,+ .dst_cols = (int) dst->ne[0],+ .bias_stride = src2_stride,+ .tile_size = kparams->tile_size,+ .aligned_tile_size = kparams->aligned_tile_size,+ .ne02 = ne02,+ .ne03 = ne03,+ .ne12 = ne12,+ .ne13 = ne13,+ .src0_nb2 = src0->nb[2],+ .src0_nb3 = src0->nb[3],+ .act_nb2 = act->nb[2],+ .act_nb3 = act->nb[3],+ .dst_nb2 = dst->nb[2],+ .dst_nb3 = dst->nb[3],+ .bias_nb2 = src2_nb2,+ .bias_nb3 = src2_nb3,+ .r2 = (ne02 > 0) ? (ne12 / ne02) : 1,+ .r3 = (ne03 > 0) ? (ne13 / ne03) : 1,+ .div_r2 = kparams->div_r2,+ .div_r3 = kparams->div_r3,};- ret = hmx_mm_f16_f32_batched(octx->ctx, &batch_params,- kparams->m_chunk, kparams->n_chunk,- kparams->pipeline, n_threads,- kparams->n_act_threads,- &kparams->div_n_act_threads,- &kparams->div_ne00_padded,- kparams->vtcm_size);+ ret = hmx_mm_batched(octx->ctx, &batch_params,+ kparams->m_chunk, kparams->n_chunk,+ kparams->pipeline, n_threads,+ kparams->n_act_threads,+ &kparams->div_n_act_threads,+ &kparams->div_ne00_padded,+ kparams->vtcm_size);} else {ret = hmx_mm_2d_f32(octx->ctx, octx->ctx->dma[0], dst_ptr, src2_addr, src2_bytes,@@ -4014,6 +3881,7 @@ static int hmx_mm_op_matmul_id(nb11, nb12,nb1, nb2,(int) src0->nb[1], (int) src0->type,+ kparams->tile_size, kparams->aligned_tile_size,matrix_rows, cur_a, mmctx->mapping_stride,m_start, m_end, (int) octx->n_threads);if (ret != 0) {@@ -4049,7 +3917,7 @@ static int hvx_mm_matmul_id(uint32_t n_quant_tasks = 1;if (act_nrows < octx->n_threads) {n_quant_tasks = MIN(total_nb, octx->n_threads);- quant_task_func = htp_mm_act_quant_block_func(src0->type);+ quant_task_func = htp_mm_act_quant_block_func(src0->type, act->type);for (uint32_t ith = 0; ith < n_quant_tasks; ++ith) {uint32_t ib_first = (total_nb * ith) / n_quant_tasks;uint32_t ib_last = (total_nb * (ith + 1)) / n_quant_tasks;@@ -4060,9 +3928,9 @@ static int hvx_mm_matmul_id(}} else {n_quant_tasks = MIN(act_nrows, octx->n_threads);- quant_task_func = htp_mm_act_quant_row_func(src0->type);+ quant_task_func = htp_mm_act_quant_row_func(src0->type, act->type);}- size_t act_row_size = htp_mm_weight_has_offset(src0->type) ? htp_mm_q8_1_tiled_row_size(ne10) : htp_mm_q8_0_tiled_row_size(ne10);+ size_t act_row_size = htp_mm_tiled_act_row_size(src0->type, ne10);struct htp_mm_hvx_vtcm_layout L;htp_mm_hvx_vtcm_layout_build(&L, kparams->kernel_type, src0->type, ne10, act_nrows, octx->n_threads,@@ -4177,6 +4045,7 @@ static int hmx_mm_op_matmul_id_nx(act->nb[1], act->nb[2],dst->nb[1], dst->nb[2],(int) src_w->nb[1], (int) src_w->type,+ kparams->tile_size, kparams->aligned_tile_size,matrix_rows, cur_a, mmctx->mapping_stride,m_start, m_end, (int) octx->n_threads);if (ret != 0) {@@ -4215,7 +4084,7 @@ static int hvx_mm_matmul_id_nx(uint32_t n_quant_tasks = 1;if (act_nrows < octx->n_threads) {n_quant_tasks = MIN(total_nb, octx->n_threads);- quant_task_func = htp_mm_act_quant_block_func(src0->type);+ quant_task_func = htp_mm_act_quant_block_func(src0->type, act->type);for (uint32_t ith = 0; ith < n_quant_tasks; ++ith) {uint32_t ib_first = (total_nb * ith) / n_quant_tasks;uint32_t ib_last = (total_nb * (ith + 1)) / n_quant_tasks;@@ -4226,9 +4095,9 @@ static int hvx_mm_matmul_id_nx(}} else {n_quant_tasks = MIN(act_nrows, octx->n_threads);- quant_task_func = htp_mm_act_quant_row_func(src0->type);+ quant_task_func = htp_mm_act_quant_row_func(src0->type, act->type);}- size_t act_row_size = htp_mm_weight_has_offset(src0->type) ? htp_mm_q8_1_tiled_row_size(act->ne[0]) : htp_mm_q8_0_tiled_row_size(act->ne[0]);+ size_t act_row_size = htp_mm_tiled_act_row_size(src0->type, act->ne[0]);struct htp_mm_hvx_vtcm_layout L;htp_mm_hvx_vtcm_layout_build(&L, kparams->kernel_type, src0->type, act->ne[0], act_nrows, octx->n_threads,@@ -4451,8 +4320,8 @@ int op_matmul_id(struct htp_ops_context * octx) {mmctx->src0_nrows_per_thread = fastdiv(nrows + octx->n_threads - 1, &octx->n_threads_div);mmctx->src0_nrows_per_thread = hex_round_up(mmctx->src0_nrows_per_thread, 32);- if (hvx_mm_init_vec_dot(mmctx, src0->type) == 0) {- s = hvx_mm_matmul_id(octx, mmctx, act_nrows > 1 ? hvx_mm_id : hvx_mv_id);+ if (hvx_mm_init_tiled_vec_dot(mmctx, src0->type) == 0) {+ s = hvx_mm_matmul_id(octx, mmctx, act_nrows > 1 ? hvx_mm_id_tiled : hvx_mv_id_tiled);} else {s = HTP_STATUS_NO_SUPPORT;}@@ -4551,8 +4420,8 @@ int op_matmul_id_nx(struct htp_ops_context * octx) {if (kparams->n_hmx) {s = hmx_mm_op_matmul_id_nx(octx, mmctx);} else {- if (hvx_mm_init_vec_dot(mmctx, src0->type) == 0) {- s = hvx_mm_matmul_id_nx(octx, mmctx, act_nrows > 1 ? hvx_mm_id_nx : hvx_mv_id_nx);+ if (hvx_mm_init_tiled_vec_dot(mmctx, src0->type) == 0) {+ s = hvx_mm_matmul_id_nx(octx, mmctx, act_nrows > 1 ? hvx_mm_id_nx_tiled : hvx_mv_id_nx_tiled);} else {s = HTP_STATUS_NO_SUPPORT;}@@ -4584,12 +4453,6 @@ static int op_matmul_nx_impl(struct htp_ops_context * octx) {const struct htp_tensor * restrict src0 = octx->src[0]; // first weightconst struct htp_tensor * restrict act = octx->src[n_weights]; // activation x- bool is_repacked = (src0->type == HTP_TYPE_Q4_0 || src0->type == HTP_TYPE_Q4_1 ||- src0->type == HTP_TYPE_Q8_0 || src0->type == HTP_TYPE_IQ4_NL ||- src0->type == HTP_TYPE_MXFP4 || src0->type == HTP_TYPE_Q4_K ||- src0->type == HTP_TYPE_Q5_K || src0->type == HTP_TYPE_Q3_K ||- src0->type == HTP_TYPE_Q2_K);-struct htp_mm_context mmctx_struct = {0};struct htp_mm_context * mmctx = &mmctx_struct;mmctx->octx = octx;@@ -4601,9 +4464,8 @@ static int op_matmul_nx_impl(struct htp_ops_context * octx) {mmctx->cur_m_rows = act_nrows;const size_t src0_row_size = src0->nb[1];- const size_t src0_row_size_padded = hex_round_up(src0_row_size, 128);- if (hvx_mm_init_vec_dot(mmctx, src0->type) != 0) {+ if (hvx_mm_init_tiled_vec_dot(mmctx, src0->type) != 0) {return HTP_STATUS_NO_SUPPORT;}@@ -4615,7 +4477,7 @@ static int op_matmul_nx_impl(struct htp_ops_context * octx) {uint32_t n_quant_tasks = 1;if (act_nrows < octx->n_threads) {n_quant_tasks = MIN(total_nb, octx->n_threads);- quant_task_func = htp_mm_act_quant_block_func(src0->type);+ quant_task_func = htp_mm_act_quant_block_func(src0->type, act->type);for (uint32_t ith = 0; ith < n_quant_tasks; ++ith) {uint32_t ib_first = (total_nb * ith) / n_quant_tasks;uint32_t ib_last = (total_nb * (ith + 1)) / n_quant_tasks;@@ -4626,12 +4488,10 @@ static int op_matmul_nx_impl(struct htp_ops_context * octx) {}} else {n_quant_tasks = MIN(act_nrows, octx->n_threads);- quant_task_func = htp_mm_act_quant_row_func(src0->type);+ quant_task_func = htp_mm_act_quant_row_func(src0->type, act->type);}- const size_t act_row_size = htp_mm_weight_has_offset(src0->type)- ? htp_mm_q8_1_tiled_row_size(act->ne[0])- : htp_mm_q8_0_tiled_row_size(act->ne[0]);+ const size_t act_row_size = htp_mm_tiled_act_row_size(src0->type, act->ne[0]);struct htp_mm_hvx_vtcm_layout L;htp_mm_hvx_vtcm_layout_build(&L, kparams->kernel_type, src0->type, act->ne[0], act_nrows, octx->n_threads,@@ -4651,7 +4511,7 @@ static int op_matmul_nx_impl(struct htp_ops_context * octx) {mmctx->vtcm_dst = VTCM_LAYOUT_PTR(uint8_t, base, L.off_dst);mmctx->vtcm_act_raw = VTCM_LAYOUT_PTR(uint8_t, base, L.off_act_raw);- mmctx->vtcm_src0_stride = is_repacked ? 0 : src0_row_size_padded;+ mmctx->vtcm_src0_stride = 0;mmctx->vtcm_act_stride = act_row_size;mmctx->vtcm_act_raw_stride = hex_round_up(act->ne[0] * sizeof(float), QK_Q8_0_TILED * sizeof(float));@@ -4659,28 +4519,6 @@ static int op_matmul_nx_impl(struct htp_ops_context * octx) {mmctx->vtcm_act_size_per_thread = L.act_bytes;mmctx->vtcm_dst_size_per_thread = fastdiv(L.dst_bytes, &octx->n_threads_div);- // Run fused matmul- const uint32_t n_matmul_jobs = octx->n_threads;- worker_callback_t matmul_job_func;- if (is_repacked) {- switch (src0->type) {- case HTP_TYPE_Q4_0: matmul_job_func = hvx_mm_nx_2d_repacked_q4_0; break;- case HTP_TYPE_Q4_1:- case HTP_TYPE_Q4_K: matmul_job_func = hvx_mm_nx_2d_repacked_q4_1; break;- case HTP_TYPE_Q5_K: matmul_job_func = hvx_mm_nx_2d_repacked_q5_k; break;- case HTP_TYPE_Q3_K: matmul_job_func = hvx_mm_nx_2d_repacked_q3_k; break;- case HTP_TYPE_Q2_K: matmul_job_func = hvx_mm_nx_2d_repacked_q2_k; break;- case HTP_TYPE_Q8_0: matmul_job_func = hvx_mm_nx_2d_repacked_q8_0; break;- case HTP_TYPE_IQ4_NL: matmul_job_func = hvx_mm_nx_2d_repacked_iq4nl; break;- case HTP_TYPE_MXFP4: matmul_job_func = hvx_mm_nx_2d_repacked_mxfp4; break;- default:- htp_trace_event_stop(tr, HTP_TRACE_EVT_INIT, 0);- return HTP_STATUS_NO_SUPPORT;- }- } else {- matmul_job_func = hvx_mm_nx_2d;- }-htp_trace_event_stop(tr, HTP_TRACE_EVT_INIT, 0);hvx_mm_transfer_act_dma(octx, kparams, act, mmctx->vtcm_act_raw, mmctx->vtcm_act_raw_stride, 0, act_nrows);@@ -4689,7 +4527,7 @@ static int op_matmul_nx_impl(struct htp_ops_context * octx) {mmctx->n_quant_tasks = n_quant_tasks;work_queue_run(octx->ctx->work_queue, quant_task_func, mmctx, n_quant_tasks);- work_queue_run(octx->ctx->work_queue, matmul_job_func, mmctx, n_matmul_jobs);+ work_queue_run(octx->ctx->work_queue, hvx_mm_nx_2d_tiled, mmctx, octx->n_threads);return HTP_STATUS_OK;}diff --git a/ggml/src/ggml-hexagon/htp/matmul-ops.h b/ggml/src/ggml-hexagon/htp/matmul-ops.hindex 522d8779fe..339eea4a7b 100644--- a/ggml/src/ggml-hexagon/htp/matmul-ops.h+++ b/ggml/src/ggml-hexagon/htp/matmul-ops.h@@ -4,6 +4,7 @@#include <stdint.h>#include <stddef.h>#include "htp-ops.h"+#include "htp-tensor.h"#include "hex-fastdiv.h"#include "hex-common.h"#include "htp-vtcm.h"@@ -19,45 +20,10 @@ extern "C" {#define HTP_MM_HMX_TILE_N_ELMS 1024#define HTP_MM_HMX_MIN_NROWS 4-// --- Weight Repacked Tile Sizes ----#define HTP_MM_WEIGHT_TILE_SIZE_Q4_0 576-#define HTP_MM_WEIGHT_TILE_SIZE_Q4_1 640-#define HTP_MM_WEIGHT_TILE_SIZE_Q8_0 1088-#define HTP_MM_WEIGHT_TILE_SIZE_IQ4_NL 576-#define HTP_MM_WEIGHT_TILE_SIZE_MXFP4 544-// Q5_K: the Q4_1 tile (640) followed by a 128-byte plane with the 5th bit of every quant, transposed so that-// plane byte l holds the eight flags of lane l: bit 2i = low nibble of nibble vector i, bit 2i+1 = high nibble-#define HTP_MM_WEIGHT_TILE_SIZE_Q5_K 768-// Q6_K native 6-bit tile (32 rows x 32 k), vrmpy-ready: byte 4*row+b of a vector holds k = 4*group+b-// vectors 0..3: low nibbles, vector i holds group 2i (low nibble) and group 2i+1 (high nibble)-// vectors 4..5: high 2 bits, vector m holds groups 4m..4m+3 at bit offsets 0,2,4,6-// vector 6: fp16 scales per row, d * scales[]: k 0..15 in lanes 0..31, k 16..31 in lanes 32..63-#define HTP_MM_WEIGHT_TILE_SIZE_Q6_K 896-// Q3_K native 3-bit tile, vrmpy-ready like Q6_K-// vectors 0..1: low 2 bits, vector m holds groups 4m..4m+3 at bit offsets 0,2,4,6-// vector 2: bit g set where the hmask bit of group g is clear (quant = low 2 bits - 4)-// vector 3: fp16 scales per row, d * (scales[] - 32): k 0..15 in lanes 0..31, k 16..31 in lanes 32..63-#define HTP_MM_WEIGHT_TILE_SIZE_Q3_K 512-// Q2_K native 2-bit tile with the ggml scale bytes, vrmpy-ready like Q6_K (same bits per weight as block_q2_K)-// bytes 0..255: unsigned 2-bit quants, vector m holds groups 4m..4m+3 at bit offsets 0,2,4,6-// bytes 256..319: scales[] bytes (min << 4 | scale), sub-block s of row r at 32 * s + r-// bytes 320..335: 8 fp16, tile j of a super-block: d (j < 4) or -dmin (j >= 4) of rows 8 * (j % 4) .. 8 * (j % 4) + 7-#define HTP_MM_WEIGHT_TILE_SIZE_Q2_K 336--// --- Weight Repacked Aligned Tile Sizes ----#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q4_0 640-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q4_1 640-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q8_0 1152-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_IQ4_NL 640-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_MXFP4 640-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q5_K 768-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q6_K 896-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q3_K 512-#define HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q2_K 384-// --- Activation Tiled Block Sizes (including padding) ---#define HTP_MM_ACT_TILE_SIZE_Q8_0 1152#define HTP_MM_ACT_TILE_SIZE_Q8_1 1280+#define HTP_MM_ACT_TILE_SIZE_F16 2048#define HTP_MM_MAX_PREFETCH 16@@ -74,11 +40,12 @@ enum htp_mm_kernel_type {// HMX pathsHTP_MM_KERNEL_HMX_2D,- HTP_MM_KERNEL_HMX_F16_BATCHED,+ HTP_MM_KERNEL_HMX_BATCHED,// HVX floating-point pathsHTP_MM_KERNEL_HVX_F16_F16_VTCM,HTP_MM_KERNEL_HVX_F32_F32_VTCM,+ HTP_MM_KERNEL_HVX_BF16_F32_VTCM,// HVX quantized pathsHTP_MM_KERNEL_HVX_QUANT_ROW, // standard row-wise parallel quantization@@ -212,57 +179,6 @@ next_nc:return 0;}-// --- Tile Size Helpers ----static inline uint32_t htp_mm_get_weight_tile_size(int weight_type) {- switch (weight_type) {- case HTP_TYPE_Q4_0:- case HTP_TYPE_IQ4_NL:- return HTP_MM_WEIGHT_TILE_SIZE_Q4_0;- case HTP_TYPE_Q4_1:- case HTP_TYPE_Q4_K:- return HTP_MM_WEIGHT_TILE_SIZE_Q4_1;- case HTP_TYPE_Q8_0:- return HTP_MM_WEIGHT_TILE_SIZE_Q8_0;- case HTP_TYPE_Q5_K:- return HTP_MM_WEIGHT_TILE_SIZE_Q5_K;- case HTP_TYPE_Q6_K:- return HTP_MM_WEIGHT_TILE_SIZE_Q6_K;- case HTP_TYPE_Q3_K:- return HTP_MM_WEIGHT_TILE_SIZE_Q3_K;- case HTP_TYPE_Q2_K:- return HTP_MM_WEIGHT_TILE_SIZE_Q2_K;- case HTP_TYPE_MXFP4:- return HTP_MM_WEIGHT_TILE_SIZE_MXFP4;- default:- return 0;- }-}--static inline uint32_t htp_mm_get_weight_aligned_tile_size(int weight_type) {- switch (weight_type) {- case HTP_TYPE_Q4_0:- case HTP_TYPE_IQ4_NL:- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q4_0;- case HTP_TYPE_Q4_1:- case HTP_TYPE_Q4_K:- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q4_1;- case HTP_TYPE_Q8_0:- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q8_0;- case HTP_TYPE_Q5_K:- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q5_K;- case HTP_TYPE_Q6_K:- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q6_K;- case HTP_TYPE_Q3_K:- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q3_K;- case HTP_TYPE_Q2_K:- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_Q2_K;- case HTP_TYPE_MXFP4:- return HTP_MM_WEIGHT_ALIGNED_TILE_SIZE_MXFP4;- default:- return 0;- }-}-// weight types whose tiles carry a per-block offset (x = d * q + m): the activations need block sums (q8_1)// (Q2_K: per-16 k sums, q8_1_s16)static inline bool htp_mm_weight_has_offset(int weight_type) {@@ -283,27 +199,17 @@ static inline size_t htp_mm_q8_1_tiled_row_size(uint32_t ne) {return nb_32 * HTP_MM_ACT_TILE_SIZE_Q8_1;}-static inline size_t htp_mm_get_tiled_row_stride(int weight_type, uint32_t k) {- uint32_t nb = (k + QK_Q4_0_TILED - 1) / QK_Q4_0_TILED;- switch (weight_type) {- case HTP_TYPE_Q4_0:- case HTP_TYPE_IQ4_NL:- case HTP_TYPE_Q4_1:- case HTP_TYPE_Q4_K:- case HTP_TYPE_Q8_0:- case HTP_TYPE_Q5_K:- case HTP_TYPE_Q6_K:- case HTP_TYPE_Q3_K:- case HTP_TYPE_Q2_K:- case HTP_TYPE_MXFP4:- return (size_t) nb * htp_mm_get_weight_tile_size(weight_type);- case HTP_TYPE_F16:- return (size_t) k * sizeof(__fp16);- case HTP_TYPE_F32:- return (size_t) k * sizeof(float);- default:- return 0;+static inline size_t htp_mm_f16_tiled_row_size(uint32_t ne) {+ const uint32_t ne_padded = ((ne + 127) / 128) * 128;+ const uint32_t nb_32 = ne_padded / 32;+ return nb_32 * HTP_MM_ACT_TILE_SIZE_F16;+}++static inline size_t htp_mm_tiled_act_row_size(int weight_type, uint32_t ne) {+ if (weight_type == HTP_TYPE_F16) {+ return htp_mm_f16_tiled_row_size(ne);}+ return htp_mm_weight_has_offset(weight_type) ? htp_mm_q8_1_tiled_row_size(ne) : htp_mm_q8_0_tiled_row_size(ne);}static inline size_t htp_mm_round_up(size_t n, size_t m) {@@ -318,8 +224,8 @@ static inline void htp_mm_hmx_get_2d_chunk_costs(int wtype, uint32_t k, bool pipeline, uint32_t aligned_tile_size,size_t * size_per_n_out, size_t * size_per_m_out, size_t * size_per_mn_out) {- const bool is_quant = (wtype != HTP_TYPE_F16 && wtype != HTP_TYPE_F32);- const size_t row_stride = htp_mm_get_tiled_row_stride(wtype, k);+ const bool is_quant = (wtype != HTP_TYPE_F16 && wtype != HTP_TYPE_F32 && wtype != HTP_TYPE_BF16);+ const size_t row_stride = htp_tensor_get_row_size(wtype, k);const size_t vec_dot_size = k * sizeof(uint16_t);const uint32_t n_k_tiles = k / HTP_MM_HMX_TILE_N_COLS;const size_t qweight_row_stride = is_quant ? (size_t)(n_k_tiles * aligned_tile_size) / 32 : 0;@@ -404,7 +310,7 @@ static inline void htp_mm_hmx_vtcm_layout_build() {size_t off = 0;- if (kernel_type == HTP_MM_KERNEL_HMX_F16_BATCHED) {+ if (kernel_type == HTP_MM_KERNEL_HMX_BATCHED) {const size_t vec_dot_size = k * sizeof(uint16_t);const size_t act_head_stride = mc * k;const size_t weight_area_size = hex_align_up(nc * vec_dot_size, HTP_MM_HMX_TILE_SIZE);@@ -450,8 +356,8 @@ static inline void htp_mm_hmx_vtcm_layout_build(off = off_group_a + hex_smax(group_b_size, group_c_size);} else {// HTP_MM_KERNEL_HMX_2D- const bool is_quant = (wtype != HTP_TYPE_F16 && wtype != HTP_TYPE_F32);- const size_t row_stride = htp_mm_get_tiled_row_stride(wtype, k);+ const bool is_quant = (wtype != HTP_TYPE_F16 && wtype != HTP_TYPE_F32 && wtype != HTP_TYPE_BF16);+ const size_t row_stride = htp_tensor_get_row_size(wtype, k);const size_t vec_dot_size = k * sizeof(uint16_t);const uint32_t n_k_tiles = k / HTP_MM_HMX_TILE_N_COLS;@@ -527,28 +433,13 @@ static inline void htp_mm_hvx_vtcm_layout_build(size_t dst_sz = 0;size_t act_raw_sz = 0;- const bool is_repack = (wtype == HTP_TYPE_Q4_0 || wtype == HTP_TYPE_Q4_1 ||- wtype == HTP_TYPE_Q8_0 || wtype == HTP_TYPE_IQ4_NL ||- wtype == HTP_TYPE_MXFP4 || wtype == HTP_TYPE_Q6_K ||- wtype == HTP_TYPE_Q4_K || wtype == HTP_TYPE_Q5_K ||- wtype == HTP_TYPE_Q3_K || wtype == HTP_TYPE_Q2_K);-if (is_fused_nx) {- const size_t src0_row_size_padded = hex_round_up(src0_row_size, 128);+ uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(wtype);+ uint32_t n_k_tiles = hex_round_up(ne10, 32) / 32;+ uint32_t tile_row_size = n_k_tiles * aligned_tile_size;+ size_t weight_sz_per_thread = hex_round_up(n_prefetch * tile_row_size, 128);- size_t weight_sz_per_thread = 0;-- if (is_repack) {- uint32_t aligned_tile_size = htp_mm_get_weight_aligned_tile_size(wtype);- uint32_t n_k_tiles = hex_round_up(ne10, 32) / 32;- uint32_t tile_row_size = n_k_tiles * aligned_tile_size;-- weight_sz_per_thread = hex_round_up(n_prefetch * tile_row_size, 128);- } else {- weight_sz_per_thread = hex_round_up(n_prefetch * src0_row_size_padded, 128);- }-- size_t tiled_act_row_size = htp_mm_weight_has_offset(wtype) ? htp_mm_q8_1_tiled_row_size(ne10) : htp_mm_q8_0_tiled_row_size(ne10);+ size_t tiled_act_row_size = htp_mm_tiled_act_row_size(wtype, ne10);size_t q_act_sz = hex_round_up(tiled_act_row_size * act_nrows, 128);size_t raw_row_size = hex_round_up(ne10 * sizeof(float), QK_Q8_0_TILED * sizeof(float));@@ -558,20 +449,13 @@ static inline void htp_mm_hvx_vtcm_layout_build(dst_sz = 0;act_raw_sz = hex_round_up(raw_row_size * act_nrows, 128);} else if (is_matmul_id) {- const size_t src0_row_size_padded = htp_mm_round_up(src0_row_size, 128);- const size_t act_row_size_tiled = htp_mm_weight_has_offset(wtype) ? htp_mm_q8_1_tiled_row_size(ne10)- : htp_mm_q8_0_tiled_row_size(ne10);-- size_t src0_sz_per_thread = htp_mm_round_up(n_prefetch * src0_row_size_padded, 256);- act_sz = htp_mm_round_up(act_row_size_tiled * act_nrows, 256);-- if (is_repack) {- const uint32_t aligned_tile_size = htp_mm_get_weight_aligned_tile_size(wtype);- const uint32_t n_k_tiles = ne10 / 32;- const uint32_t tile_row_size = n_k_tiles * aligned_tile_size;- size_t repacked_vtcm_size = htp_mm_round_up(n_prefetch * tile_row_size, 256);- src0_sz_per_thread = repacked_vtcm_size;- }+ const size_t act_row_size_tiled = htp_mm_tiled_act_row_size(wtype, ne10);+ const uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(wtype);+ const uint32_t n_k_tiles = ne10 / 32;+ const uint32_t tile_row_size = n_k_tiles * aligned_tile_size;+ size_t src0_sz_per_thread = htp_mm_round_up(n_prefetch * tile_row_size, 256);++ act_sz = htp_mm_round_up(act_row_size_tiled * act_nrows, 256);size_t raw_row_size = hex_round_up(ne10 * sizeof(float), QK_Q8_0_TILED * sizeof(float));@@ -592,7 +476,8 @@ static inline void htp_mm_hvx_vtcm_layout_build(act_raw_sz = hex_round_up(hex_round_up(ne10 * sizeof(float), 128) * act_nrows, 128);break;}- case HTP_MM_KERNEL_HVX_F32_F32_VTCM: {+ case HTP_MM_KERNEL_HVX_F32_F32_VTCM:+ case HTP_MM_KERNEL_HVX_BF16_F32_VTCM: {size_t f32_act_row_size = htp_mm_round_up(ne10 * 4, 128);act_sz = htp_mm_round_up(f32_act_row_size * act_nrows, 256);src0_sz = htp_mm_round_up(n_prefetch * src0_row_size_padded, 256) * n_threads;@@ -602,21 +487,15 @@ static inline void htp_mm_hvx_vtcm_layout_build(}case HTP_MM_KERNEL_HVX_QUANT_BLOCK:case HTP_MM_KERNEL_HVX_QUANT_ROW: {- size_t q_act_row_size = htp_mm_weight_has_offset(wtype) ? htp_mm_q8_1_tiled_row_size(ne10) : htp_mm_q8_0_tiled_row_size(ne10);+ size_t q_act_row_size = htp_mm_tiled_act_row_size(wtype, ne10);+ uint32_t aligned_tile_size = htp_tensor_get_tile_padded_size(wtype);+ uint32_t n_k_tiles = (ne10 + 31) / 32;+ uint32_t tile_row_size = n_k_tiles * aligned_tile_size;+ size_t tiled_vtcm_size = htp_mm_round_up(n_prefetch * tile_row_size, 256);- src0_sz = htp_mm_round_up(n_prefetch * src0_row_size_padded, 256);+ src0_sz = tiled_vtcm_size * n_threads;act_sz = htp_mm_round_up(q_act_row_size * act_nrows, 256);- src0_sz = src0_sz * n_threads;-- if (is_repack) {- uint32_t aligned_tile_size = htp_mm_get_weight_aligned_tile_size(wtype);- uint32_t n_k_tiles = ne10 / 32;- uint32_t tile_row_size = n_k_tiles * aligned_tile_size;- size_t repacked_vtcm_size = htp_mm_round_up(n_prefetch * tile_row_size, 256);- src0_sz = repacked_vtcm_size * n_threads;- }-size_t dst_slice_per_thread = (dst_nrows > 0 && act_nrows == 1) ? htp_mm_round_up((dst_row_size + n_threads - 1) / n_threads, 128) : 0;dst_sz = dst_slice_per_thread * n_threads;size_t raw_row_size = hex_round_up(ne10 * sizeof(float), QK_Q8_0_TILED * sizeof(float));@@ -687,9 +566,7 @@ static inline bool htp_mm_hvx_solve_vtcm_params(const size_t avail_act = vtcm_budget - fixed_bytes;size_t row_size = 0;if (kernel_type == HTP_MM_KERNEL_HVX_QUANT_ROW || kernel_type == HTP_MM_KERNEL_HVX_QUANT_BLOCK) {- row_size = htp_mm_weight_has_offset(wtype)- ? htp_mm_q8_1_tiled_row_size(ne10)- : htp_mm_q8_0_tiled_row_size(ne10);+ row_size = htp_mm_tiled_act_row_size(wtype, ne10);} else if (kernel_type == HTP_MM_KERNEL_HVX_F16_F16_VTCM) {row_size = hex_round_up(ne10 * 2, 128);} else {@@ -749,7 +626,7 @@ static inline size_t htp_mm_hmx_get_batched_vtcm_size(int wtype, uint32_t k, size_t mc, size_t nc, uint32_t group_size, bool pipeline, uint32_t act_threads, size_t bias_size) {(void)pipeline;struct htp_mm_hmx_vtcm_layout L;- htp_mm_hmx_vtcm_layout_build(&L, HTP_MM_KERNEL_HMX_F16_BATCHED, wtype, k, mc, nc, group_size, false, act_threads, 0, bias_size);+ htp_mm_hmx_vtcm_layout_build(&L, HTP_MM_KERNEL_HMX_BATCHED, wtype, k, mc, nc, group_size, false, act_threads, 0, bias_size);return L.total_bytes;}diff --git a/scripts/snapdragon/run.py b/scripts/snapdragon/run.pyindex dec4b996fc..7e11d90de8 100755--- a/scripts/snapdragon/run.py+++ b/scripts/snapdragon/run.py@@ -27,6 +27,7 @@ MANAGED_ENV_NAMES = ("GGML_HEXAGON_OPPOLL","GGML_HEXAGON_OPFILTER","GGML_HEXAGON_OPFUSION",+ "GGML_HEXAGON_RT_REPACK","GGML_HEXAGON_VMEM","GGML_HEXAGON_MBUF","GGML_HEXAGON_MM_SELECT",@@ -167,6 +168,7 @@ def main():parser.add_argument("--hex-oppoll", default="1", help="Enable (1) or Disable (0) polling for NPU opbatch completion (GGML_HEXAGON_OPPOLL) (default: 1)")parser.add_argument("--hex-opfilter", help="Regex pattern to filter/select which operators are offloaded to NPU (GGML_HEXAGON_OPFILTER)")parser.add_argument("--hex-opfusion", help="NPU graph node fusion optimization level (0: disabled, 1: enabled) (GGML_HEXAGON_OPFUSION)")+ parser.add_argument("--hex-rt-repack", help="Runtime weight repacking mode (0: disabled, 1: quantized only, 2: always) (GGML_HEXAGON_RT_REPACK)")parser.add_argument("--hex-vmem", help="Maximum NPU VMEM size limit in MB to allocate (GGML_HEXAGON_VMEM)")parser.add_argument("--hex-mbuf", help="Host buffer size limits in MB (supports K/M/G suffix): <dyn>[,<static>[,<total>]] (default: 512,1024,0) (GGML_HEXAGON_MBUF)")parser.add_argument("--hex-mm-select", help="Select MUL_MAT and MUL_MAT_ID kernel (GGML_HEXAGON_MM_SELECT) 2:HMX,1:HVX,0:disable")@@ -311,6 +313,7 @@ def main():set_env("GGML_HEXAGON_OPPOLL", args.hex_oppoll)set_env("GGML_HEXAGON_OPFILTER", args.hex_opfilter)set_env("GGML_HEXAGON_OPFUSION", args.hex_opfusion)+ set_env("GGML_HEXAGON_RT_REPACK", args.hex_rt_repack)set_env("GGML_HEXAGON_VMEM", args.hex_vmem)set_env("GGML_HEXAGON_MBUF", args.hex_mbuf)set_env("GGML_HEXAGON_MM_SELECT", args.hex_mm_select)diff --git a/tests/test-backend-ops.cpp b/tests/test-backend-ops.cppindex ba7f7eb4e5..b50364660e 100644--- a/tests/test-backend-ops.cpp+++ b/tests/test-backend-ops.cpp@@ -10741,11 +10741,14 @@ static std::vector<std::unique_ptr<test_case>> make_test_cases_eval() {// testing very large Ntest_cases.emplace_back(new test_mul_mat(GGML_TYPE_F16, GGML_TYPE_F32, 16, 2105344, 32, {1, 1}, {1, 1}));++#if 0test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F16, GGML_TYPE_F16, 16, 8400000, 32, {1, 1}, {1, 1}));test_cases.emplace_back(new test_mul_mat(GGML_TYPE_Q4_0, GGML_TYPE_F32, 16, 8400000, 32, {1, 1}, {1, 1}));// same shapes with a repeat in dim 3test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F16, GGML_TYPE_F16, 16, 8400000, 32, {1, 1}, {2, 1}));test_cases.emplace_back(new test_mul_mat(GGML_TYPE_Q4_0, GGML_TYPE_F32, 16, 8400000, 32, {1, 1}, {2, 1}));+#endiftest_cases.emplace_back(new test_mul_mat(GGML_TYPE_F32, GGML_TYPE_F32, 31, 509, 2051, {1, 1}, {1, 1}));test_cases.emplace_back(new test_mul_mat(GGML_TYPE_F32, GGML_TYPE_F32, 32, 509, 2112, {1, 1}, {1, 1}));