Skip to content

Run frontier MoE models on hardware you already own — pure C, zero deps, experts streamed from disk. Tiny engine, immense model. 🐦

C

perf(ablate): stop re-scanning for an already-full top-k slot buffer

monotophic95b70e576995
diff --git a/c/colibri.c b/c/colibri.c
index a5ac503a..2b84b666 100644
--- a/c/colibri.c
+++ b/c/colibri.c
@@ -8083,7 +8083,18 @@ static void logprob_refusal(const char *surface, unsigned long long owner,
* k=2 emits token ids [2,1].
*
* Exceptional rows are rejected by the classified numeric-status layer before
- * this finite-only selector is reached. */
+ * this finite-only selector is reached.
+ *
+ * The k slots fill in index order 0..k-1 during the first k iterations and
+ * never empty again, so "is there still an empty slot" is exactly "have
+ * fewer than k been filled" -- an O(1) counter, not the O(k) linear scan of
+ * tk_id[] a previous version of this loop re-ran on every one of the V-k
+ * remaining iterations even though it could only ever find one answer by
+ * then. That scan was loop-invariant work paid V-k times over for a result
+ * that is invariant after the first k: replacing it measurably speeds up
+ * the selection without changing which slot is chosen or when (verified by
+ * running both forms against several thousand rows, real vocab size
+ * included, and diffing tk_id[]/tk_val[]/the return value bit for bit). */
static int logit_topk_select(const float *lo, int V, int requested,
int *tk_id, float *tk_val,
int *status_out){
@@ -8100,11 +8111,12 @@ static int logit_topk_select(const float *lo, int V, int requested,
return 0;
}
- for(int j=0;j<k;j++) tk_id[j]=-1;
+ int filled=0;
for(int i=0;i<V;i++){
- int slot=-1;
- for(int j=0;j<k;j++) if(tk_id[j]<0){ slot=j; break; }
- if(slot<0){
+ int slot;
+ if(filled<k){
+ slot=filled++;
+ }else{
int mn=0;
for(int j=1;j<k;j++) if(tk_val[j]<tk_val[mn]) mn=j;
if(!(lo[i]>tk_val[mn])) continue;