Run frontier MoE models on hardware you already own β pure C, zero deps, experts streamed from disk. Tiny engine, immense model. π¦
C
Commits on main
Merge pull request #397 from NeuralNotwerk/fix/mtp-int8-repair
66b5e57 Vincenzo Fornaro committed 4 months ago
Merge pull request #395 from Stonki13/fix/windows-cuda-detection
3f99d2b Vincenzo Fornaro committed 4 months ago
Merge pull request #404 from anrasi/fix/indir-meta-resume
19bab42 Vincenzo Fornaro committed 4 months ago
Merge pull request #402 from anrasi/fix/coli-run-think
36d389b Vincenzo Fornaro committed 4 months ago
Fix CUDA detection on Windows: cuda_binary()/cuda_linkage() always returned False
d2d3a7b Stonki13 committed 4 months ago
tools: repair_mtp_int8.py β fix int4-converted MTP heads in place; warn on --mtp --ebits <8
d6676c1 recviking committed 4 months ago
convert(fp8->int4): --indir copies the full metadata set and resumes interrupted passes (#383)
308e269 andres-gb10 committed 4 months ago
convert: resolve --mtp/--indexer shards from the local index instead of scanning all of them (#383)
6a2ab47 bokiko committed 4 months ago
Merge pull request #384 from ZacharyZcR/perf/auto-numa
caa49f7 Vincenzo Fornaro committed 4 months ago
Merge pull request #372 from ZacharyZcR/feat/paged-ragged-kv
3fed654 Vincenzo Fornaro committed 4 months ago
serve: honor --ngen when the client omits max_tokens; convert: print the resolved plan (#382, #383)
f8e4dc9 JustVugg committed 4 months ago
security: extend the audit β olmoe loader, tokenizer, server slowloris
fb45de2 KingIcyCreamProjects committed 4 months ago
security: SEC-9 β bound untrusted config/oracle JSON slurps
df644dc KingIcyCreamProjects committed 4 months ago
security: harden model-file parsing, server, and build per audit
72dd9ac KingIcyCreamProjects committed 4 months ago
docs(api): tell coding-CLI users about the prefill cost before they hit it (#373)
8a1dcca JustVugg committed 4 months ago
Merge pull request #365 from ZacharyZcR/feat/ragged-batch-attention-upstream
605b0e2 Vincenzo Fornaro committed 4 months ago
serve: MTP status message tells the truth in multiplexed mode (#358)
b09273f JustVugg committed 4 months ago
telemetry: split expert hits into pin-tier vs LRU ecache (#336)
679c074 JustVugg committed 4 months ago
convert(fp8->int4): --mtp/--indexer respected on the --indir path, no more container overwrite (#355)
db09466 JustVugg committed 4 months ago
docs: collapse WINDOWS.md/windows.md case collision into one file (#371)
eb6d432 JustVugg committed 4 months ago
sampling: survive non-finite logits instead of emitting token 0 forever (#369)
4fb5b4e JustVugg committed 4 months ago