Alpha-Orionis)
Out-of-core frontier MoE inference and pre-packed streaming bundles for consumer hardware (12 GB VRAM + 16 GB RAM).
Betelgeuse-AI builds bel-cli / bel-server and pre-split GGUF + contiguous expert-pack bundles designed to run 70Bβ125B+ Mixture-of-Experts models at interactive speeds directly off consumer NVMe SSDsβwith zero gate pruning, zero dropped experts, and 100% byte-exact output parity.
Betelgeuse-AI/Alpha-Orionis-XS
A zero-duplication, streaming-ready bundle of Qwen 3.8 Flash Next (125B MoE, 48 layers, 512 experts/layer, top-10 routed, GSQ-RCO IQ2_XS):
| Artifact | Size | Role |
|---|---|---|
Qwen3.8-Flash-Next-Dense.gguf |
3.2 GB | Dense backbone (Attention, 36 Gated DeltaNet layers, 12 QSA layers, Shared Experts, Routers, LM Head, Tokenizer) β resident in VRAM |
experts.bin |
35.3 GB | All 24,576 routed experts pre-packed & 4096-byte aligned ([gate | up | down]) for single-read io_uring streaming |
Qwen3.8-Flash-Next-PLE.gguf |
26.8 GB | Per-layer token embedding (per_layer_token_embd.weight) shard, active rows staged to CUDA0 |
profile-qwen4exp.bin |
10 KB | Pre-computed arrival-slack hot expert profile |
RTX 3060 12 GB + 16 GB Single-Channel DDR4, MemoryMax=12G)
| Runtime | Cold Decode (Run 1) | Warm Decode (Run 2) | Peak Bursts | Prompt Eval (PP) | Physical NVMe Reads | Output Parity |
|---|---|---|---|---|---|---|
Stock llama.cpp (mmap) |
1.47 tok/s | 1.63 tok/s | ~2.0 tok/s | 14.1 β 20.8 tok/s | 208 β 312 MiB/tok | Reference |
Strata (--low-ram mmap) |
8.11 tok/s | 8.11 tok/s | ~9.0 tok/s | 34.6 tok/s | ~105 ms pool wait | ~0.022 KL |
Betelgeuse (bel-server) |
16.80 tok/s | 22.03 tok/s | Up to 48 tok/s | 16.2 β 19.4 tok/s | 12.43 MiB/tok | 100% Byte-Exact |
KICK / WAIT): Resolves resident VRAM slots in ~2 Β΅s on-device and overlaps Shared-Expert GPU compute with host miss servicing (1 graph split, full CUDA Graph replay).0..4) where compute slack is zero, paired with an exclusive host RAM victim cache that eliminates duplicate caching between VRAM and the OS page cache.Betelgeuse-AI/Alpha-Orionis-XS β Qwen 3.8 Flash Next (IQ2_XS) pre-packed streaming bundle + bel-cli / bel-server.1017) out-of-core streaming bundle (currently reaching ~25 tok/s on 12 GB VRAM + 16 GB RAM in internal testing).