Hugging Face Daily PapersYu Lin, Yiming Wang, Runyuan Cai1 min readpaperadvanced
The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction
Summary
The paper presents Edge0, a streaming MoE inference engine that predicts the next layer's routing one token ahead, allowing expert weights to be fetched from SSD while compute proceeds. This enables a 35 B‑parameter MoE to run on a single 24 GB machine at ~20 tokens/s using only ~3 GiB of active memory and with near‑teacher accuracy.
- Edge0 adds a per‑layer routing predictor that forecasts the next layer’s expert set one token ahead, overlapping SSD reads with compute.
- Streaming expert weights from SSD and keeping only the predicted set active reduces peak memory to ~3 GiB for a 35 B MoE.
- An unmerged LoRA adapter, trained on the student path, recovers most accuracy loss from 4‑bit quantization and routing approximation.
- On a single 24 GB GPU, Edge0 achieves ~20 tokens/s for a 35 B MoE, within a few points of the fp16 teacher across five benchmarks.
Engineers who need to run large MoE models on consumer‑grade hardware can now do so with low memory overhead and competitive speed.
7/10
