Hugging Face Daily PapersYanbei Chen, Anirudh Goyal, Raghuraman KrishnamoorthiSep 30, 20261 min readpaperadvanced

Scaling Laws for Looped Mixture of Experts

Summary

This paper introduces Loop Scaling Laws, which jointly model recurrence and sparsity in Mixture-of-Experts (MoE) transformers. It finds that recurrence and sparsity offer complementary efficiency gains, improving prediction accuracy and enabling more efficient large model design.

New Loop Scaling Laws model recurrence and MoE sparsity together for transformer performance.
Sparsity provides ~3x active-parameter efficiency in MoE models.
Recurrence yields ~2x total-parameter efficiency, especially for reasoning tasks.
Jointly scaling recurrence and sparsity further improves performance frontiers.

This work provides a principled framework for researchers and engineers to design more efficient and performant large language models by leveraging both recurrence and sparse MoE architectures.

8/10

llm machine learning architecture performance moe scaling laws

Related reading

IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts arxiv.org
IntBMoE introduces block‑level conditioning to MoE, decoupling token participation, compute execution, and memory materialization. A hypernetwork merges all experts into a composed expert per block, while routing remains sparse. Dual‑Path Residual Gating further mixes two composed paths. Experiments show consistent gains on vision, language, and recommendation tasks, and the model is live in AMap…
Hugging Face Daily Papersarxiv.orgRan Cheng, Longfei Xu, Zheng Liu18d2 minpaper
Disentangling Representation Evolution in Transformers through Directional Decomposition arxiv.org
The paper decomposes transformer representation updates into parallel and perpendicular components, showing that parallel updates beyond the residual path are significant. Suppressing parallel updates during pretraining improves validation loss and downstream metrics.
Hugging Face Daily Papersarxiv.orgShwai He, Haichao Zhang, Shen Yan22d1 minpaper
Scaling Laws for Neural Language Models arxiv.org
This paper empirically studies scaling laws for neural language model performance, finding that cross-entropy loss scales as a power-law with model size, dataset size, and compute. It shows that optimal compute-efficient training involves using very large models, training on relatively modest data, and stopping significantly before convergence.
Hall of Famearxiv.orgJared Kaplan et al.7y67 minpaper
WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing arxiv.org
WhiteMatter introduces all-to-all cross-layer connections in Transformers by mixing past-token representations from any depth into shared KV cache channels. This approach allows for performance comparable to 50% larger standard Transformers or improved performance with half the KV cache size. It also addresses training slowdowns with a novel cyclic iteration method.
Hugging Face Daily Papersarxiv.orgWenbo Zhang, Xiang Ren9d1 minpaper
Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It arxiv.org
The paper shows that standard pretrained transformers stop using deeper layers after only a few lines of context. Adding a rank‑8 LoRA to an early layer while keeping the rest frozen dramatically extends effective reasoning depth, boosting exact accuracy on long‑chain tasks from ~15% to near‑100%.
Hugging Face Daily Papersarxiv.orgZehao Jin, Ruixuan Deng, Junran Wang7d1 minpaper
The current balance of power in open models interconnects.ai
Hacker News front pageinterconnects.ai13dHN12858

Scaling Laws for Looped Mixture of Experts

Summary

Related reading

IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Expertsarxiv.org

Disentangling Representation Evolution in Transformers through Directional Decompositionarxiv.org

Scaling Laws for Neural Language Modelsarxiv.org

WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixingarxiv.org

Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes Itarxiv.org

The current balance of power in open modelsinterconnects.ai

IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts arxiv.org

Disentangling Representation Evolution in Transformers through Directional Decomposition arxiv.org

Scaling Laws for Neural Language Models arxiv.org

WhiteMatter: All-to-All Cross-Layer Connections via KV Source Mixing arxiv.org

Transformers Stop Thinking Too Early, and a Tiny LoRA Fixes It arxiv.org

The current balance of power in open models interconnects.ai