mixture of experts posts and papers

IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts arxiv.org

IntBMoE introduces block‑level conditioning to MoE, decoupling token participation, compute execution, and memory materialization. A hypernetwork merges all experts into a composed expert per block, while routing remains sparse. Dual‑Path Residual Gating further mixes two composed paths. Experiments show consistent gains on vision, language, and recommendation tasks, and the model is live in AMap…

Hugging Face Daily Papersarxiv.orgRan Cheng, Longfei Xu, Zheng Liu4d2 minpaper

MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup arxiv.org

The paper proposes Mixture of Memory Embeddings (MoME), a context‑aware sparse lookup that replaces each token’s single memory row with a gated mixture of multiple slots. Experiments on Llama‑3, MobileLLM and Qwen3 show MoME outperforms existing memory‑embedding baselines at equal parameter and FLOP budgets and exhibits interpretable routing for polysemous tokens.

Hugging Face Daily Papersarxiv.orgMuchen Li, Leonid Sigal, Renjie Liao8d1 minpaper

mixture of experts

IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Expertsarxiv.org

MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookuparxiv.org

IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts arxiv.org

MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup arxiv.org