proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersRan Cheng, Longfei Xu, Zheng Liu2 min readpaperadvanced

IntBMoE: Integrating Block-Level Conditioning into Expert Composition for Full-Participation Mixture-of-Experts

Read on arxiv.org

Summary

IntBMoE introduces block‑level conditioning to MoE, decoupling token participation, compute execution, and memory materialization. A hypernetwork merges all experts into a composed expert per block, while routing remains sparse. Dual‑Path Residual Gating further mixes two composed paths. Experiments show consistent gains on vision, language, and recommendation tasks, and the model is live in AMap…

  • Full token participation is achieved without dense compute by composing experts per block via a learned codebook.
  • Sparse routing limits execution cost; materialization is bounded by the fixed codebook size.
  • Dual‑Path Residual Gating (DPRG) multiplies two independently composed expert paths for richer representations.
  • Empirical results: +0.5‑1.2 % top‑1 accuracy on ImageNet‑like benchmarks vs. sparse/dense MoE baselines; language modeling perplexity improvements; 2.4 % UVCTR lift in production recommendation under 60 ms latency.

MoE scaling traditionally trades off between compute, memory, and the number of experts a token sees. IntBMoE’s block‑conditioned composition breaks this trade‑off, enabling high‑capacity models that stay cheap to run and fit in memory—critical for latency‑sensitive, high‑traffic services like reco…

8/10
machine learningarchitectureperformanceml inframixture of expertsrecommendation

Related reading

  1. Article: Beyond Relevance: A Governance-First Architecture for Enterprise Personalization

    The article proposes a governance‑first architecture for enterprise personalization, where policy‑driven steps (memory, journey graph, AI routing, scoring, trust checks, outcome simulation) shape the recommendation before it is returned. A reference FastAPI implementation demonstrates the pattern with external YAML policies and optional LLM assistance.

    InfoQinfoq.com19 min

  2. Beyond Top-k Skill Retrieval: Diversity-Aware Skill Routing for LLM Agents

    Beyond Top‑k Skill Retrieval: Diversity‑Aware Skill Routing (DSR) applies a Determinantal Point Process with a query‑residual diversity kernel to rerank skill candidates, balancing relevance and redundancy. On the SkillRouter benchmark it raises recall and full‑coverage, especially for multi‑skill queries, showing that skill routing benefits from set‑selection rather than independent ranking.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. Online Learning with LLM Experts from Limited Feedback

    The paper models prompt routing to multiple LLM experts as a bandit problem with limited feedback and proposes algorithms that achieve sublinear regret in both full‑information and bandit settings. Experiments demonstrate that the methods learn effective routing strategies across diverse LLMs using only a small feedback budget.

    Hugging Face Daily Papersarxiv.org2 minpaper

  4. MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup

    The paper proposes Mixture of Memory Embeddings (MoME), a context‑aware sparse lookup that replaces each token’s single memory row with a gated mixture of multiple slots. Experiments on Llama‑3, MobileLLM and Qwen3 show MoME outperforms existing memory‑embedding baselines at equal parameter and FLOP budgets and exhibits interpretable routing for polysemous tokens.

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction

    The paper presents Edge0, a streaming MoE inference engine that predicts the next layer's routing one token ahead, allowing expert weights to be fetched from SSD while compute proceeds. This enables a 35 B‑parameter MoE to run on a single 24 GB machine at ~20 tokens/s using only ~3 GiB of active memory and with near‑teacher accuracy.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. BI-Agent and BI-Bench: Towards Automating End-to-End Business Intelligence

    The paper introduces BI‑Bench, a new benchmark of real‑world BI questions derived from public dashboards, and BI‑Agent, a tool‑augmented LLM system that breaks BI workflows into search, join, and transform subtasks. Baseline LLMs hit <50 % accuracy on BI‑Bench. By orchestrating specialized data‑management tools and post‑training the model with supervised fine‑tuning and reinforcement learning on…

    Hugging Face Daily Papersarxiv.org2 minpaper