proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersShwai He, Haichao Zhang, Shen Yan1 min readpaperadvanced

Disentangling Representation Evolution in Transformers through Directional Decomposition

Read on arxiv.org

Summary

The paper decomposes transformer representation updates into parallel and perpendicular components, showing that parallel updates beyond the residual path are significant. Suppressing parallel updates during pretraining improves validation loss and downstream metrics.

  • Parallel components of attention/MLP updates are substantial and space‑dependent, unlike the residual identity path.
  • Excluding self‑value parallel manipulation preserves the token's own message while scaling non‑self aggregates, improving edit robustness.
  • Perpendicular error isolates compression method differences more clearly than parallel error, aiding compression diagnostics.
  • Training‑time suppression of full‑aggregate parallel updates lowers validation loss trajectories; the value‑space variant yields the strongest downstream gains.

LLM researchers and engineers should care because the decomposition offers a practical tool for diagnosing representation dynamics, improving robustness to edits, and guiding training interventions.

8/10
llmmachine learningtransformerscompressionrepresentation learning

Related reading

  1. Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem

    The authors cast transformer block removal as a constrained binary optimization problem equivalent to an Ising glass, using a Hessian‑derived energy as a proxy for downstream quality. Solving the resulting QUBO with classical or quantum‑inspired solvers yields up to 23 MMLU points improvement over prior block‑removal baselines at 50 % depth compression.

    Hugging Facehuggingface.co8 min

  2. Learning Sparse Decision Trees via Transformer Variational Auto-Encoders

    The paper presents TREVIS, a method that encodes decision trees into a continuous latent space using a Tree Transformer VAE, allowing gradient-based optimization of both accuracy and structural sparsity. Experiments claim TREVIS matches the predictive performance of near-optimal algorithms while producing sparser trees.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus

    This paper investigates the "lossless" claim of Orthrus, a hybrid architecture for accelerating LLM inference. It finds that under BF16 precision, Orthrus diverges from the exact autoregressive output trajectory in over 50% of cases, though FP32 maintains exact matching. Despite BF16 divergence, downstream task performance was not systematically degraded.

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention

    VC-Attention introduces a training‑free low‑bit attention pipeline for diffusion transformers. It smooths value tensors via lightweight online clustering (V‑Smooth) and quantizes only the residual after subtracting block means, restoring the mean from the softmax row sum. It also replaces the FP32 softmax exponential with a fused FP8 cast (ExpCast‑FP8) that maps log‑scores directly to E4M3 probab…

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. The Router Within: Eliciting Native Skill Routing from a Frozen LLM

    The paper introduces Gavel, a method that extracts a frozen LLM's internal routing signal via two trained linear maps, eliminating the need to embed skill descriptions in the prompt. Experiments on Qwen3‑32B show up to 13.4‑point improvements on task benchmarks and higher skill‑use accuracy compared to larger retrieval‑based systems.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. Article: Architecting Secure and Scalable Facial Verification Systems

    A real‑world post‑mortem of a high‑volume face verification service that moved from a naïve synchronous API to an async, layered pipeline (edge validation, preprocessing, decoupled detection/verification, decision engine) to achieve 8.5k rpm, p99 < 1.8 s, 30 % cost savings, and strict privacy controls.

    InfoQinfoq.com15 min