proomt

Search

Search posts, papers, and topics

Hall of Fame

Hall of FameAshish Vaswani et al.201727 min readpaperadvanced

Attention Is All You Need

Read on arxiv.org

Summary

The paper proposes the Transformer, a sequence‑to‑sequence model that relies solely on self‑attention, eliminating recurrence and convolutions. It achieves state‑of‑the‑art translation BLEU scores while training orders of magnitude faster.

  • Self‑attention (scaled dot‑product) and multi‑head attention replace RNN/CNN layers, enabling full parallelism across sequence positions.
  • Positional encodings using sinusoids inject order information without recurrence.
  • Six identical encoder and decoder layers with residual connections and layer‑norm yield a 512‑dimensional model.
  • Transformer reaches 28.4 BLEU (EN‑DE) and 41.8 BLEU (EN‑FR) with 12‑hour training on 8 GPUs, far cheaper than prior models.

Anyone building or researching sequence models should understand the Transformer, as it underpins modern LLMs and most state‑of‑the‑art NLP systems.

10/10
architecturenlpdeep learningattentiontransformers

Related reading

  1. Transformers Explained Visually

    The article walks through the core components of a text‑generative Transformer—embedding, multi‑head self‑attention, MLP, and output projection—using GPT‑2 small as a concrete example. It shows the dimensions, parameter counts, and step‑by‑step calculations that underlie token prediction.

    Hacker News front pagegithub.io11 minHN59487

  2. VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention

    VC-Attention introduces a training‑free low‑bit attention pipeline for diffusion transformers. It smooths value tensors via lightweight online clustering (V‑Smooth) and quantizes only the residual after subtracting block means, restoring the mean from the softmax row sum. It also replaces the FP32 softmax exponential with a fused FP8 cast (ExpCast‑FP8) that maps log‑scores directly to E4M3 probab…

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models

    Latent Interface Training (LIT) first learns a goal‑conditioned action prior without visual input, then adds a pose‑supervised latent interface as the only visual conditioning path. Applied to several vision‑language‑action models, LIT cuts vision‑action shortcuts and lifts LIBERO‑Plus success by 3.9–10.7 points and real‑world task success by 13.3–16.7 points under distribution shifts.

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. HuRo: Robotizing Human Videos for Scalable VLA Pretraining

    The paper introduces a pipeline that converts heterogeneous human videos into robot‑aligned observations and actions, creating the 630K‑episode HuRo dataset. Pretraining vision‑language‑action (VLA) policies on this data boosts real‑world manipulation success from ~51% to ~80% and improves out‑of‑distribution robustness.

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. The Router Within: Eliciting Native Skill Routing from a Frozen LLM

    The paper introduces Gavel, a method that extracts a frozen LLM's internal routing signal via two trained linear maps, eliminating the need to embed skill descriptions in the prompt. Experiments on Qwen3‑32B show up to 13.4‑point improvements on task benchmarks and higher skill‑use accuracy compared to larger retrieval‑based systems.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. Think Like a World Model, Act Like a VLA: Distilling World-Model Representations into Compact Robot Policies

    Distill frozen world‑model features into a Vision‑Language‑Action policy via a single alignment loss; no teacher at train time, no extra runtime cost. 0.8 B student runs 32 ms / 1.86 GB on RTX 5090, hits 97.9 % on LIBERO and improves RoboCasa‑GR1 from 48.2 % to 50.5 %, with transfer to real single‑arm and bimanual robots.

    Hugging Face Daily Papersarxiv.org1 minpaper