proomt

Search

Search posts, papers, and topics

All posts

Hacker News front page11 min readintermediate

Transformers Explained Visually

Read on github.ioHN 238·39 comments

Summary

The article walks through the core components of a text‑generative Transformer—embedding, multi‑head self‑attention, MLP, and output projection—using GPT‑2 small as a concrete example. It shows the dimensions, parameter counts, and step‑by‑step calculations that underlie token prediction.

  • GPT-2 small uses 12 transformer blocks, each with 12 attention heads, 768‑dim embeddings, and a 4× expansion MLP (768→3072→768).
  • The embedding matrix shape is (50,257,768) ≈ 39 M parameters; positional encodings are learned jointly.
  • Self‑attention computes Q, K, V via linear projections, scales dot‑product, masks future tokens, then applies softmax to get attention weights.
  • Multi‑head attention splits Q/K/V into 12 heads, enabling parallel capture of different syntactic and semantic relationships.

Anyone building, fine‑tuning, or debugging LLMs needs a clear mental model of the architecture’s shapes and operations.

6/10
machine learningml infradeep learningattentiontransformers

Related reading

  2. VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention

    VC-Attention introduces a training‑free low‑bit attention pipeline for diffusion transformers. It smooths value tensors via lightweight online clustering (V‑Smooth) and quantizes only the residual after subtracting block means, restoring the mean from the softmax row sum. It also replaces the FP32 softmax exponential with a fused FP8 cast (ExpCast‑FP8) that maps log‑scores directly to E4M3 probab…

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. Register Tokens for Bounded-State Reasoning in Diffusion Language Models

    Register tokens are fixed‑position embeddings that store a compact hidden state across diffusion‑based language model generation chunks, enabling bounded‑state reasoning without retaining all prior text. Post‑training on LLaDA and Dream shows up to +8.5 math and +19.5 code benchmark points versus plain text carry, and RL fine‑tuning further improves long‑horizon tasks.

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. The top AI text generators in 2026

    Zapier’s 2026 roundup treats AI text generation as a ubiquitous feature rather than a standalone product. It explains that modern LLMs with long context windows have pushed dedicated generators into niche roles (marketing, SEO, agents) and then surveys the most useful built‑in generators across email (Gmail Gemini, Outlook Copilot), document suites (Google Docs, Microsoft Word, Notion), chatbots…

    Zapier Engineeringzapier.com9 min

  5. Breaking the Vision-Action Shortcut: Latent Interface Training for Generalizable Robotics Foundation Models

    Latent Interface Training (LIT) first learns a goal‑conditioned action prior without visual input, then adds a pose‑supervised latent interface as the only visual conditioning path. Applied to several vision‑language‑action models, LIT cuts vision‑action shortcuts and lifts LIBERO‑Plus success by 3.9–10.7 points and real‑world task success by 13.3–16.7 points under distribution shifts.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. In-Context Robot Learning with VLM Agents

    GPT‑Policy is a framework that lets a large vision‑language model (e.g. GPT‑6 Astra) perform in‑context robot learning: a context compiler extracts visual transitions from demos, the VLM proposes tool actions, and a constrained controller verifies and executes them. Real‑robot experiments show that raw video demos improve success rates even without explicit action labels, and that providing align…

    Hugging Face Daily Papersarxiv.org1 minpaper