proomt

Search

Search posts, papers, and topics

Hall of Fame

Hall of FameJordan Hoffmann et al.202266 min readpaperadvanced

Training Compute-Optimal Large Language Models

Read on arxiv.org

Summary

The paper derives a compute‑optimal scaling law showing model size and training tokens should grow together, and validates it by training a 70B‑parameter model (Chinchilla) on 1.4 T tokens that outperforms much larger LLMs.

  • Optimal scaling: for a fixed FLOP budget, double the number of parameters and double the number of training tokens.
  • Chinchilla (70B, 1.4 T tokens) beats larger models like Gopher (280B) and GPT‑3 (175B) across many benchmarks.
  • Aligning the learning‑rate schedule length with the token count yields lower final loss than a fixed schedule.
  • Compute‑optimal models cut inference FLOPs, reducing downstream cost and enabling deployment on smaller hardware.

Anyone budgeting compute for training LLMs should use the equal‑scaling rule to get better performance and lower inference costs.

9/10
llmperformancetransformerslanguage modelsscaling lawscompute optimal

Related reading

  1. Register Tokens for Bounded-State Reasoning in Diffusion Language Models

    Register tokens are fixed‑position embeddings that store a compact hidden state across diffusion‑based language model generation chunks, enabling bounded‑state reasoning without retaining all prior text. Post‑training on LLaDA and Dream shows up to +8.5 math and +19.5 code benchmark points versus plain text carry, and RL fine‑tuning further improves long‑horizon tasks.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. tokenizers v1: encode, decode and scaling, measured

    Hugging Face has released `tokenizers` v1, a major performance update that achieves 3-30x faster encoding than v0.23 while maintaining identical output and API compatibility. Key optimizations include a SIMD-accelerated splitter, a thread-local word cache, and an allocation-free BPE merge loop, ensuring tokenization doesn't bottleneck ML workflows.

    Hugging Facehuggingface.co10 min

  4. Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data

    The paper presents Infinite-Parameter LLMs, where a compact hypernetwork creates feed‑forward weights from live user data and updates a Bayesian latent code online, keeping the stored model size constant while effectively having infinite parameters. This design aims to improve over standard in‑context learning and retrieval by persisting knowledge in weights and freeing context space.

    Hacker News front pagearxiv.org2 minpaperHN15743

  6. onPanda: Efficient Annotation of On-Policy Alignment Data for LLMs and Agents via Token-Level Correction

    onPanda is an interactive annotation tool that lets humans correct LLM outputs token‑by‑token, then resumes generation from the corrected prefix. In a controlled study it cut median annotation time by 52% and the authors release a token‑level correction dataset (Panda‑CVL) for on‑policy fine‑tuning.

    Hugging Face Daily Papersarxiv.org1 minpaper