proomt

Search

Search posts, papers, and topics

Hall of Fame

Hall of FameAndrej Karpathy201534 min readintermediate

The Unreasonable Effectiveness of Recurrent Neural Networks

Read on github.ioHN 65·6 comments

Summary

Andrej Karpathy demonstrates the "unreasonable effectiveness" of Recurrent Neural Networks (RNNs), particularly LSTMs, for sequence modeling. He shows how character-level language models can learn to generate text by predicting the next character, even releasing code for it.

  • RNNs handle variable-length sequences, unlike fixed-size vanilla NNs, making them suitable for tasks like image captioning or machine translation.
  • The core RNN computation involves updating a hidden state based on the current input and previous state, effectively acting as a program.
  • Even fixed-size data can be processed sequentially by RNNs, allowing models to learn attention mechanisms or generate images step-by-step.
  • LSTMs are a more robust and commonly used variant of RNNs, offering improved performance in practice.

This foundational post introduced many to the practical power of RNNs and LSTMs for sequence generation, inspiring a wave of research and applications in NLP and beyond.

9/10
machine learningnlpdeep learningrnnlstmsequence models

Related reading

  1. A Recipe for Training Neural Networks

    Karpathy outlines a disciplined recipe for training neural nets that treats the process as a debugging problem. He stresses data inspection, minimal baselines, fixed seeds, and extensive visual checks to catch silent failures early.

    Hall of Famegithub.io17 minHN41332

  2. Attention Is All You Need

    The paper proposes the Transformer, a sequence‑to‑sequence model that relies solely on self‑attention, eliminating recurrence and convolutions. It achieves state‑of‑the‑art translation BLEU scores while training orders of magnitude faster.

    Hall of Famearxiv.org27 minpaper

  3. Register Tokens for Bounded-State Reasoning in Diffusion Language Models

    Register tokens are fixed‑position embeddings that store a compact hidden state across diffusion‑based language model generation chunks, enabling bounded‑state reasoning without retaining all prior text. Post‑training on LLaDA and Dream shows up to +8.5 math and +19.5 code benchmark points versus plain text carry, and RL fine‑tuning further improves long‑horizon tasks.

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. Training Compute-Optimal Large Language Models

    The paper derives a compute‑optimal scaling law showing model size and training tokens should grow together, and validates it by training a 70B‑parameter model (Chinchilla) on 1.4 T tokens that outperforms much larger LLMs.

    Hall of Famearxiv.org66 minpaper

  6. Learning to solve hard problems in RL for LLMs by never giving up

    The post introduces the *Matthew Effect* in RL‑fine‑tuning of LLMs—performance gains concentrate on tasks the model already solves— and proposes *Never Give Up* (NGU), an adaptive sampling scheme that uses a small k for easy prompts and retries hard prompts with a high‑probability “never give up” loop. Experiments on math (AIME, GSM8k), code (Manufactoria), and larger‑scale setups (DeepScaler) sh…

    Hacker News front pagegithub.io11 minHN1179