Hugging Face Daily PapersIvan Kobyzev, Abbas Ghaddar, Ali Nasiri-SarviSep 28, 20261 min readpaperadvanced

Fractional State Space Transition for Long Sequence Modeling

Summary

The paper introduces FRAC, a state‑space model that uses fractional dynamics to achieve power‑law long memory instead of exponential forgetting. Approximating the fractional kernel with a log‑spaced exponential sum yields an efficient recurrent module that improves long‑context performance in large language models.

FRAC replaces exponential decay in SSMs with power‑law memory using fractional dynamics.
Implements the fractional kernel as a log‑spaced sum of exponentials for efficient recurrent computation.
Supports parallel training and prefill while keeping a bounded autoregressive state.
1.3B‑parameter language model experiments show consistent gains on long‑context tasks and parity on short contexts.

ML engineers and researchers building long‑context LLMs should care because FRAC offers a practical way to extend memory without sacrificing efficiency.

6/10

llm machine learning performance ml infra state space models

Related reading

Continual Learning Mechanisms Compose for Long-Horizon Memorization arxiv.org
The paper studies long‑horizon memorization where a language model must learn 100 tasks via continual fine‑tuning without retaining data or task IDs. By composing data, function, and weight anchors with merged LoRA, they boost final retention from 1.2% to 34.9%, a 28× improvement.
Hugging Face Daily Papersarxiv.orgZheyuan Zhang, Alvin Zhang, Daniel Khashabi29d1 minpaperHN2
The Unreasonable Effectiveness of Recurrent Neural Networks github.io
Andrej Karpathy demonstrates the "unreasonable effectiveness" of Recurrent Neural Networks (RNNs), particularly LSTMs, for sequence modeling. He shows how character-level language models can learn to generate text by predicting the next character, even releasing code for it.
Hall of Famegithub.ioAndrej Karpathy12y34 minHN656
The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends arxiv.org
The paper surveys recent attention variants in large language models, introducing a five‑dimensional framework (Memory Representation, Update, Access, Readout, Integration) to compare them. It shows that modern LLMs increasingly treat contextual memory as a coordinated, multi‑layer resource rather than a single attention operator.
Hugging Face Daily Papersarxiv.orgZhentao Tan, Jingyi Shen, Yanbo Li6d1 minpaper
Scaling Laws for Neural Language Models arxiv.org
This paper empirically studies scaling laws for neural language model performance, finding that cross-entropy loss scales as a power-law with model size, dataset size, and compute. It shows that optimal compute-efficient training involves using very large models, training on relatively modest data, and stopping significantly before convergence.
Hall of Famearxiv.orgJared Kaplan et al.7y67 minpaper
FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models arxiv.org
FactorEngram proposes a factorized n-gram memory with basis-level contextual gating for LLMs. This design allows polysemous patterns to selectively retrieve relevant memory components from a shared dictionary, leading to improved language modeling and downstream task performance.
Hugging Face Daily Papersarxiv.orgBowen Yang, Jingbo Zhou, Qinghong Miao8d1 minpaper
Attention Is All You Need arxiv.org
The paper proposes the Transformer, a sequence‑to‑sequence model that relies solely on self‑attention, eliminating recurrence and convolutions. It achieves state‑of‑the‑art translation BLEU scores while training orders of magnitude faster.
Hall of Famearxiv.orgAshish Vaswani et al.10y27 minpaper

Fractional State Space Transition for Long Sequence Modeling

Summary

Related reading

Continual Learning Mechanisms Compose for Long-Horizon Memorizationarxiv.org

The Unreasonable Effectiveness of Recurrent Neural Networksgithub.io

The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trendsarxiv.org

Scaling Laws for Neural Language Modelsarxiv.org

FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Modelsarxiv.org

Attention Is All You Needarxiv.org

Continual Learning Mechanisms Compose for Long-Horizon Memorization arxiv.org

The Unreasonable Effectiveness of Recurrent Neural Networks github.io

The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends arxiv.org

Scaling Laws for Neural Language Models arxiv.org

FactorEngram: Factorized N-gram Memory with Basis-Level Gating for Language Models arxiv.org

Attention Is All You Need arxiv.org