proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersZian Liu, Yiwen Hu, Zican Dong1 min readpaperadvanced

SpectralShift: Effective Context Window Extension of Gated DeltaNet via Spectral Reparameterization

Read on arxiv.org

Summary

This paper introduces SpectralShift, a spectral reparameterization method for extending the context window of Gated DeltaNet (GDN) linear attention models. It reconfigures the decay spectrum by enhancing slow propagation and preserving fast-decaying modes, consistently improving long-context capabilities during continual pretraining.

  • Linear attention layers are increasingly used for long-context modeling in large language models.
  • Effective context extension for Gated DeltaNet requires considering the spectral properties of its transition matrix.
  • Long-range information retrieval depends on a broad slow spectral band and preserved fast-decaying modes.
  • SpectralShift reparameterizes alpha projections to reshape the decay spectrum for better long-context capacity.

Engineers and researchers working on large language models, particularly those using linear attention architectures, should care about this for improving long-context performance and efficiency.

8/10
llmmachine learningarchitecturellm inferencedeep learningattention

Related reading

  1. Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation

    Video DeltaNet (VDN) replaces full‑softmax attention in video diffusion models with a hybrid: per‑frame local Softmax for fine detail and a bidirectional linear memory (Video Delta Attention) for long‑range context. A teacher‑alignment schedule injects the linear branch into a pretrained MiniMax H3 model, preserving Softmax for text/audio streams. On eight NVIDIA B200 GPUs VDN‑H3 denoises a 14.3‑…

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention

    VC-Attention introduces a training‑free low‑bit attention pipeline for diffusion transformers. It smooths value tensors via lightweight online clustering (V‑Smooth) and quantizes only the residual after subtracting block means, restoring the mean from the softmax row sum. It also replaces the FP32 softmax exponential with a fused FP8 cast (ExpCast‑FP8) that maps log‑scores directly to E4M3 probab…

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. LimiX-2: A Contextual Mechanism Network Towards General Structured-Data Intelligence

    LimiX-2 is a new tabular foundation model that replaces the usual target‑centric in‑context learning objective with a joint‑distribution objective via Contextual Mechanism Networks (CMNs). Trained on synthetic causal graphs using Context‑Conditional Masked Modeling, it outperforms prior tabular PFNs on TabArena, TALENT, and BCCO and can recover causal skeletons from attention patterns.

    Hugging Face Daily Papersarxiv.org2 minpaper

  4. Infinite-Parameter LLMs: Generating and Adapting Weights from Live Data

    The paper presents Infinite-Parameter LLMs, where a compact hypernetwork creates feed‑forward weights from live user data and updates a Bayesian latent code online, keeping the stored model size constant while effectively having infinite parameters. This design aims to improve over standard in‑context learning and retrieval by persisting knowledge in weights and freeing context space.

    Hacker News front pagearxiv.org2 minpaperHN15743

  5. Dynamically Scaled Activation Steering

    Dynamically Scaled Activation Steering (DSAS) is a method‑agnostic framework that learns per‑token, per‑layer scaling factors to turn existing activation‑steering interventions on only when a model is likely to produce undesired output (e.g., toxic text). The scaling can be optimized jointly with any steering function, improves the toxicity‑utility trade‑off on language models, transfers to text‑…

    Apple Machine Learning Researchapple.com1 minpaper