proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersQuan Nguyen-Tri, Mukul Ranjan, Zhiqiang Shen1 min readpaperadvanced

Flash-dLLM: IO-Aware KV Caching and Parallel Decoding for Fast, Memory-Efficient Diffusion LLMs

Read on arxiv.org

Summary

Flash-dLLM is a training-free framework that accelerates Diffusion LLM inference by addressing GPU memory I/O bottlenecks with an I/O-aware KV-cache kernel. It also introduces a KV-cache-driven draft-and-verify decoding strategy, achieving significant speedups (up to 11x) over prior methods.

  • Flash-dLLM optimizes Diffusion LLM inference by jointly addressing KV caching and parallel decoding inefficiencies.
  • It identifies GPU memory I/O as a dominant bottleneck and mitigates it with an I/O-aware fused KV-cache kernel.
  • A novel draft-and-verify decoding strategy uses the dLLM itself as both drafter and verifier, eliminating auxiliary models.
  • The framework improves scalability for longer sequences and larger batch sizes while preserving generation quality.

Engineers deploying or optimizing Diffusion LLMs will find this critical for significantly improving inference speed and memory efficiency in non-autoregressive text generation.

8/10
llmmachine learningperformancegpullm inferencecaching

Related reading

  1. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression

    DeepSeek‑V4.1‑Flash is a 552B‑parameter multimodal Mixture‑of‑Experts LLM that supports up to 1 M‑token contexts while slashing KV‑cache memory to 890 bytes/token (≈¼ of its predecessor) via cross‑layer reuse (CSA2) and FP4 quantisation, plus a SWA‑Bounded Replay scheme that cuts persistent cache to 1/8. The Causal Encoder‑Decoder design halves prefill compute (8B vs 16B active parameters) and th…

    Hugging Face Daily Papersarxiv.org3 minpaperHN12710

  2. Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches

    Fathom introduces a per-query read depth mechanism for sparse decoding over offloaded KV caches, allowing each query to adaptively decide how many bits of each key channel to read. This method significantly speeds up decoding for large language models with long contexts by reducing host memory traffic, achieving 1.67x faster GPU decoding on Qwen3-8B at one million tokens.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus

    This paper investigates the "lossless" claim of Orthrus, a hybrid architecture for accelerating LLM inference. It finds that under BF16 precision, Orthrus diverges from the exact autoregressive output trajectory in over 50% of cases, though FP32 maintains exact matching. Despite BF16 divergence, downstream task performance was not systematically degraded.

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktops

    SiliconBench benchmarks nine Apple‑Silicon LLM serving engines on unified‑memory desktops, measuring speed, memory usage, and output fidelity across Qwen3, Qwen3.5, and Gemma‑4 models. It finds vllm‑metal doubles throughput at modest concurrency, memory budgets often fail to preserve headroom, and only three stacks satisfy all fidelity and model‑coverage requirements, with tensor‑parallel scaling…

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. Breaking the 1.58-bit Barrier for Ternary LLMs

    BITCOS is a distribution‑aware storage layout for ternary LLM weights that replaces the standard five‑trit packing. By storing a presence bitmap and a compact sign vector, it reduces the effective bits‑per‑weight to 2 − z (z = zero density), achieving as low as 1.485 b/w on sparse models. The authors provide AVX‑512, AVX2, and Xe2 GPU unpacking kernels and show up to 1.28× speedup in matrix‑vecto…

    Hacker News front pagearxiv.org1 minpaperHN24241

  6. GLM 5.3 FlashX now available on AI Gateway

    Vercel AI Gateway now offers the GLM‑5.3‑FlashX model, a fast (~200 tps) multimodal coding LLM. The post includes a TypeScript streaming example, notes the model’s fit for coding agents and interactive tools, and lists AI Gateway features (unified API, usage tracking, retries/failover, custom reporting, key budgets, routing rules) with no platform fee.

    Vercelvercel.com1 minrelease