proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersIlya Koziev, Leonid Sinev, Ivan Oseledets1 min readpaperadvanced

How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus

Read on arxiv.org

Summary

This paper investigates the "lossless" claim of Orthrus, a hybrid architecture for accelerating LLM inference. It finds that under BF16 precision, Orthrus diverges from the exact autoregressive output trajectory in over 50% of cases, though FP32 maintains exact matching. Despite BF16 divergence, downstream task performance was not systematically degraded.

  • Orthrus's "lossless" speculative decoding is highly sensitive to numerical precision.
  • BF16 inference causes Orthrus to produce different token sequences than the reference model in ~55% of cases.
  • FP32 inference ensures exact trajectory matching for Orthrus across all evaluated prompts.
  • Trajectory divergence with BF16 did not lead to systematic degradation on standard lm-eval-harness benchmarks.

Engineers deploying LLMs with speculative decoding or reduced precision should understand that "lossless" claims can be precision-dependent and may not guarantee exact output sequences, even if downstream metrics are unaffected.

7/10
llmmachine learningperformancellm inferencedeep learningnumerical precision

Related reading

  1. Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches

    Fathom introduces a per-query read depth mechanism for sparse decoding over offloaded KV caches, allowing each query to adaptively decide how many bits of each key channel to read. This method significantly speeds up decoding for large language models with long contexts by reducing host memory traffic, achieving 1.67x faster GPU decoding on Qwen3-8B at one million tokens.

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. A Zeroth-Order Paradigm for LLM Preference Alignment

    The paper proposes Comparison-based Preference Optimization (ComPO), a zeroth‑order method that uses comparison oracles to align LLMs without a differentiable loss. Experiments on several LLM families show it improves win rates and mitigates likelihood displacement compared to direct alignment approaches.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. A theoretical separation between quantum computers & LLMs

    The IBM research blog explains two new theoretical results that prove shallow constant‑depth quantum circuits can outperform decoder‑only transformers on a functional task (iterated index) and diffusion language models on a sampling task (parity‑sampling). The proofs give asymptotic separations but are not yet practical.

    IBM Researchibm.com6 min

  5. SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktops

    SiliconBench benchmarks nine Apple‑Silicon LLM serving engines on unified‑memory desktops, measuring speed, memory usage, and output fidelity across Qwen3, Qwen3.5, and Gemma‑4 models. It finds vllm‑metal doubles throughput at modest concurrency, memory budgets often fail to preserve headroom, and only three stacks satisfy all fidelity and model‑coverage requirements, with tensor‑parallel scaling…

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. Decoy Direction Optimization: A Post-Hoc Defense Against LLM Abliteration

    Decoy Direction Optimization (DDO) is a post‑hoc weight‑editing defense for open‑weight LLMs that injects a high‑magnitude nonlinear decoy into MLP neurons, corrupting contrastive estimators used by Refusal Feature Ablation (RFA) attacks. The paper proves a spectral bound on the effect, evaluates DDO on six model families (including Llama‑3‑8B‑Instruct), and shows <10 % attack success rate (ASR)…

    Hugging Face Daily Papersarxiv.org1 minpaper