proomt

Search

Search posts, papers, and topics

All posts

Hacker News front pageEvangelos Georganas, Alexander Heinecke, Pradeep Dubey1 min readpaperadvanced

Breaking the 1.58-bit Barrier for Ternary LLMs

Read on arxiv.orgHN 242·41 comments

Summary

BITCOS is a distribution‑aware storage layout for ternary LLM weights that replaces the standard five‑trit packing. By storing a presence bitmap and a compact sign vector, it reduces the effective bits‑per‑weight to 2 − z (z = zero density), achieving as low as 1.485 b/w on sparse models. The authors provide AVX‑512, AVX2, and Xe2 GPU unpacking kernels and show up to 1.28× speedup in matrix‑vecto…

  • Ternary LLMs have a high zero proportion (up to 51.5 %); treating symbols as equiprobable wastes space.
  • BITCOS stores a dense bitmap of non‑zero positions plus a packed sign vector, costing 2 − z bits per weight.
  • In 26/29 evaluated models BITCOS beats the conventional five‑trit packing, with a best‑case 1.485 b/w.
  • Optimized unpacking routines for AVX‑512, AVX2, and Intel Xe2 GPUs enable low‑overhead conversion to dense format.

Reducing storage and memory bandwidth for ternary LLMs directly translates to higher inference throughput on commodity hardware, making ultra‑compact models more practical for edge and low‑cost server deployments.

8/10
llmmachine learningstorageperformanceml infragpu

Related reading

  1. SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktops

    SiliconBench benchmarks nine Apple‑Silicon LLM serving engines on unified‑memory desktops, measuring speed, memory usage, and output fidelity across Qwen3, Qwen3.5, and Gemma‑4 models. It finds vllm‑metal doubles throughput at modest concurrency, memory budgets often fail to preserve headroom, and only three stacks satisfy all fidelity and model‑coverage requirements, with tensor‑parallel scaling…

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint

    Ternary Bonsai 2 27B is a new multimodal LLM based on Qwen3.8 27B, utilizing ternary weights for a 5.9GB model footprint. It achieves over 9x compression while retaining 98.2% of the full-precision model's aggregate benchmark performance across various tasks. This enables highly capable 27B-class models to run efficiently on local devices.

    Hacker News front pageprismml.com5 minreleaseHN579198lobste.rs12

  3. Vectorized and performance-portable Quicksort (2022)

    Google’s Highway library now includes a fully portable SIMD‑vectorized quicksort that runs 9‑19× faster than C++ std::sort. By using compress‑store (or permute‑based emulation) for partitioning, the same C++ code targets AVX2, AVX‑512, NEON, SVE and RISC‑V V. Benchmarks show 0.5 GB/s on an Apple M1 and >1 GB/s on a 3 GHz Skylake, beating prior architecture‑specific sorts. The implementation and a…

    Hacker News front pagegoogleblog.com3 minHN460142

  4. Shapelearn Qwen 3.8 27B (13.1 GB VRAM)

    ByteShape releases full ShapeLearn quantizations for Qwen 3.8 27B, showing that their GPU‑specific GGUFs (GPU‑1…GPU‑5) dominate the quality‑throughput frontier across six GPUs, with GPU‑5 hitting 99.63 % of BF16 accuracy at 90 TPS on a 13.1 GB model. Speculative decoding (MTP, DFlash2) further boosts throughput, and the Lite set remains competitive.

    Hacker News front pagebyteshape.com17 minHN10439

  5. DeepSeek-V4.1-Flash: Pushing the Limits of KV Cache Compression

    DeepSeek‑V4.1‑Flash is a 552B‑parameter multimodal Mixture‑of‑Experts LLM that supports up to 1 M‑token contexts while slashing KV‑cache memory to 890 bytes/token (≈¼ of its predecessor) via cross‑layer reuse (CSA2) and FP4 quantisation, plus a SWA‑Bounded Replay scheme that cuts persistent cache to 1/8. The Causal Encoder‑Decoder design halves prefill compute (8B vs 16B active parameters) and th…

    Hugging Face Daily Papersarxiv.org3 minpaperHN12710

  6. How Lossless Is Lossless Speculative Decoding? The Role of Numerical Precision in Orthrus

    This paper investigates the "lossless" claim of Orthrus, a hybrid architecture for accelerating LLM inference. It finds that under BF16 precision, Orthrus diverges from the exact autoregressive output trajectory in over 50% of cases, though FP32 maintains exact matching. Despite BF16 divergence, downstream task performance was not systematically degraded.

    Hugging Face Daily Papersarxiv.org1 minpaper