proomt

Search

Search posts, papers, and topics

All posts

Hacker News front pagePrismML5 min readrelease notesintermediate

Bonsai 2 27B: Near-Lossless Compression in a 9x Smaller Footprint

Read on prismml.comHN 579·198 commentsLobsters 12

Summary

Ternary Bonsai 2 27B is a new multimodal LLM based on Qwen3.8 27B, utilizing ternary weights for a 5.9GB model footprint. It achieves over 9x compression while retaining 98.2% of the full-precision model's aggregate benchmark performance across various tasks. This enables highly capable 27B-class models to run efficiently on local devices.

  • Uses ternary {-1, 0, +1} weights with FP16 group-wise scaling for 1.76 effective bits per weight.
  • Achieves a 5.9GB model footprint, over 9x smaller than its full-precision counterpart.
  • Retains 98.2% of Qwen3.8 27B's aggregate benchmark performance across reasoning, coding, vision, and agentic tasks.
  • Supports a 262K-token context window and multimodal text-and-image input.

Engineers building AI products with tight memory, latency, or power requirements should care, as this model enables deploying highly capable 27B-class LLMs on local devices and edge hardware with near-lossless performance.

7/10
llmmultimodalperformanceml infraquantizationedge ai

Related reading

  1. Shapelearn Qwen 3.8 27B (13.1 GB VRAM)

    ByteShape releases full ShapeLearn quantizations for Qwen 3.8 27B, showing that their GPU‑specific GGUFs (GPU‑1…GPU‑5) dominate the quality‑throughput frontier across six GPUs, with GPU‑5 hitting 99.63 % of BF16 accuracy at 90 TPS on a 13.1 GB model. Speculative decoding (MTP, DFlash2) further boosts throughput, and the Lite set remains competitive.

    Hacker News front pagebyteshape.com17 minHN10439

  3. SiliconBench: Speed, Memory, and Fidelity for LLM Serving on Unified-Memory Desktops

    SiliconBench benchmarks nine Apple‑Silicon LLM serving engines on unified‑memory desktops, measuring speed, memory usage, and output fidelity across Qwen3, Qwen3.5, and Gemma‑4 models. It finds vllm‑metal doubles throughput at modest concurrency, memory budgets often fail to preserve headroom, and only three stacks satisfy all fidelity and model‑coverage requirements, with tensor‑parallel scaling…

    Hugging Face Daily Papersarxiv.org1 minpaper

  4. Breaking the 1.58-bit Barrier for Ternary LLMs

    BITCOS is a distribution‑aware storage layout for ternary LLM weights that replaces the standard five‑trit packing. By storing a presence bitmap and a compact sign vector, it reduces the effective bits‑per‑weight to 2 − z (z = zero density), achieving as low as 1.485 b/w on sparse models. The authors provide AVX‑512, AVX2, and Xe2 GPU unpacking kernels and show up to 1.28× speedup in matrix‑vecto…

    Hacker News front pagearxiv.org1 minpaperHN24241

  5. Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches

    Fathom introduces a per-query read depth mechanism for sparse decoding over offloaded KV caches, allowing each query to adaptively decide how many bits of each key channel to read. This method significantly speeds up decoding for large language models with long contexts by reducing host memory traffic, achieving 1.67x faster GPU decoding on Qwen3-8B at one million tokens.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. Anatomy of a Texture

    A practical deep‑dive into modern texture memory layout: block‑compression (BC7), swizzle ordering (Morton/Z‑order), mip‑map hierarchies, and concrete debugging tricks for cross‑platform texture conversion.

    Hacker News front pagegithub.io12 minHN10816