proomt

Search

Search posts, papers, and topics

All posts

InfoQOlimpiu Pop2 min readintermediate

From Memory-Hungry HNSW to Quantized SPANN: The Technical Evolution of Pinterest's Manas Platform

Read on infoq.com

Summary

Pinterest reengineered its Manas search platform to replace memory‑heavy HNSW with scalar and product quantization, cutting index size by up to 93% while keeping recall above 70% and achieving 20‑30% cost savings. They also moved ANN storage to SSD using SPANN, gaining 3× query throughput over DiskANN with modest latency and recall impact.

  • Product Quantization shrinks HNSW from 121 GB to 32 GB (‑74%) with recall ~77% and similar QPS.
  • Scalar Quantization reduces HNSW to 50 GB (‑59%) while preserving >90% recall and query speed.
  • IVF + SQ yields the best recall (95.7%) on a 25 GB index with high QPS.
  • SPANN on SSD delivers 3× the QPS of DiskANN, 1/3 the latency, and only ~5% recall loss, saving ~40% CPU time.

Engineers building billion‑scale vector search systems need practical techniques to cut memory and cost while maintaining performance, and Pinterest's quantization and SSD‑backed SPANN provide proven solutions.

6/10
performancedistributed systemsquantizationvector searchannssd

Related reading

  1. tokenizers v1: encode, decode and scaling, measured

    Hugging Face has released `tokenizers` v1, a major performance update that achieves 3-30x faster encoding than v0.23 while maintaining identical output and API compatibility. Key optimizations include a SIMD-accelerated splitter, a thread-local word cache, and an allocation-free BPE merge loop, ensuring tokenization doesn't bottleneck ML workflows.

    Hugging Facehuggingface.co10 min
  3. 1 points

    Saving another 100TB of RAM with math (and Rust)

    Cloudflare reduced the memory footprint of its Pingora Backend Router by re‑examining the consistent‑hashing implementation in the pingora‑ketama library. By increasing the number of virtual hash points per server from the default 1 to the standard 160 (and applying weighted hashing based on disk capacity), they cut the per‑node overhead enough to reclaim >100 TB of RAM across the fleet. The post…

    Hacker News front pagecloudflare.com13 minHN478120lobste.rs33

  4. MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup

    The paper proposes Mixture of Memory Embeddings (MoME), a context‑aware sparse lookup that replaces each token’s single memory row with a gated mixture of multiple slots. Experiments on Llama‑3, MobileLLM and Qwen3 show MoME outperforms existing memory‑embedding baselines at equal parameter and FLOP budgets and exhibits interpretable routing for polysemous tokens.

    Hugging Face Daily Papersarxiv.org1 minpaper

  5. Introducing TIN: full-text search for Postgres

    PlanetScale’s TIN is a new PostgreSQL extension that implements a full‑text search index supporting Boolean, phrase, span, fuzzy, wildcard, regex, case/accent folding, COUNT(*) and BM25 top‑k queries. In a suite of benchmarks on an i7i.8xlarge EC2 instance (8 vCPU, 32 GB RAM), TIN built in 8 min 10 s (50.7 GB index) and outperformed ParadeDB, pg_textsearch and the built‑in GIN index by large marg…

    PlanetScaleplanetscale.com15 minHN20175

  6. Fathom: Per-Query Read Depth for Sparse Decoding over Offloaded KV Caches

    Fathom introduces a per-query read depth mechanism for sparse decoding over offloaded KV caches, allowing each query to adaptively decide how many bits of each key channel to read. This method significantly speeds up decoding for large language models with long contexts by reducing host memory traffic, achieving 1.67x faster GPU decoding on Qwen3-8B at one million tokens.

    Hugging Face Daily Papersarxiv.org1 minpaper