Related reading
WeVisDoc: From Coverage to Capability for Robust End-to-End Document Parsing
WeVisDoc introduces a two‑stage data‑centric pipeline for end‑to‑end document parsing. Stage I expands coverage using heterogeneous data and structure‑preserving degradations. Stage II probes the Stage I model with a held‑out set, clusters residual errors, and directs targeted data creation and token‑budget reallocation. The 4‑billion‑parameter model reaches 95.38 Overall on OmniDocBench v1.6 and…
Hugging Face Daily Papersarxiv.org1 minpaperHow LLMs Can Find a Needle in a Haystack
The post explains how retrieval‑augmented generation (RAG) lets LLM‑based assistants answer questions from private corpora. It covers chunking documents into passages, embedding queries and chunks, similarity metrics, and the trade‑offs of different vector indexes (flat, IVF, HNSW). The focus is on practical design choices rather than new research.
ByteByteGobytebytego.com12 minFrom Memory-Hungry HNSW to Quantized SPANN: The Technical Evolution of Pinterest's Manas Platform
Pinterest reengineered its Manas search platform to replace memory‑heavy HNSW with scalar and product quantization, cutting index size by up to 93% while keeping recall above 70% and achieving 20‑30% cost savings. They also moved ANN storage to SSD using SPANN, gaining 3× query throughput over DiskANN with modest latency and recall impact.
InfoQinfoq.com2 minIntroducing TIN: full-text search for Postgres
PlanetScale’s TIN is a new PostgreSQL extension that implements a full‑text search index supporting Boolean, phrase, span, fuzzy, wildcard, regex, case/accent folding, COUNT(*) and BM25 top‑k queries. In a suite of benchmarks on an i7i.8xlarge EC2 instance (8 vCPU, 32 GB RAM), TIN built in 8 min 10 s (50.7 GB index) and outperformed ParadeDB, pg_textsearch and the built‑in GIN index by large marg…
PlanetScaleplanetscale.com15 minHN20175Self-Evolving Search Index
The paper introduces SELF-INDEX, a framework that lets a search index automatically diagnose retrieval failures, revise its keys, and validate changes, using a query simulator to anticipate future queries. Experiments show consistent gains across corpora and downstream LLM agents.
Hugging Face Daily Papersarxiv.org1 minpaper
