Hacker News front page7 min readrelease notesintermediate
Show HN: Cactus Needle 3: 8-29MB automation models can match DeepSeek V4 Flash
Summary
Cactus Needle 3 is a laddered attention LLM (29‑121 M params, CQ2‑bit quantised) that fits in 9‑29 MB binaries and runs on devices like a Raspberry Pi 5 (400‑4k t/s decode, 1‑10k t/s pre‑fill). It can perform tool‑calling, structured extraction, and text embedding locally, matching DeepSeek V4 Flash on downstream tasks after a single fine‑tune epoch. The post includes architecture details, benchm…
- Laddered Simple Attention Networks use a depth‑configurable sub‑network (2‑20 layers) that can be fine‑tuned independently, enabling a trade‑off between model size (9‑29 MB) and capability.
- CQ2 quantisation reduces model size dramatically while keeping inference speed high on edge hardware (e.g., 850 t/s decode on a Pi 5).
- The model supports three unified primitives: tool‑calling (auto‑selecting functions and filling arguments), structured extraction (typed JSON via Pydantic), and embedding generation for local semantic search.
- Benchmarks show Needle 3 beating models 10× its size on tool‑call tasks and matching 2‑3× larger models on extraction, while using ~2× fewer MFLOPs per token than a comparable transformer.
Running LLM‑powered assistants entirely on‑device removes latency, privacy concerns, and dependence on cloud APIs—critical for smart‑home, robotics, wearables, and automotive use‑cases where connectivity is intermittent or data must stay local. Needle 3 demonstrates that sub‑30 MB models can still…
6/10
.png)