Hacker News front pageawlevin11 min readintermediate
Typesafe-computer-use drives a Mac toward a goal for 1/50th of a cent per step
Summary
A Python tool for macOS that replaces expensive LLM‑driven UI automation with a cheap, deterministic classifier (TypeSafe) for per‑step decisions, cutting cost by ~150‑300× and latency by ~10‑40× while still handling text entry via a small writer model.
- Uses screen capture + Vision OCR + macOS accessibility tree to build a concise list of actionable items per step.
- A TypeSafe decision model (up to 255 choices) selects the next action with a confidence score, avoiding full‑frame LLM planning for most steps.
- Smart OCR reuse: compares scaled‑down screenshots, re‑reads only changed tiles, reducing Vision token usage.
- Off‑screen controls are discovered via the accessibility tree and offered as separate actions, enabling clicks on hidden UI elements.
Frontier‑model UI agents are prohibitively expensive for high‑frequency automation. By delegating deterministic UI parsing to OCR/AX and only invoking a tiny classifier for action selection, the system achieves near‑real‑time interaction at a fraction of the cost, making large‑scale desktop automat…
7/10
