Hacker News front pageSep 26, 20267 min readintermediate

Generate fonts where every LLM token is the same width

Summary

This article introduces "token-space fonts" which render each LLM token with equal width, allowing users to visualize how text is tokenized. It provides a web-based compiler to generate these fonts for various LLM tokenizers and instructions for integrating them into Discord and Slack.

Token-space fonts visualize LLM tokenization by rendering each token at a fixed width (e.g., 3 em).
A web-based compiler allows generating custom token fonts for various LLM tokenizers (DeepSeek, Llama 3, Claude ctok, etc.).
Detailed instructions are provided for integrating these fonts into Discord (via Vesktop/Vencord) and Slack (via userscript).
The tool is transparent about known discrepancies and limitations, including whitespace handling, normalization, and browser rendering issues.

Engineers working with LLMs can use these fonts to visually debug and understand how different tokenizers segment text, which is crucial for prompt engineering and model behavior analysis.

7/10

llm frontend visualization developer tools typography tokenization

Related reading

Font where each token is equal-width twitter.com
Hacker News front pagetwitter.com9dHN2614
A font that reads what you wrote github.io
Hacker News front pagegithub.io15dHN9544
TokenCast: Forecasting Token Consumption During LLM Agent Execution arxiv.org
LLM agent token consumption varies widely and is hard to predict due to dynamic execution and growing context. TokenCast proposes a composable cost representation for execution segments to forecast token usage, reducing prediction error and enabling better budget control.
Hugging Face Daily Papersarxiv.orgChaoqian Ouyang, Ling Yue, Libin Zheng8d1 minpaper
tokenizers v1: encode, decode and scaling, measured huggingface.co
Hugging Face has released `tokenizers` v1, a major performance update that achieves 3-30x faster encoding than v0.23 while maintaining identical output and API compatibility. Key optimizations include a SIMD-accelerated splitter, a thread-local word cache, and an allocation-free BPE merge loop, ensuring tokenization doesn't bottleneck ML workflows.
Hugging Facehuggingface.coArthur Zucker, Simon Brandeis, Luc Georges, Lysandre15d10 min
Beyond Selection: Token Parameterization for Extreme Visual Token Compression arxiv.org
This paper introduces Braco, a lightweight coder for extreme visual token compression in vision-language models. It uses a novel token parameterization approach to maintain visual grounding and achieve high accuracy at 23x-64x compression, significantly improving efficiency over prior methods.
Hugging Face Daily Papersarxiv.orgRui Zhong, Yu Li, Zheyu Yan8d1 minpaper
Register Tokens for Bounded-State Reasoning in Diffusion Language Models arxiv.org
Register tokens are fixed‑position embeddings that store a compact hidden state across diffusion‑based language model generation chunks, enabling bounded‑state reasoning without retaining all prior text. Post‑training on LLaDA and Dream shows up to +8.5 math and +19.5 code benchmark points versus plain text carry, and RL fine‑tuning further improves long‑horizon tasks.
Hugging Face Daily Papersarxiv.orgAlbert Ge, Chandan Singh, Yufan Zhuang22d1 minpaper

Generate fonts where every LLM token is the same width

Summary

Related reading

Font where each token is equal-widthtwitter.com

A font that reads what you wrotegithub.io

TokenCast: Forecasting Token Consumption During LLM Agent Executionarxiv.org

tokenizers v1: encode, decode and scaling, measuredhuggingface.co

Beyond Selection: Token Parameterization for Extreme Visual Token Compressionarxiv.org

Register Tokens for Bounded-State Reasoning in Diffusion Language Modelsarxiv.org

Font where each token is equal-width twitter.com

A font that reads what you wrote github.io

TokenCast: Forecasting Token Consumption During LLM Agent Execution arxiv.org

tokenizers v1: encode, decode and scaling, measured huggingface.co

Beyond Selection: Token Parameterization for Extreme Visual Token Compression arxiv.org

Register Tokens for Bounded-State Reasoning in Diffusion Language Models arxiv.org