tokenization

The Functionalizer: Lossless Functional Decomposition for Subword Tokenizationarxiv.org

The Functionalizer is a lossless pre‑tokenizer that factors casing, diacritics and character repetitions into reversible opcode prefixes before subword tokenization. Experiments show up to 19.7% smaller vocabularies and a 9.12% syntax‑validity gain on GPT‑2 code models versus standard tokenizers.