proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersConnor Makowski, Willem Guter1 min readpaperintermediate

The Functionalizer: Lossless Functional Decomposition for Subword Tokenization

Read on arxiv.org

Summary

The Functionalizer is a lossless pre‑tokenizer that factors casing, diacritics and character repetitions into reversible opcode prefixes before subword tokenization. Experiments show up to 19.7% smaller vocabularies and a 9.12% syntax‑validity gain on GPT‑2 code models versus standard tokenizers.

  • Functionalizer encodes orthographic variations as Unicode Private Use Area opcodes (CAPITALIZE, diacritics, REPEAT, etc.) preceding a canonical base token.
  • Vocabulary size drops up to 19.7% while maintaining full corpus coverage, eliminating OOVs under unconstrained exhaustion.
  • GPT‑2 models (98 M parameters) trained on Python code see syntax validity rise from 7.70% to 9.12% with Functionalizer.
  • The opcode/operand stream is fully reversible, allowing exact reconstruction of the original text.

NLP and code‑model engineers should care because it offers vocab‑efficient, structure‑aware tokenization without sacrificing recoverability.

7/10
llmmachine learningpythonnlptokenization

Related reading

  1. tokenizers v1: encode, decode and scaling, measured

    Hugging Face has released `tokenizers` v1, a major performance update that achieves 3-30x faster encoding than v0.23 while maintaining identical output and API compatibility. Key optimizations include a SIMD-accelerated splitter, a thread-local word cache, and an allocation-free BPE merge loop, ensuring tokenization doesn't bottleneck ML workflows.

    Hugging Facehuggingface.co10 min

  2. Register Tokens for Bounded-State Reasoning in Diffusion Language Models

    Register tokens are fixed‑position embeddings that store a compact hidden state across diffusion‑based language model generation chunks, enabling bounded‑state reasoning without retaining all prior text. Post‑training on LLaDA and Dream shows up to +8.5 math and +19.5 code benchmark points versus plain text carry, and RL fine‑tuning further improves long‑horizon tasks.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. Reverse-engineered Jev-like model

    Jevlike is an open‑source starter model that scores a list of text options in a single forward pass. It provides a minimal architecture (option queries, shared dot‑product scorer), synthetic data generation, training/evaluation CLI, and examples on Doom and chess. The repo supports a byte‑level encoder or a frozen Hugging‑Face encoder (e.g., Qwen2.5‑0.5B), runs on CPU/MPS/CUDA, and reports benchm…

    Hacker News front pagegithub.com4 minreleaseHN16224

  5. Dynamically Scaled Activation Steering

    Dynamically Scaled Activation Steering (DSAS) is a method‑agnostic framework that learns per‑token, per‑layer scaling factors to turn existing activation‑steering interventions on only when a model is likely to produce undesired output (e.g., toxic text). The scaling can be optimized jointly with any steering function, improves the toxicity‑utility trade‑off on language models, transfers to text‑…

    Apple Machine Learning Researchapple.com1 minpaper

  6. App Hardening: One Obfuscation Pipeline Across Every Port

    Codename One adds a cloud‑side hardening step that runs on the merged JAR before it is split into Android, iOS, JavaScript, and desktop binaries. It can rename symbols, encrypt string literals, and insert opaque‑predicate control‑flow guards at configurable levels (off → standard → aggressive → paranoid). The transforms are selective per platform to avoid breaking optimizers, and a mapping is kep…

    CodeName Onecodenameone.com6 min