proomt

Search

Search posts, papers, and topics

All posts

Hugging Face Daily PapersGiacomo Fidone, Alessio Cascione, Riccardo Guidotti1 min readpaperadvanced

Learning Sparse Decision Trees via Transformer Variational Auto-Encoders

Read on arxiv.org

Summary

The paper presents TREVIS, a method that encodes decision trees into a continuous latent space using a Tree Transformer VAE, allowing gradient-based optimization of both accuracy and structural sparsity. Experiments claim TREVIS matches the predictive performance of near-optimal algorithms while producing sparser trees.

  • TREVIS maps discrete decision trees to latent vectors via a Tree Transformer VAE, converting the search space to continuous.
  • A differentiable surrogate model enables joint gradient optimization of predictive performance and sparsity.
  • Empirical results show TREVIS achieves comparable accuracy to state‑of‑the‑art methods with reduced tree size.

Practitioners building interpretable models will care because it offers a way to enforce sparsity without sacrificing accuracy.

5/10
machine learningtransformersdecision treesvariational autoencoderssparsity

Related reading

  1. VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention

    VC-Attention introduces a training‑free low‑bit attention pipeline for diffusion transformers. It smooths value tensors via lightweight online clustering (V‑Smooth) and quantizes only the residual after subtracting block means, restoring the mean from the softmax row sum. It also replaces the FP32 softmax exponential with a fused FP8 cast (ExpCast‑FP8) that maps log‑scores directly to E4M3 probab…

    Hugging Face Daily Papersarxiv.org1 minpaper

  2. Training-Adaptive Convolutional Sparse Coding via Information Bottleneck for Robust Visual Representation

    This paper introduces a training-adaptive Convolutional Sparse Coding (CSC) framework where the sparsity coefficient is learned end-to-end via FISTA unfolding. It uses an information bottleneck perspective to balance representation compression and content preservation, showing improved robustness to input perturbations on CIFAR and ImageNet.

    Hugging Face Daily Papersarxiv.org1 minpaper

  3. Laya the open source version of Jev

    Laya is an open‑source, bidirectional‑encoder model family for ultra‑fast, calibrated decision‑making (choice, score, boolean) over structured schemas. It runs 6‑8× faster than the closed‑source Jev, supports 100+ languages via three checkpoints, and includes a lightweight router that selects the appropriate checkpoint before inference. Benchmarks show higher accuracy, far better calibration (ECE…

    Hacker News front pageconvaiinnovations.com8 minreleaseHN1326313lobste.rs2

  5. MoME: Mixture-of-Memory Embeddings for Context-Aware Sparse Lookup

    The paper proposes Mixture of Memory Embeddings (MoME), a context‑aware sparse lookup that replaces each token’s single memory row with a gated mixture of multiple slots. Experiments on Llama‑3, MobileLLM and Qwen3 show MoME outperforms existing memory‑embedding baselines at equal parameter and FLOP budgets and exhibits interpretable routing for polysemous tokens.

    Hugging Face Daily Papersarxiv.org1 minpaper

  6. Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem

    The authors cast transformer block removal as a constrained binary optimization problem equivalent to an Ising glass, using a Hessian‑derived energy as a proxy for downstream quality. Solving the resulting QUBO with classical or quantum‑inspired solvers yields up to 23 MMLU points improvement over prior block‑removal baselines at 50 % depth compression.

    Hugging Facehuggingface.co8 min