Almost all language models read text by splitting it into chunks called tokens, and almost no one questions it. A new research paper breaks with that habit: it trains small models that read raw bytes, that is, text character by character, with no tokenizer. According to the paper, these models start from weaker performance, but training curves suggest that with more compute they end up beating token-based models.
The work, posted on arXiv under the title "Breaking the Token Ceiling", uses distillation: a larger model, Llama 3 8B, teaches a smaller one of about 1 billion parameters. The technical problem is that the teacher reasons in tokens while the student reads bytes, so the team had to define a clean method to translate information between the two formats. One source says the work, under the name "byteification", was also published in Nature and covers open models such as Olmo 3 7B, OLMo 2 1B, Qwen3 8B Base and Llama 3 8B, retrofitted to operate on bytes.

Image: arXiv — https://arxiv.org/abs/2609.12303
The results follow a clear trajectory. With little training, token models win clearly. With more training, token models seem to flatline while byte models keep improving. Extending the curves, the paper estimates the byte model would end up beating Llama 3.2 1B and Gemma 3 1B by a few points. This is a projection, not a direct measurement, and should be read as such.
The point matters to model builders because the tokenizer has always been treated as an almost fixed part of the recipe. It has concrete effects: it determines how much vocabulary is needed, how languages and rare words are handled, and how costly it is to store probability distributions during distillation. The authors note that operating on a 256-byte vocabulary instead of around 100,000 tokens removes the need to truncate logits during distillation and cuts logit storage costs. The paper also reports better data efficiency: the byte model matches a distilled token model using only one-sixth of the training data.
Byte-level research is not new. Meta had already presented the Byte Latent Transformer, an architecture grouping bytes into dynamic patches, arguing that patches scale better than tokens. This new work follows that line but with a different approach: rather than building a byte model from scratch, it adapts models already trained. For people running local models, the question is also practical, since smaller models are the ones that run on home hardware and personal devices.
For now there are no release dates or public weights for the byte models described in the paper, and it is not clear whether Meta intends to adopt the approach in its production models. The projection figures are those relayed by the post that spread the work on the r/LocalLLaMA community and still need confirmation against the original paper, where the estimates are framed as forecasts based on training curves.

No comments yet. Be the first!