Published in the journal Nature, the research, co-authored by Dr Edoardo Maria Ponti, Associate Professor in Natural Language Processing in the Department of Computing at Imperial College London, presents a new approach called "byteification". This process allows AI models to work directly with the underlying digital representation of text, known as bytes, while retaining the capabilities of current leading AI systems.
This flexibility could lead to AI systems that operate more effectively across different languages and scientific data, such as code, biological sequences, and multilingual text. The approach is less biased towards high-resource languages, potentially enabling more efficient processing across a wider range of languages and writing systems. The research was conducted with colleagues from the Allen Institute for AI (Ai2) and partner institutions.
Dr Ponti explained: "This project aligns the unit in which language models perceive text with the unit in which all digital data is stored: the byte. Our models instead learn, at the same time, how to predict text and how to merge bytes into larger units."
Large language models typically divide text into units called tokens, usually words or parts of words. While this has driven AI advancements, it can obscure information within individual characters, posing challenges for domains like software code and biological sequences. The new "byteification" method converts existing models with minimal additional training, outperforming previous byte-level approaches and maintaining competitive performance on broader language benchmarks.
The findings suggest a path towards more capable AI for tasks requiring fine-grained textual understanding, benefiting software engineering, biology, and multilingual applications by removing a major barrier to AI systems operating directly on raw text encodings.
