Marcel Røed released Gigatoken, an open-source Python library that claims to tokenize text nearly 1,300 times faster than Hugging Face’s widely used tokenizers package on the same hardware. Every large-scale training run and every high-volume inference deployment depends on tokenization somewhere in the pipeline, so a step normally invisible to developers is the one generating the headline number. The figure comes entirely from benchmark tables that Røed wrote, ran, and published himself.
The library is pitched as a drop-in swap. A developer can wrap an existing Hugging Face tokenizer object and call it in compatibility mode, or wrap a tiktoken tokenizer the same way, changing only a few lines of code while keeping the rest of the call site untouched. Røed’s documentation notes that compatibility mode costs some throughput to guarantee matching output, and that the largest gains require calling Gigatoken’s native API directly against raw text files rather than routing Python strings through it.
The raw numbers are specific. On an AMD EPYC 9565 system with 144 cores across two sockets, encoding an 11.9 gigabyte OpenWebText sample with a GPT-2 tokenizer, Gigatoken reached 24.53 GB/s against Hugging Face’s 24.8 MB/s and tiktoken’s 36.0 MB/s, a gap of 989x and 681x respectively. On a 16-core Apple M4 Max, the same GPT-2 comparison produced 8.79 GB/s against Hugging Face’s 6.9 MB/s, a 1,268x difference. Røed says a validation pass on the same run confirmed that all 20,401 documents matched Hugging Face’s output token-for-token.
The mechanism Røed describes is concrete rather than vague. Pretokenization, the step that splits raw text into word-like chunks before encoding, is usually handed off to a general-purpose regex engine. Gigatoken instead implements that step with SIMD instructions tuned separately for x86 and ARM chips, caches token mappings for words it has already seen so repeats skip re-encoding, and minimizes how often the Rust core has to cross back into Python.
The skepticism here is structural, not a hedge. These are the project’s own benchmarks, run on hardware Røed selected and reported without independent verification. More importantly, the multiple is not uniform across tokenizers. BPE-style tokenizers used by GPT-2, Llama, Qwen, and DeepSeek see gains in the 300x to 1,300x range in Røed’s tables. SentencePiece-based tokenizers, the family used by Gemma, Mistral, and CodeLlama, see only 7x to 22x, because Røed’s own documentation states that SentencePiece has not received the same optimization work. WordPiece tokenizers are not supported at all, and Windows use is untested, with the project recommending WSL instead.
Tokenization speed is not a bottleneck most teams ever notice, because a single chat turn tokenizes in microseconds regardless of which library handles it. It becomes a real constraint in three places: preparing terabyte-scale corpora before a pretraining run, running tokenization inline in a high-throughput inference server, and iterating repeatedly over a large dataset during preprocessing experiments. Røed’s own math illustrates the scale where this matters: at the EPYC throughput reported in his tables, tokenizing a dataset the size of the full Common Crawl corpus, cited elsewhere at roughly 130 trillion tokens, would take just under six and a half hours.
Teams evaluating Gigatoken should check which tokenizer family their models actually use before assuming the headline multiple applies. A team standardized on GPT-2, Llama, or Qwen-style BPE tokenizers stands to see the largest gains reported here; a team on Gemma or Mistral’s SentencePiece tokenizers should expect a single-digit speedup, not a thousand-fold one, and should benchmark on their own corpus before rearchitecting a data pipeline around it.
Published on GitHub by the gigatoken project (Marcel Røed) on July 21, 2026.
