What Is Subword Tokenization?
- 2 days ago
- 28 min read
Every prompt you type into an AI chatbot gets chopped into pieces before the model ever "reads" a word — and how those pieces are cut shapes how much you pay, how far your text reaches, and how well the model handles your name, your code, or your language. That quiet slicing step is called tokenization, and the method behind nearly every modern large language model is subword tokenization.
TL;DR
Subword tokenization splits words into units larger than a single character but often smaller than a whole word, so common words stay intact while rare words break into reusable pieces.
It solves the out-of-vocabulary problem that plagued word-level systems, letting models handle new names, typos, and technical terms without a fixed dictionary.
The four algorithms behind most production tokenizers are BPE, byte-level BPE, WordPiece, and the Unigram language model — SentencePiece is a toolkit that implements BPE or Unigram, not a fifth algorithm.
Tokenization has real tradeoffs: it affects context-window usage, inference cost, multilingual fairness, and how well a model handles spelling, arithmetic, and code.
No tokenizer is neutral — research shows some languages need up to 15 times more tokens than English for the same meaning under the same tokenizer.
What Is Subword Tokenization?
Subword tokenization is a text-processing method that splits words into smaller, reusable pieces — larger than single characters but often smaller than whole words. Frequent words stay whole, while rare or unfamiliar words break into common fragments. This lets language models handle any text, including new words, misspellings, and multiple languages, without storing every possible word in advance.
Table of Contents
Subword Tokenization: A Plain-English Definition
Subword tokenization divides a stream of text into units that are generally larger than individual characters or bytes but smaller than, or sometimes equal to, whole words. A common word like "the" or "running" might stay as one intact token, while a rare or unfamiliar word — a brand-new product name, a misspelling, or a technical term — gets broken into smaller, reusable fragments.
The intuition is simple. If a tokenizer has already learned pieces like "token," "iz," and "ation" from its training data, it can represent the word "tokenization" as a short sequence of familiar pieces, even if that whole word never appeared often enough to earn its own vocabulary slot. This is illustrative rather than a guaranteed output — the exact split of any word depends on which tokenizer, trained on which text, produced which vocabulary.
This single idea — reuse small, frequent pieces instead of memorizing every possible word — is what lets a machine learning system built around a fixed-size list of tokens still handle language that keeps generating new words forever.
Why Language Models Need Tokenization
Neural networks do not read letters or words the way people do — they perform mathematical operations on numbers, so raw text must be converted into a numerical form first. The pipeline runs, at a high level: text goes into a tokenizer, the tokenizer splits it into tokens, each token maps to an integer ID from a fixed vocabulary, and each ID is looked up in an embedding table to produce a vector of numbers. That sequence of vectors is what enters the neural network as the model's input sequence.
It would be misleading to simplify this to "the model only sees numbers." The IDs themselves are arbitrary labels — what matters is the embedding step, where each ID maps to a learned vector capturing how that token relates to others based on training patterns. Two different vocabularies could assign the same word different IDs, but as long as each model trained its own embeddings against its own IDs, both can represent that word's meaning internally.
Tokenization is the broader umbrella term for this text-to-unit conversion; a "token" itself is not guaranteed to be a full word, syllable, or morpheme — it is whatever unit the specific tokenizer's vocabulary happens to contain.
Words vs. Characters vs. Bytes vs. Subwords
Before subword methods became standard, NLP systems mostly chose between whole-word vocabularies and individual characters; byte-level representations sit at a further extreme, and subwords occupy the practical middle ground dominating today's NLP systems.
Representation | Basic Unit | Vocabulary Characteristics | Typical Sequence Length | Strengths | Weaknesses | Appropriate Use Cases |
Word-level | Whole words | Very large (100,000+) to cover a language, still incomplete | Shortest | Tokens often align with meaning | Cannot handle unseen words; huge embedding tables | Older statistical NLP, closed-domain systems |
Character-level | Single characters | Very small (dozens to a few hundred) | Very long | No out-of-vocabulary words at all; tiny vocabulary | Long sequences increase compute; harder to learn long-range patterns | Some historical models; ingredient for hybrid systems |
Byte-level | Raw bytes (UTF-8) | Fixed and small (256 base values) | Longest for non-Latin scripts | Covers every possible input losslessly; simple base alphabet | Sequences can be very long for non-English text before merges are applied | Foundation for byte-level BPE tokenizers |
Subword | Learned fragments | Moderate (commonly 30,000–260,000) | Balanced | Reuses frequent pieces; handles rare words gracefully | Boundaries don't always match linguistic units | Nearly all modern LLMs and encoders |
The central tradeoff running through this table is between vocabulary size, sequence length, and coverage. A larger vocabulary can represent more text in fewer tokens, shortening sequences and reducing compute per document, but it also means a bigger embedding table, more memory, and less statistical reuse across related word forms — since "run," "running," and "runner" might each need their own slot rather than sharing a common piece. Subword tokenization exists to sit at a workable point on this curve.
The Out-of-Vocabulary Problem
Any system built around a fixed vocabulary must decide what happens when it meets a word not in that vocabulary — the out-of-vocabulary, or OOV, problem. Purely word-based vocabularies struggle badly here, because language is open-ended: new proper names appear constantly, misspellings are common, technical terms multiply, product names get invented, URLs and code identifiers follow no natural-language pattern, and languages with productive morphology generate huge numbers of inflected or compound forms from a modest set of roots.
A traditional word-level system had only bad options for unfamiliar input: map it to a generic "unknown" placeholder (destroying information), or maintain an ever-growing dictionary that could never keep pace.
Subword tokenization reduces this problem by decomposing unfamiliar words into pieces individually common enough to already be in the vocabulary. A previously unseen term can often be represented as a short sequence of familiar subwords, or in byte-level schemes as raw bytes, rather than a single opaque unknown symbol. This doesn't eliminate every OOV-adjacent problem, but it converts a hard failure into graceful degradation.
Inside a Modern Tokenizer Pipeline
A production tokenizer is rarely just "run one algorithm." It is a pipeline of distinct stages, and the exact order and set of stages varies by implementation:
Input text — the raw string as received.
Unicode handling — deciding how to interpret code points, combining characters, and encoding.
Normalization — general cleanup such as Unicode normalization forms (NFC or NFKC), optional lowercasing, and accent stripping.
Pre-tokenization — a rough initial split, often on whitespace and punctuation, producing candidate "words" before subword segmentation.
Subword segmentation — the trained model (BPE, WordPiece, or Unigram) splits each pre-token into final subword units.
Special-token handling — inserting markers such as beginning-of-sequence, end-of-sequence, or classification tokens.
Token-to-ID conversion — mapping each subword string to its integer ID in the vocabulary.
Optional truncation and padding — fitting the sequence to a fixed length for batching.
Model input — the final integer sequence, ready for embedding lookup.
Decoding or detokenization — converting generated token IDs back into readable text.
Per Hugging Face's documentation, normalization commonly performs cleanup like removing needless whitespace, optional lowercasing, and accent removal, and may apply Unicode normalization forms like NFC or NFKC [9]. These choices aren't cosmetic: lowercasing merges "Apple" the company with "apple" the fruit at the token level, and stripping accents merges words differing only by diacritics — both ripple forward and affect whether detokenization can perfectly reconstruct the original string.
How Subword Vocabularies Are Trained
Tokenizer training and using a tokenizer to encode new text are two separate processes. Training happens once, offline, against a large training corpus, and produces a fixed vocabulary and, in the case of BPE, a fixed list of merge rules. Encoding then applies that already-trained vocabulary to new text at inference time — no further learning happens during encoding itself.
Several choices during training materially change the resulting vocabulary:
Corpus selection and frequency distributions — a tokenizer trained on legal documents allocates its limited vocabulary slots differently than one trained on source code, biomedical papers, informal chat, or broad multilingual web text.
Initial symbols — most subword algorithms start from an initial alphabet of characters or bytes.
Target vocabulary size — a hyperparameter chosen before training, commonly in the tens of thousands for BPE and WordPiece, though modern byte-level tokenizers used with the largest models often use larger vocabularies.
Merge or pruning decisions — how the algorithm decides which pieces to combine (BPE) or remove (Unigram).
Special and reserved tokens — markers like padding, unknown, or task-specific control tokens added regardless of frequency.
Minimum frequency thresholds — pieces occurring too rarely may be excluded.
Language balance — for multilingual tokenizers, how much training text comes from each language directly shapes which languages get efficient, compact tokens and which get fragmented ones.
This is why the same sentence can segment completely differently under two tokenizers that both nominally use "BPE" — the training corpus and hyperparameters, not just the algorithm name, determine the final vocabulary.
How Byte Pair Encoding Works
Byte Pair Encoding began as a general-purpose data-compression algorithm. Philip Gage described it in a 1994 article in The C Users Journal, where the technique replaced the most frequently occurring pair of adjacent bytes in a file with a single unused byte, repeating the process to shrink the file [1]. It had nothing to do with language models at the time — it was simply a compression trick.
Rico Sennrich, Barry Haddow, and Alexandra Birch adapted this idea for open-vocabulary neural machine translation in their 2016 ACL paper. Instead of compressing arbitrary bytes, they applied the merge logic to characters within words, using the intuition that many word classes — names, compounds, cognates — are translatable via units smaller than whole words [2]. They reported BLEU-score improvements over a back-off dictionary baseline on English–German and English–Russian WMT 2015 tasks [2].
The training algorithm, in simplified form:
Start with a vocabulary of individual characters (or bytes), often marking word boundaries with a special end-of-word symbol.
Represent every word in the training corpus as a sequence of these initial symbols.
Count all adjacent symbol pairs across the corpus.
Find the single most frequent pair.
Merge that pair everywhere it occurs, creating one new symbol.
Repeat the counting-and-merging process.
Stop once a target vocabulary size is reached (or another stopping criterion is met).
To encode unseen text, the trained merges are applied in the order they were learned — called the merge rank — making encoding deterministic: the same input with the same trained tokenizer always produces the same output.
A worked toy example. A tiny illustrative training corpus contains "low" (5 occurrences), "lower" (2), and "newest" (6), each ending with an end-of-word marker written _. This is a simplified example, not output from any named production tokenizer.
Initial character sequences: l o w (×5), l o w e r (×2), n e w e s t _ (×6).
Round | Most Frequent Pair | New Symbol | Updated Segmentation (example word) |
1 | e + s | es | "newest" → n e w es t _ |
2 | es + t | est | "newest" → n e w est _ |
3 | l + o | lo | "low" → lo w ; "lower" → lo w e r |
4 | lo + w | low | "low" → low ; "lower" → low e r |
After four rounds, "newest" segments as n e w est and "low" as low . A real tokenizer would continue merging for many more rounds until reaching its target vocabulary size.
Trained BPE vocabularies are deterministic and fast to apply, but the merges are purely frequency-driven and can produce linguistically awkward splits — joining a suffix to a stem for no reason beyond statistical co-occurrence.
How Byte-Level BPE Works
Character-level BPE must decide what its initial "characters" are, and a system guaranteeing it can represent any Unicode text runs into a problem: the full space of Unicode symbols is very large. OpenAI's GPT-2 technical report noted that including every Unicode code point would require a base vocabulary of more than 130,000 symbols before any multi-symbol merges — prohibitively large compared to the 32,000-to-64,000-token vocabularies typical of word-level BPE at the time [7].
Byte-level BPE solves this by starting from raw UTF-8 bytes rather than Unicode characters. A byte can only take 256 values, so the base alphabet is fixed at exactly 256 symbols regardless of language or script, guaranteeing any UTF-8 text can be represented without an unknown-token fallback. The GPT-2 authors noted that applying BPE naively at the byte level produces suboptimal merges — common words end up with near-duplicate variants (separate merges around "dog," "dog.", "dog!") wasting vocabulary slots — and addressed this by preventing certain merges across character categories [7]. GPT-2's final vocabulary held 50,257 tokens, up from GPT-1's roughly 40,000, alongside a context-size increase from 512 to 1,024 tokens [7].
Byte-level BPE improves coverage — no text is fundamentally unencodable — but not every token maps neatly to something visually simple. A character outside the Latin alphabet, or an emoji that is a multi-byte Unicode sequence, can require several byte-level tokens, since a raw byte isn't by itself a meaningful character boundary. Any specific token-count example belongs to that tokenizer's particular vocabulary and merge table, not to "how GPT models tokenize" universally — different model families, and different versions within a family, use different vocabularies and pre-tokenization rules.
How WordPiece Works
WordPiece originates from Mike Schuster and Kaisuke Nakajima's 2012 ICASSP paper "Japanese and Korean Voice Search," which described techniques for handling an effectively infinite vocabulary for languages without simple space-based word boundaries [3]. The approach later became closely associated with the original BERT paper [6].
WordPiece's vocabulary-training objective differs from BPE's frequency-based merge criterion. Rather than always merging whichever pair is most frequent, training favors merges that improve a language-model-style likelihood objective over the training data — evaluated by how much a candidate merge improves the model's ability to predict the corpus, not just raw pair counts. It is important not to call this identical to BPE: the two are similar merge-based cousins, not the same procedure, and the original training objective is not re-implemented identically, verbatim, across every library offering "WordPiece" today.
WordPiece's encoding procedure is more consistently documented. Given a trained vocabulary, it commonly encodes new words using greedy longest-match-first: for a word, it finds the longest prefix already in the vocabulary, splits it off, and repeats on the remainder, marking non-initial pieces with a continuation prefix (rendered as ## in common BERT-style visualizations). If a word can't be decomposed this way, the encoder falls back to an unknown-token symbol.
Illustrative example (not verified live-tokenizer output): "unhappiness" might greedily decompose as un + ##happi + ##ness, assuming those pieces exist in the vocabulary. A different vocabulary could split the same word differently.
WordPiece's strengths are fast, deterministic, greedy encoding and its association with strong bidirectional encoders. Its limitation mirrors BPE's: greedy segmentation isn't guaranteed to align with morphology, and missing pieces trigger unknown-token fallback.
How the Unigram Algorithm Works
The Unigram algorithm takes the opposite direction from BPE's build-up-from-characters approach. Taku Kudo introduced it in the 2018 ACL paper "Subword Regularization," proposing a segmentation algorithm based on a unigram language model as part of a broader technique that samples multiple subword segmentations to make translation models more robust [4].
Unigram training starts from a large candidate vocabulary — generated from substrings observed in the corpus — then assigns each candidate piece a probability. Since a given string can typically be split into subwords more than one way, the algorithm defines a segmentation's likelihood as the product of its pieces' probabilities, and can compute a sentence's total probability by summing over possible segmentations. Training then iteratively removes lower-value pieces — pruning toward a smaller target vocabulary while trying to preserve overall corpus likelihood — until reaching the target size.
At deployment, the algorithm typically selects the single most likely segmentation (deterministic), but during training, subword regularization deliberately samples among several valid segmentations as a form of noise, making the downstream model more robust [4].
Illustrative example of segmentation ambiguity: "unwanted" could plausibly segment as un + want + ed, or unw + ant + ed, or un + wanted, depending on vocabulary pieces and their probabilities — exactly the multiplicity subword regularization exploits during training, which a deterministic encoder resolves to one answer at inference.
This probabilistic foundation, rather than a purely greedy or frequency-driven rule, is Unigram's key conceptual difference from BPE and WordPiece.
What SentencePiece Is
SentencePiece is often mistaken for a fifth segmentation algorithm alongside BPE, WordPiece, and Unigram. It is not. Kudo and Richardson describe it, in their 2018 EMNLP demo paper, as a language-independent subword tokenizer and detokenizer toolkit, providing open-source C++ and Python implementations, capable of training directly from raw sentences rather than requiring pre-tokenized word sequences [5]. SentencePiece implements BPE or Unigram as its underlying algorithm — it is not itself a third, separate algorithm, and should never be called a synonym for Unigram.
The problem SentencePiece solves is that earlier subword tools assumed input text had already been split into words, usually by whitespace, before segmentation began. That assumption breaks down for languages like Japanese or Chinese, which don't reliably use whitespace as a word boundary. Because SentencePiece trains directly on raw sentences, it sidesteps language-specific pre-tokenization rules, making the pipeline more genuinely language-independent [5].
To make raw-text training reversible, SentencePiece treats whitespace as part of the input rather than stripping it beforehand, commonly rendering a space as a visible marker (often ▁). Per Hugging Face's documentation, this lets decoding simply concatenate tokens and replace the marker with a literal space, producing reversible, lossless detokenization — a property plain whitespace-stripping tokenizers, like the classic BERT tokenizer, lack [9]. This whitespace convention is a design choice about representation, separate from whichever underlying algorithm decides the segmentation itself.
SentencePiece also directly supports the subword-sampling behind subword regularization when run in Unigram mode, giving practitioners Kudo's stochastic-segmentation technique through a production-grade toolkit [4][5].
BPE vs. WordPiece vs. Unigram vs. SentencePiece
Category | Starting Representation | Vocabulary-Learning Strategy | Encoding Strategy | Unknown-Text Handling | Whitespace Treatment | Deterministic? | Can Sample Segmentations? |
Standard subword BPE | Characters (with word-boundary markers) | Iterative frequency-based merges | Apply learned merges in rank order | Falls back to smallest known units; rare unknowns possible | Usually depends on separate pre-tokenization | Yes | No (standard form) |
Byte-level BPE | Raw UTF-8 bytes (256-symbol base alphabet) | Iterative frequency-based merges on bytes, with category constraints | Apply learned byte-level merges | Effectively none — any byte sequence is representable | Bytes for whitespace are part of the base alphabet | Yes | No (standard form) |
WordPiece | Characters | Merges chosen to improve a likelihood-style objective | Greedy longest-match-first | Falls back to an explicit unknown token if a word can't decompose | Typically depends on separate pre-tokenization; continuation pieces marked (e.g., ##) | Yes | No |
Unigram | Large candidate piece set | Probabilistic pruning toward target size | Most-likely segmentation (deterministic mode) | Depends on implementation; broad candidate sets reduce unknowns | Depends on implementation | Yes (at inference) | Yes (during training, via subword regularization) |
SentencePiece (framework) | Raw sentences (no pre-tokenization required) | Implements BPE or Unigram internally | Determined by whichever algorithm it runs | Depends on underlying algorithm | Whitespace encoded explicitly as part of input (e.g., ▁ marker) | Depends on mode | Yes, when running in Unigram mode |
A note on model associations: BERT is documented as using WordPiece [6]. GPT-2 is documented as using byte-level BPE [7]. T5-family and many multilingual models commonly use SentencePiece running a Unigram model, per Hugging Face's documentation, though the specific tokenizer configuration should always be verified for any given model release rather than assumed from a family-wide default [16].
A Complete Tokenization Example
To see the whole pipeline in one pass, follow a short, simplified example sentence — "Tokenizers split text." — through a conceptual (not verified-live-system) pipeline:
Original text: Tokenizers split text.
Normalized text: tokenizers split text. (illustrative lowercase normalization)
Pre-tokenized form: ["tokenizers", "split", "text", "."]
Subword pieces (illustrative): ["token", "izers", "split", "text", "."]
Token IDs (hypothetical, not from any real system): [10432, 8821, 5190, 2201, 18]
Conceptual embedding lookup: each hypothetical ID above is used to look up a corresponding vector in the model's embedding table, producing a sequence of numerical vectors that the model actually processes.
Decoded text: Tokenizers split text.
Two things are worth flagging. First, every token ID above is invented purely for illustration — it does not belong to BERT, GPT, T5, Claude, or any other real system. Second, decoding is not always as simple as decoding each token independently: some tokenizers (particularly byte-level ones) can produce tokens representing only a partial byte sequence, so a single token in isolation may not correspond to valid, displayable text until combined with its neighbors during decoding.
Morphology, Languages, and Subword Boundaries
Because subword pieces are learned purely from frequency and probability, they sometimes resemble real linguistic morphemes — prefixes, suffixes, roots — and sometimes do not. A frequency-driven merge can look exactly like a morpheme boundary by coincidence, but nothing in the BPE, WordPiece, or Unigram objectives explicitly enforces morphological correctness.
This matters more in some languages than others. Agglutinative languages, which build long words by stringing together many meaningful morphemes (commonly cited examples include Turkish and Finnish), can generate huge numbers of distinct surface forms from a comparatively small set of roots and affixes. Hugging Face's tokenizer documentation specifically notes that subword splitting is especially useful for a language like Turkish, where long, complex words form by stringing subwords together [16]. Fusional languages, which pack multiple grammatical meanings into a single affix, present a different challenge, since a suffix may need to be split or preserved in ways that don't map onto any single semantic unit. Languages that don't use whitespace as a reliable word boundary — Japanese and Chinese among them — are precisely why SentencePiece was designed to train directly on raw, unsegmented sentences [5].
The safe, general claim is that data-driven subword pieces can sometimes align with morphemes, especially for productive, regular affixes in well-represented languages, but this alignment is never guaranteed for any specific word without checking the actual tokenizer output.
Multilingual Tokenization and Token Inequality
When a single tokenizer trains on text spanning many languages, the shared vocabulary has to be allocated across all of them — and that allocation is rarely even. Because vocabulary slots are finite and training corpora are typically dominated by a handful of high-resource languages, languages making up a smaller share of the data tend to receive fewer dedicated pieces, forcing their text into longer, more fragmented token sequences.
Researchers measure this using fertility — roughly, the average tokens a tokenizer produces per word in a language. Aleksandar Petrov and colleagues, in their 2023 paper "Language Model Tokenizers Introduce Unfairness Between Languages," formalize this disparity, showing that tokenizing equivalent content can require dramatically more tokens in some languages than English under the same tokenizer, with cost discrepancies meaning some users can pay multiple times more for equivalent content [13]. Orevaoghene Ahia and colleagues extend this to commercial deployment in "Do All Languages Cost the Same?," showing that under per-token pricing and context-window limits, higher-fertility languages face real, measurable cost and usability penalties [14].
The practical consequences are threefold: higher-fertility languages consume more of a fixed context window per unit of content, they increase latency and compute cost proportionally to sequence length, and under token-based pricing they translate into measurable cost disparities — a pattern researchers increasingly frame as a fairness issue, though one that remains an active area of research rather than fully solved.
Code-switching, where a single message mixes scripts or languages, adds further difficulty, since a tokenizer's vocabulary allocation across scripts directly shapes how efficiently it represents mixed-language text.
Tokenizing Code, Numbers, Emojis, and Unusual Text
Subword tokenizers were designed with natural-language text in mind, and several categories of "unusual" input reveal the edges of that design:
Misspellings and new proper names typically decompose into smaller familiar fragments rather than an unknown token, though very unusual spellings can still fragment heavily.
Numbers and dates are handled inconsistently; some tokenizers split multi-digit numbers into short chunks, which can make arithmetic harder to learn since the same value may tokenize differently depending on context.
Punctuation, whitespace, and repeated characters (like "sooooo") follow pre-tokenization rules that vary by implementation and can produce unusual, lower-frequency sequences.
Emojis and combining marks can, at the byte level, require multiple bytes and therefore multiple tokens despite displaying as one glyph.
URLs, email addresses, and hashtags rarely appear as complete units in training corpora, so they often fragment into several tokens.
Source code and identifiers — including CamelCase and snake_case — often split at capitalization or underscore boundaries, depending on whether training data included substantial code.
Mathematical notation and mixed-language text can be especially fragment-prone if underrepresented in training.
None of these behaviors are universal facts about "tokenizers" — they vary by the specific vocabulary and training corpus, and any claim about a real system should be checked against that system directly.
How Tokenization Affects Model Performance and Cost
Tokenization choices ripple through nearly every practical dimension of working with a language model, though tokenization alone does not determine model quality — it interacts with training data, architecture, scale, and training procedure.
Sequence length and context-window utilization. Higher fertility for a piece of text consumes more of the model's fixed context window per unit of content, leaving less room for the rest of a conversation — a direct consequence of the fertility disparities discussed above [13][14].
Training throughput and inference latency. Because self-attention scales in compute with sequence length, longer token sequences for the same content mean slower training and more expensive inference.
Memory, spelling, and arithmetic. A larger vocabulary means a larger embedding table and output layer. Because subword pieces don't necessarily preserve character-level structure, exact spelling manipulation or digit-by-digit arithmetic can be harder for a subword-tokenized model, depending on how numbers and rare strings happen to segment.
Code generation and multilingual performance. Both are highly sensitive to whether the training corpus included enough representative code or non-English text — a tokenizer that fragments unfamiliar syntax or scripts increases the burden on the model to reconstruct meaning across a longer sequence.
How to Evaluate a Tokenizer
No single metric fully captures tokenizer quality — practitioners look at several together, distinguishing intrinsic tokenizer metrics from downstream model evaluation:
Fertility (tokens per word), characters/bytes per token, and compression ratio — lower fertility and higher compression generally mean more efficient representation for a language or domain.
Vocabulary coverage and unknown-token rate — how rarely it falls back to an unknown symbol.
Sequence-length distribution and whole-word retention — useful for anticipating context-window usage and how often common words stay single tokens.
Morphological alignment and cross-language parity — whether boundaries coincide with actual affixes, and whether fertility is measured consistently across target languages.
Domain coverage, encoding/decoding speed, and memory footprint — performance on target text types (legal, medical, code, chat) and throughput under real workloads.
Round-trip reversibility — whether decoding regenerates the exact original text.
Downstream task performance, robustness to noise, and fairness across scripts and languages — ultimately, how a model using this tokenizer performs, including on typos and across a balanced user base.
How to Choose or Train a Tokenizer
A practical decision framework for teams building or adapting an AI system that needs a tokenizer:
Start with an existing, well-documented tokenizer if your domain and languages resemble its training data — reusing a proven tokenizer avoids re-solving an already-validated problem. Train from scratch only if your domain or language mix is genuinely different, such as a heavily multilingual system, a specialized legal or scientific corpus, or a code-only assistant.
Choose vocabulary size deliberately. Larger vocabularies shorten sequences but increase embedding-table and output-layer size; there is no universally "correct" size — it depends on target languages, domain, and model scale.
Decide between BPE and Unigram based on priorities. Unigram's probabilistic foundation naturally supports subword regularization for robustness; BPE's frequency-driven merges are simple, fast, and extremely well-supported.
Decide whether byte-level coverage is worth it. It guarantees zero unknown-token failures on any UTF-8 input, at the cost of potentially longer sequences for underrepresented scripts.
Handle normalization and special tokens deliberately. Lowercasing, accent-stripping, and Unicode normalization choices affect reversibility; reserve padding, unknown, and task-specific control tokens up front.
Evaluate before a full pretraining run, using the intrinsic metrics above on a representative sample of your real target text, and version tokenizer files explicitly alongside model checkpoints.
Keep tokenizer and model checkpoints tightly aligned. Replacing a tokenizer after training is not a drop-in swap — token IDs, embeddings, and every learned representation are coupled to the original vocabulary, so a new tokenizer effectively invalidates them unless the model is retrained or carefully adapted.
Practical Python Example
The following short example trains a tiny tokenizer on a toy corpus using Hugging Face's tokenizers library, based on the library's documented BPE trainer API [140][143]. It is meant purely to illustrate the mechanics; a real production tokenizer is trained on a corpus many orders of magnitude larger.
# pip install tokenizers
from tokenizers import Tokenizer
from tokenizers.models import BPE
from tokenizers.trainers import BpeTrainer
from tokenizers.pre_tokenizers import Whitespace
# 1. Initialize an empty tokenizer using the BPE model
tokenizer = Tokenizer(BPE(unk_token="[UNK]"))
# 2. Split on whitespace before subword segmentation runs
tokenizer.pre_tokenizer = Whitespace()
# 3. Configure training: target vocabulary size and special tokens
trainer = BpeTrainer(
vocab_size=500,
special_tokens=["[UNK]", "[PAD]", "[CLS]", "[SEP]"]
)
# 4. Train on a small toy corpus (replace with real files for production use)
toy_corpus = ["toy_corpus.txt"] # a plain text file, one example per line
tokenizer.train(files=toy_corpus, trainer=trainer)
# 5. Encode a sample sentence and inspect the resulting tokens and IDs
output = tokenizer.encode("Subword tokenization splits rare words.")
print(output.tokens)
print(output.ids)Outputs will vary by library version, training corpus, vocabulary size, and special-token configuration — this toy example illustrates mechanics, not production-ready settings, and shouldn't be treated as a deployment recommendation.
Common Misconceptions
Myth | Reality |
One token always equals one word | Common words may be a single token, but rare words split into multiple subword pieces, and some tokens represent only part of a word. |
A token is a fixed number of characters | Token length varies — a single token can be one character, several characters, or (in byte-level schemes) part of a multi-byte character. |
Subwords are always real linguistic morphemes | Subwords are learned from frequency or probability; they sometimes align with morphemes and sometimes do not. |
All language models use the same tokenizer | Different model families — and different versions within a family — commonly use different vocabularies, merge tables, or algorithms entirely. |
SentencePiece is the same thing as Unigram | SentencePiece is a toolkit that can run either BPE or Unigram internally; it is not itself a segmentation algorithm. |
BPE and WordPiece are identical | Both are merge-based, but WordPiece's training objective favors a likelihood-style criterion rather than raw pair frequency, and its standard encoding is a distinct greedy longest-match procedure. |
Byte-level tokenization means one visible character equals one token | A single visually simple character outside the base byte alphabet can require multiple byte-level tokens. |
A larger vocabulary is always better | Larger vocabularies shorten sequences but increase embedding-table size, output-layer compute, and memory footprint. |
Subword tokenization completely eliminates unknown text | It greatly reduces, but does not eliminate, out-of-vocabulary and fragmentation problems — extremely unusual input can still fragment heavily. |
Changing the tokenizer does not affect a trained model | Token IDs, embeddings, and learned representations are tightly coupled to the original vocabulary; swapping tokenizers generally requires retraining or careful adaptation. |
The tokenizer understands meaning before the model processes the tokens | Tokenization is a mechanical, frequency- or probability-driven splitting process; meaning is learned later by the model itself through embeddings and subsequent layers. |
Advantages and Limitations
Potential benefits:
Open-vocabulary behavior that gracefully handles previously unseen words.
Reuse of shared subword pieces across related word forms, improving statistical efficiency.
A manageable, fixed vocabulary size compared to enumerating every possible word, with shorter sequences than pure character-level tokenization.
Better handling of rare terms than strict word-level vocabularies, with practical, well-supported compatibility with today's training and inference tooling.
Potential limitations:
Token boundaries do not reliably correspond to linguistic meaning or morpheme boundaries.
Languages or domains underrepresented in the training corpus suffer disproportionate fragmentation, and sequence lengths for equivalent content vary substantially across languages [13][14].
Sensitivity to training-corpus composition means the same algorithm can produce very different vocabularies.
Difficulty with tasks requiring exact character-level manipulation, such as certain spelling or digit-precise arithmetic.
Tight coupling between vocabulary and trained model, making tokenizer changes costly after pretraining, plus inconsistent token costs across languages under token-based pricing.
Complex Unicode behavior and added preprocessing assumptions that can affect reversibility if not handled carefully.
None of these benefits or limitations apply identically to every algorithm and system — the degree to which they manifest depends heavily on training corpus, vocabulary size, and implementation.
Token-Free Alternatives and Future Directions
Given the tradeoffs above, researchers continue exploring approaches that reduce or eliminate the fixed-vocabulary subword step entirely.
CANINE, from Jonathan H. Clark, Dan Garrette, Iulia Turc, and John Wieting, is a neural encoder operating directly on character sequences without explicit tokenization or a fixed vocabulary, with a pre-training strategy that can optionally use subwords as only a soft inductive bias [15].
ByT5, from Linting Xue and colleagues, shows a largely standard Transformer architecture can process raw UTF-8 byte sequences directly, characterizing the tradeoffs in parameter count, training compute, and inference speed, and demonstrating byte-level models can be competitive with token-level counterparts on several benchmarks [11].
Both belong to a broader "token-free" research direction: models operating on raw bytes or characters can process any language out of the box, may be more robust to noisy input, and remove tokenizer preprocessing complexity. The consistently cited tradeoff is that byte- or character-level sequences run considerably longer than subword sequences for the same content, increasing computational burden unless the architecture adds downsampling or patching to compensate.
Beyond these two systems, active directions include dynamic or adaptive tokenization, vocabulary-expansion techniques for adapting an existing tokenizer to new domains, morphology-aware segmentation, and multilingually balanced tokenizer training aimed at reducing the fertility disparities documented above [13][14]. These remain areas of ongoing exploration, and any specific efficiency claim should be treated as tied to the paper and benchmark that produced it.
FAQ
What is subword tokenization in simple terms?
A way of splitting text into pieces bigger than single letters but often smaller than whole words. Frequent words stay whole; rare or unfamiliar words break into smaller, reusable fragments the tokenizer already knows.
Why do large language models use subword tokenization?
It balances vocabulary size against sequence length, letting a moderate-sized vocabulary represent essentially any input, including unseen words, without an impossibly large word dictionary or the very long sequences pure character-level tokenization would require.
Is a subword the same as a morpheme?
Not necessarily. Subwords are learned from statistical frequency or probability, and sometimes coincide with real morphemes like prefixes or suffixes, but this is never guaranteed for any specific token.
What is the difference between BPE and WordPiece?
Both are merge-based, but BPE always merges the most frequent adjacent pair, while WordPiece selects merges based on a likelihood-style objective rather than raw frequency, and typically uses a distinct greedy longest-match-first encoding procedure.
Is SentencePiece a tokenizer or an algorithm?
A language-independent tokenizer toolkit, not a standalone algorithm. It can run either BPE or Unigram internally and adds its own conventions for whitespace handling and training directly from raw, unsegmented text.
How does Unigram tokenization work?
It starts from a large candidate piece set, assigns each piece a probability, and iteratively prunes the least useful pieces while preserving overall corpus likelihood, until it reaches a target vocabulary size. Encoding typically selects the most probable segmentation, though training can sample among multiple valid ones.
Can subword tokenizers handle unknown words?
Yes, far better than word-level vocabularies. Unfamiliar words are usually decomposed into smaller, known pieces rather than mapped to a single unknown-token placeholder, though extremely unusual input can still fragment heavily.
Why do different languages produce different token counts for the same meaning?
Vocabulary allocation in a shared multilingual tokenizer depends on how much training data each language contributed. Underrepresented languages get fewer dedicated pieces, forcing more fragmented tokens — a disparity researchers call tokenizer fertility [13][14].
Why do emojis sometimes use multiple tokens?
In byte-level tokenizers, many emoji are multi-byte Unicode sequences. Since the base alphabet works at the byte level, one visible emoji can require several tokens even though it displays as a single character.
Does tokenization affect API cost?
Yes, in systems metering usage by token count. Because token counts vary by language, content type, and tokenizer, the same content can cost different amounts depending on how efficiently it tokenizes [14].
Can a tokenizer be changed after model training?
Not as a simple swap. Token IDs, embeddings, and learned representations are all tied to the original vocabulary. A new tokenizer generally requires retraining or careful adaptation.
How is vocabulary size selected?
It's a hyperparameter chosen before training, balancing sequence length against embedding-table and output-layer size. There is no single correct value — it depends on target languages, domain, and model scale.
What tokenizer does BERT use?
The original BERT paper is associated with WordPiece [6], following the vocabulary-learning approach first described for Japanese and Korean voice search [3].
What is byte-level BPE?
A version of BPE that starts from raw UTF-8 bytes, using a small, fixed 256-symbol base alphabet instead of Unicode characters, guaranteeing any UTF-8 text can be represented — as demonstrated in OpenAI's GPT-2 technical report [7].
Are token-free models better?
They offer real advantages — no fixed vocabulary, no out-of-vocabulary problem, and potentially better robustness to noisy or multilingual text, as shown by CANINE and ByT5 [11][15]. The tradeoff is longer raw sequences, which increases computational cost unless the architecture compensates. This remains active research rather than a settled replacement.
Key Takeaways
Subword tokenization sits between character-level and word-level representations, splitting rare words into reusable pieces while keeping common words intact.
It solves the out-of-vocabulary problem inherent to any fixed-vocabulary system, without the full sequence-length cost of character-level tokenization.
BPE, byte-level BPE, WordPiece, and Unigram are four distinct algorithms; SentencePiece is a toolkit that can implement BPE or Unigram, not a separate fifth algorithm.
BPE originated as a data-compression technique before Sennrich, Haddow, and Birch adapted it for neural machine translation in 2016.
WordPiece traces to Schuster and Nakajima's 2012 voice-search research and later became associated with BERT.
Unigram, introduced by Kudo in 2018, takes a probabilistic pruning approach and enables subword regularization during training.
Tokenizer training and applying a trained tokenizer to new text are separate processes; the training corpus and hyperparameters shape the vocabulary as much as the algorithm choice.
Multilingual research documents real fertility and cost disparities, with some languages needing far more tokens than English for equivalent content.
Replacing a tokenizer after a model is trained is not a simple swap, since token IDs and embeddings are coupled to the original vocabulary.
Token-free alternatives like CANINE and ByT5 remove the fixed-vocabulary step entirely, trading shorter sequences for added computational cost.
Actionable Next Steps
Inspect how your own target text tokenizes under a few existing tokenizers before assuming any one fits your domain or languages.
If you work across multiple languages, measure fertility on representative samples rather than assuming a general-purpose multilingual tokenizer is equally efficient everywhere.
For specialized systems (legal, medical, code-only), check whether a general-purpose tokenizer fragments your domain vocabulary excessively, and train a domain-specific tokenizer if it does.
Read the original Sennrich, Schuster and Nakajima, Kudo, and Kudo and Richardson papers directly for a rigorous, primary-source understanding of any single algorithm.
Experiment with the Hugging Face tokenizers library on a small toy corpus to build hands-on intuition before working with production-scale vocabularies.
If cost or context-window usage matters, measure actual token counts on real sample text rather than estimating from word counts.
Version and document your tokenizer choice alongside any model you train, since the two are permanently coupled once training begins.
Revisit this topic periodically, since multilingual fairness research and token-free architectures are active, evolving areas.
Glossary
Byte Pair Encoding (BPE): Iteratively merges the most frequent adjacent symbol pair; originally a compression algorithm.
Byte-level BPE: BPE starting from raw UTF-8 bytes instead of Unicode characters, guaranteeing full coverage with a small, fixed base alphabet.
Detokenization: Converting tokens or token IDs back into readable text.
Embedding: A learned numerical vector representing a token's usage patterns.
Fertility: How many tokens a tokenizer produces per word for a given text; used to compare efficiency across languages.
Fine-tuning: Continuing to train a pretrained model on additional, often domain-specific, data.
Merge rank: The order BPE merge rules were learned; applied identically at encoding time, making it deterministic.
Normalization: Preprocessing cleanup — Unicode normalization, optional lowercasing, accent handling — before tokenization.
Out-of-vocabulary (OOV): A word absent from a fixed vocabulary, historically requiring an unknown-token fallback.
Pre-tokenization: An initial rough split, often on whitespace and punctuation, before subword segmentation.
Subword regularization: Kudo's technique of sampling among multiple valid segmentations to make training more robust.
Token: A unit of text from a tokenizer; not necessarily a whole word, syllable, or character.
Token ID: The integer a vocabulary assigns to a token, used for embedding lookup.
Tokenization: Converting raw text into tokens mappable to numerical identifiers.
Transformer: The self-attention architecture underlying most models that consume tokenized input.
Unigram language model: Starts from a large candidate vocabulary and probabilistically prunes toward a target size; introduced by Kudo in 2018.
Vocabulary: The fixed set of tokens and IDs a tokenizer produces.
WordPiece: Traces to Schuster and Nakajima's voice-search research and later BERT; uses a likelihood-style objective and greedy longest-match encoding.
Sources & References
Gage, P. (1994). A New Algorithm for Data Compression. The C Users Journal, 12(2), 23–38. https://www.derczynski.com/papers/archive/BPE_Gage.pdf
Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1715–1725. ACL Anthology. https://aclanthology.org/P16-1162/
Schuster, M., & Nakajima, K. (2012). Japanese and Korean Voice Search. 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 5149–5152. https://ieeexplore.ieee.org/document/6289079
Kudo, T. (2018). Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). ACL Anthology. https://aclanthology.org/P18-1007/
Kudo, T., & Richardson, J. (2018). SentencePiece: A Simple and Language Independent Subword Tokenizer and Detokenizer for Neural Text Processing. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 66–71. ACL Anthology. https://aclanthology.org/D18-2012/
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, 4171–4186. ACL Anthology. https://aclanthology.org/N19-1423/
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners. OpenAI. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
Google. (2018). SentencePiece: Unsupervised Text Tokenizer for Neural Network-Based Text Generation [Software repository]. GitHub. https://github.com/google/sentencepiece
Hugging Face. (2024). Normalization and Pre-tokenization. LLM Course, Chapter 6. https://huggingface.co/learn/llm-course/en/chapter6/4
Hugging Face. (2024). The Tokenization Pipeline. Tokenizers Documentation. https://huggingface.co/docs/tokenizers/en/pipeline
Xue, L., Barua, A., Constant, N., Al-Rfou, R., Narang, S., Kale, M., Roberts, A., & Raffel, C. (2022). ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models. Transactions of the Association for Computational Linguistics, 10, 291–306. https://arxiv.org/abs/2105.13626
OpenAI. (2022). tiktoken: A Fast BPE Tokeniser for Use with OpenAI's Models [Software repository]. GitHub. https://github.com/openai/tiktoken
Petrov, A., La Malfa, E., Torr, P. H. S., & Bibi, A. (2023). Language Model Tokenizers Introduce Unfairness Between Languages. arXiv preprint arXiv:2305.15425. https://arxiv.org/abs/2305.15425
Ahia, O., Kumar, S., Gonen, H., Kasai, J., Mortensen, D. R., Smith, N. A., & Tsvetkov, Y. (2023). Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models. arXiv preprint arXiv:2305.13707. https://arxiv.org/abs/2305.13707
Clark, J. H., Garrette, D., Turc, I., & Wieting, J. (2022). CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation. Transactions of the Association for Computational Linguistics, 10, 73–91. https://arxiv.org/abs/2103.06874
Hugging Face. (2024). Tokenization Algorithms — Byte-Pair Encoding, WordPiece, and Unigram. Transformers Documentation. https://huggingface.co/docs/transformers/tokenizer_summary
