If you have ever wondered why an AI bill counts tokens instead of words, start with this: an LLM token is not the same thing as a word. LLM tokenization is the process that turns text into small pieces a model can process. A piece might be a whole word, part of one, punctuation, or another unit chosen by that model’s tokenizer.
Featured image: AI-generated illustration.
That difference matters when you compare an English paragraph with Hindi text, paste in source code, send a long list of numbers, or estimate how much a model request will cost. Word count gives you a rough sense of length. It does not tell you the exact token count.
What LLM tokenization means
A language model works with numbers, not directly with the letters on your screen. Before text reaches the model, a tokenizer splits it into tokens and maps those tokens to numeric IDs. The model processes the IDs. When it replies, its output is also generated as tokens and then converted back into readable text.
Think of a token as a model’s working unit for text. It is a little like a puzzle piece, except the boundaries are chosen by a learned vocabulary and a set of tokenizer rules. The same visible word can be split differently by two models because their tokenizers may have different vocabularies and rules.
Tokenization is not a glossary lookup. The model is not necessarily retrieving a stored entry for each word. It receives a sequence of token IDs, which are the numeric labels for the pieces its tokenizer created.
Characters, words, tokens and token IDs are different
Take the phrase cats sleep. It has ten visible characters if you count the space, two words, and some number of tokens determined by a particular tokenizer. A tokenizer might keep cats together, or it might split it into smaller pieces. It might also treat the leading space as part of a token.
- Character: a visible text element, such as
cor!. Unicode text can be more complicated than one character per byte, so “character” itself has technical details. You do not need those details to understand the basic idea. - Word: a language unit separated by spaces or punctuation in many writing systems. Some languages do not mark every word boundary with spaces.
- Token: a unit produced by a specific tokenizer. It can be a word, part of a word, punctuation, or another piece.
- Token ID: the number assigned to a token in that tokenizer’s vocabulary. IDs are labels, not scores or meanings by themselves.
So a sentence’s word count and token count answer different questions. A word counter follows language-like boundaries. A tokenizer follows the model’s own encoding rules. OpenAI’s token guide gives rough English estimates such as about four characters or three-quarters of a word per token, but explicitly treats them as estimates. They are not a reliable conversion rule, especially outside English.
How a tokenizer breaks text into pieces
Many widely used language-model tokenizers use subword pieces. Hugging Face’s tokenizer lesson gives examples of how these pieces work. Common text can often be represented by larger chunks; less common words may be split into smaller parts that also appear in other words. For example, a tokenizer might break an uncommon word into a familiar root and ending. The exact split is tokenizer-specific, so an example from one model should not be assumed to match another.
This approach avoids two awkward extremes. A word-only tokenizer needs a separate vocabulary entry for every spelling it might encounter, while a character-only tokenizer can turn ordinary text into a very long sequence. Subword tokenization reuses pieces: a new or rare word can often be represented by combining pieces the tokenizer already knows.
The split is not a language teacher’s analysis of meaning. A tokenizer may divide a word in a way that looks strange to a person. It follows patterns learned when the tokenizer vocabulary was built. A token boundary can also depend on capitalization, punctuation, and spaces around the text.
Why English, Hindi, code, numbers and emoji can tokenize differently
A tokenizer’s vocabulary is built from data, so text patterns that were common in that data may be represented more compactly than rare ones. That can make token counts differ across languages and writing systems. It can also make a familiar English word short in tokens while an uncommon name, a technical term, or a word in another script takes more pieces.
Compare English and Hindi
Consider the English phrase Good morning and a Hindi equivalent such as सुप्रभात. They communicate a similar greeting, but their number of words, characters, and tokens need not line up. Hindi is written in Devanagari; a tokenizer may have different coverage for those patterns than for common English text. To compare exact counts, use the tokenizer for the model you plan to call.
Mixed Hindi-English text
People often mix scripts in one message: Kal meeting 3 baje hai, please send the notes. A tokenizer handles the whole sequence, including Devanagari if present, Latin-script English, digits, spaces, and punctuation. Do not estimate its cost by counting just the English words or by assuming each Hindi word is one token.
Code and number strings
Code contains punctuation, indentation, symbols, and names that may not occur often in ordinary prose. A compact line can therefore take more tokens than its character count suggests. Number strings are another useful test: a tokenizer may split digits into chunks rather than treating a long number as one unit. The way numbers are split can affect how reliably a model handles arithmetic, but tokenization alone does not explain every math error.
Emoji and unusual spellings
An emoji that looks like one symbol to you may be represented by several underlying Unicode parts and may take multiple tokens. Skin-tone modifiers and combined emoji can add further complexity. Intentional misspellings and long strings of repeated characters can also tokenize differently from common spellings. For exact behavior, inspect the chosen model’s tokenizer rather than relying on a screenshot or example from another model.
How tokens affect API cost, context limits and response time
For a token-priced API, input and output token counts are part of the bill. The exact rates depend on the provider, model, and token category, so check the provider’s current pricing page before estimating a real workload. Some services distinguish cached input or internal reasoning usage from ordinary visible output. A short-looking answer is not always proof that the request used very few tokens.
Tokens also use space in a model’s context window, which is the amount of material it can consider in a request. The prompt, conversation history, attached text, tool information, and generated output can all matter. A request can hit a limit even when its word count seems manageable.
Output length often affects how long generation takes because models produce output token by token. But token count is not a stopwatch: model speed, request processing, server load, batching, network delay, and other factors matter too. Treat token count as a useful input to planning, not a promise about cost or latency.
One more catch: counting only the plain text you typed may not equal the complete request count. Message formatting, tool definitions, schemas, images, and files can contribute to a request. Provider usage reports or a provider’s request-counting tool are better for checking a real call.
How to count tokens for your model
- Choose the model you intend to use. Different models can use different tokenizer encodings. OpenAI provides an interactive tokenizer for inspecting how its encodings split text.
- Use the provider’s tokenizer or token-counting tool to inspect representative text. OpenAI has an interactive tokenizer and recommends the matching encoding for programmatic counting; Hugging Face models provide their tokenizer with the model checkpoint.
- Test the material you actually send: include instructions, examples, code, non-English text, and repeated conversation history where relevant.
- After an API call, inspect its usage report. That is the practical check against a preflight estimate.
If you are building retrieval-augmented generation, use the model’s tokenizer when you split documents into token-sized chunks. Our guide to chunking and reranking in RAG explains why a character-count shortcut can cut text at the wrong place. For the wider model picture, read what an LLM actually does.
Try this: compare four kinds of text
Pick one model and paste these into its tokenizer: a short English paragraph, a Hindi translation of the same idea, a few lines of code, and a list of numbers such as 17, 2048, 3.14159, 987654321. Record the word count and token count for each. Then swap the model or tokenizer and run the same examples again.
Before checking the result, guess which sample will produce the most tokens. Then ask why your guess was right or wrong. You are not trying to prove that one language is “more efficient” or that one model is better. You are seeing how tokenizer vocabularies represent different text patterns.
For a more useful cost estimate, put the actual input and a realistic expected response through the provider’s tools, then compare the reported usage after a test call. Keep the model fixed while you compare text; otherwise, you may be comparing both the language and the tokenizer at once.
Common misconceptions
- “One word equals one token.” Sometimes, but not as a rule. A word can map to one token or several.
- “Four characters always equal one token.” That is only a rough English estimate for some contexts, not a universal conversion.
- “All models count the same text the same way.” Tokenizer and encoding choices can differ. Use the target model’s tokenizer.
- “A tokenizer understands the word the way a person does.” It applies learned segmentation rules; the boundary need not match a linguistic root or syllable.
- “The text-only count is my full API bill.” Request structure and other inputs can affect usage, and providers may bill token categories differently.
Frequently asked questions
What is a token in an LLM?
A token is a piece of text produced by a model-specific tokenizer. It might be a whole word, part of a word, punctuation, or another unit. The tokenizer maps it to an ID the model can process.
How many tokens are in one English word?
There is no fixed number. Common short words may be one token, while longer or less common words may split into multiple tokens. The exact answer depends on the tokenizer and surrounding text.
Why can Hindi or code use more tokens than English?
Tokenizers have different vocabularies and may represent some scripts or patterns in smaller or larger pieces. Code, number strings, and mixed-script text also have different patterns from ordinary English prose. Check the target model to know the count.
Do more tokens always mean a more expensive or slower response?
More billable tokens can raise token-based API cost, but rates vary by model and token type. More generated tokens often take longer to produce, but latency also depends on the model, service, and network. Measure a representative request.
Can I use a word counter to estimate my prompt?
Only for a rough sense of size. Use a tokenizer for the model you plan to call, and check the API usage report after a test request.
The useful rule to remember
Tokens are the model’s units for processing text; words are ours. That is why two passages with similar word counts can have different token counts, context use, and API costs. If the number matters, count with the tokenizer for the model you will actually use.
Try the four-text comparison above, then check the token count before your next API call. Once you can explain why your samples differ, you have the core M1.2 idea down.