What is a token in AI?
In AI, a token is a small unit of text that a language model reads and produces. A token can be a whole word, part of a word, a number, or punctuation. Models process and count text in tokens rather than in words or characters.
Last reviewed .
How tokens work
Before a model reads your text, a tokenizer splits it into tokens. Common words are often a single token, while long or unusual words may be split into several pieces.
Different models split text differently, so the same passage can have different token counts in different tools.
Why tokens matter
Tokens set the limits and often the price of using a model. The context window is measured in tokens, and many AI services charge by the number of tokens sent in and generated.
Shorter, focused prompts use fewer tokens, which can lower cost and speed up responses.
Estimating token counts
Token counts depend on the model and language, so use the provider's tokenizer or usage reports for accurate numbers. Treat any rule of thumb as a rough guide only.
Common questions
Is a token the same as a word?
No. A token can be a whole word, part of a word, or a punctuation mark, so a passage usually contains a different number of tokens than words.
Why do AI tools charge by tokens?
Processing text takes computing power, and tokens are a consistent way to measure how much text is handled. Pricing usually counts both input and output tokens.
How do tokens relate to the context window?
The context window is the maximum number of tokens a model can handle at once. Your prompt, any attached material, and the response all count toward it.
How can I use fewer tokens?
Write concise prompts, send only the material the task needs, and ask for shorter outputs when that suits you.
Want help putting this to work?
Tell us where your marketing stands today, and we'll show you what to do first.