Glossary · Beginner
What Are Tokens? Definition and Example
A token is the unit of text a language model processes: a whole word, part of a word, a character or a punctuation mark. Google's Gemini documentation says that "a token is equivalent to about 4 characters" and that "100 tokens is equal to about 60-80 English words" (Google).
On this page
- How do tokens work?
- What's an example?
- Why do tokens matter for GEO and marketing?
- What are the common misconceptions?
- From MentionX data: why token-level claims are hard to test
- Worked example: Ledgerline estimates a monitoring budget
- Why do brand names split into several tokens?
- Where will marketers meet tokens?
- Common questions
- Related
- Sources
Checked on 31 October 2026. "Our view" marks MentionX opinion. Ledgerline is a fictional company used for illustration.
How do tokens work?
Before a model reads text, a tokeniser splits it up. Google explains that tokens "can be single characters like z or whole words like cat", while "long words are broken up into several tokens". The full set is "called the vocabulary", and the splitting process is "called tokenization" (Google). Each model family has its own tokeniser, so the same sentence can produce different token counts in different tools.
What's an example?
Fictional example: a 1,200-word product guide comes to roughly 1,500–2,000 tokens using Google's rule of thumb (100 tokens ≈ 60–80 words). A 200,000-token context window could therefore hold dozens of such guides. In practice, AI search engines usually include only short passages.
Why do tokens matter for GEO and marketing?
- Limits: a context window is measured in tokens.
- Cost: API usage is often billed per token. Google notes that Gemini API cost is determined partly by token counts (Google). OpenAI bills embedding requests "based on the number of tokens in the input" (OpenAI).
- Output length: settings such as "max output tokens" cap how long a response can be (Google).
Our view: you don't need to write for tokens. It helps to know that brand names may be split into several tokens, and that dense, clear passages make better use of limited context.
What are the common misconceptions?
- "One token is one word." That's often wrong, especially for long words, names and languages other than English.
- "Token counts are the same everywhere." They differ by tokeniser.
From MentionX data: why token-level claims are hard to test
Method. Read-only aggregate queries over MentionX scans from 22 August to 29 October 2026 (UTC): 2,217 engine runs (2,154 completed) across 18 anonymised projects and six engines.
MentionX measures answers: which brands appear, how often and with which sources. It doesn't see token counts inside consumer AI products, so we can't tell you whether shorter passages are favoured because they use fewer tokens.
What we do see is that engines differ widely in how they handle sources:
| Engine | Brand mentioned | Answers with no citations |
|---|---|---|
| Gemini | 47.3% | 50.7% |
| OpenAI | 42.9% | 30.2% |
| Perplexity | 31.5% | 4.1% |
| AI Mode | 31.3% | 12.6% |
| Copilot | 26.0% | 47.0% |
| DeepSeek | 24.0% | 99.0% |
Our view: differences this large come from product design, not from token counts on your page. Focus on being retrievable and clear rather than on token budgets.
Limits. A convenience sample of mostly B2B software projects, run through engine APIs, which can behave differently from the consumer apps. Perplexity, Copilot and DeepSeek figures come from a single day (26 October 2026), so treat them as a snapshot, not a trend.
Worked example: Ledgerline estimates a monitoring budget
Ledgerline is a fictional accounts-payable tool. Figures are illustrative.
Ledgerline's marketing team wants to run 50 buyer prompts through an AI model's API each week to monitor answers. The provider bills by tokens, so they estimate costs using the documentation's rule of thumb: each prompt is short, but each answer may run to a few hundred words.
They test five prompts first and check the token counts the API reports, then multiply up. The estimate based on real counts is more reliable than one based on word counts alone, because brand and product names split into several tokens.
Why do brand names split into several tokens?
Tokenisers learn their vocabulary from large amounts of text. Common words often become a single token, while rare words, invented brand names and words in less common languages are broken into pieces. That doesn't stop a model from recognising a brand, but it's one reason unusual names can be handled less consistently, especially when they look like common words or other brands.
Where will marketers meet tokens?
Mostly in tools: API pricing, limits on how much text a tool can process at once and settings such as maximum output length. If you build or buy AI monitoring, token costs shape how many prompts and runs you can afford. For writing, the practical lesson is simpler: say the important thing clearly and early.
Common questions
How many tokens is a typical web page? It depends on length and language. Use Google's rule of thumb for a rough estimate, or a tokeniser tool for an exact count.
Do tokens affect AI visibility directly? There's no public evidence of that. They mainly affect limits and costs.
Related
Sources
- Google AI for Developers, Understand and count tokens (accessed 31 Oct 2026)
- Google Cloud, Experiment with parameter values (accessed 31 Oct 2026)
- OpenAI, Vector embeddings guide (accessed 31 Oct 2026)
Questions
How many words is a token?
Google's Gemini docs say a token is about 4 characters and 100 tokens equal about 60-80 English words. Other models tokenise differently.
Why do tokens matter to marketers?
They set context window limits and API costs, and they're how AI tools measure prompt and response length.