Tokens, Explained
A token is the unit of text a language model reads and bills for - roughly three to four characters of English - which is why AI pricing and context limits are counted in tokens rather than words.
Tokens decide what an AI costs, how much it can hold, and where it starts forgetting. Understanding them is the cheapest upgrade available to anyone testing AI systems.
Open Tokens, Explained on Testing Titbits Playground · 1-2 min · Context Engineering · Free, no sign-up
How to read it
- See a sentence split into its actual tokens.
- Compare token count against word count and watch them diverge.
- Watch the context limit fill as more text goes in.
- See what happens at the boundary.
What you will learn
- What a token is and why it is not a word.
- How token counts drive AI cost.
- Why context limits bite earlier than expected.
- What to check when testing a system that has a token budget.
Frequently asked questions
What is a token in AI?
A token is the smallest unit of text a language model processes, averaging about four characters of English. Common words are one token; rare words, names and code split into several.
How many tokens is a word?
Roughly 0.75 words per token for English prose, so 1000 tokens is about 750 words. Code, names and non-English text use more tokens per word.
Why does AI pricing use tokens?
Because tokens are the unit of work. The model processes one token at a time, so cost tracks tokens rather than words or characters.
More free games and infographics for testers
- Context Handoff
- Deep Context
- The Number
- Context Matters
- Guard Rails
- Bug Hunter
- Testing Emoji Charades
- AI Readiness Test
- Bug Maze
- Break the Field
- AI Engineering, Explained by Magic
- Exploratory Testing Dice
- The Tech Divide
- Context Travel
- The Context Paradox
- How GenAI Works
- Loops at Work
- All games on Testing Titbits Playground
- Global leaderboard
Created by Rahul Parwal · TestingTitbits.com