AI Token Counter
Count tokens for GPT-5.5, Claude 4.7, Gemini 3.1, Llama, Grok and DeepSeek. See estimated API input costs and compare token counts side by side.
This AI token counter runs real tokenizer libraries in your browser to count tokens for GPT-5.5, GPT-5.4, Claude 4.7, Gemini 3.1, Llama, Mistral, Grok, and DeepSeek models. Paste text into the editor and instantly see exact token counts, estimated API input costs, and tokens-per-word ratios across providers. Your text never leaves your device - tokenization happens entirely client-side via the gpt-tokenizer package.
About AI Token Counter
What Is a Token?
A token is a chunk of text a language model processes as a single unit. Tokens are not the same as words - they can be whole words, pieces of words (subwords), individual characters, or punctuation marks. The tokenizer splits text into these pieces based on a learned vocabulary built from large training corpora using byte-pair encoding (BPE). Token counts matter because every major LLM API charges per token and enforces context window limits measured in tokens, not words or characters.
| Text Example | Approximate Tokens | Why |
|---|---|---|
| "Hello" | 1 token | Common word, single token in most vocabularies |
| "extraordinary" | 2-3 tokens | Longer word split into subword pieces |
| "Hello, world!" | 4 tokens | Word + comma + space-word + exclamation |
| "const x = 42;" | 5-6 tokens | Code tokens include operators and numbers |
| A 500-word essay | ~650-700 tokens | English prose averages ~1.3 tokens per word |
| A 10KB JSON payload | ~2,500-3,500 tokens | Structural punctuation and keys each tokenize separately |
How Tokenizers Differ Across Providers
Every model family ships its own tokenizer with a distinct vocabulary. OpenAI's GPT-5.5, GPT-5.4, and GPT-4.1/o-series use o200k_base, a 200,000-token vocabulary that is more efficient on code, multilingual text, and emoji than its predecessors. Anthropic's Claude 4.7 uses a refreshed proprietary BPE tokenizer that can consume up to roughly 35% more tokens than 4.6 on the same text. Meta's Llama 4 family uses SentencePiece with a vocabulary of around 128,000 tokens; Llama 3.x also uses SentencePiece. Google Gemini 3.1 uses an internal SentencePiece variant. xAI's Grok and DeepSeek V4 both ship custom BPE variants. Since this tool runs locally, it uses o200k_base for OpenAI models (exact) and cl100k_base as an approximation for everyone else. Estimated counts are marked with a tilde (~) and typically land within a few percent of the native tokenizer.
| Model Family | Tokenizer | Vocabulary Size | Counting Method |
|---|---|---|---|
| GPT-5.5, GPT-5.4, GPT-4.1, o3/o4 | o200k_base | ~200,000 tokens | Exact (runs actual tokenizer) |
| GPT-4, GPT-3.5 (legacy) | cl100k_base | ~100,000 tokens | Exact |
| Claude Fable 5 / Opus 4.8 / 4.7 / Sonnet 4.6 / Haiku 4.5 | Anthropic BPE (refreshed for the 4.7+ generation) | Proprietary | Estimate (cl100k_base proxy) |
| Gemini 3.1 Pro / 3 Flash / 2.5 Pro | SentencePiece (Google) | ~256,000 tokens | Estimate (cl100k_base proxy) |
| Llama 4 Scout / Llama 3.3 / Llama 3.1 | SentencePiece | ~128,000 tokens | Estimate (cl100k_base proxy) |
| Mistral Large 3 / Medium 3.5 / Small 4 | Mistral BPE | ~32,000-131,000 tokens | Estimate (cl100k_base proxy) |
| Grok 4.3 (xAI) | xAI BPE | Proprietary | Estimate (cl100k_base proxy) |
| DeepSeek V4 Flash | DeepSeek BPE | ~129,000 tokens | Estimate (cl100k_base proxy) |
Tokens Per Word by Content Type
Token density varies sharply by what you paste in. English prose is roughly 1.3 tokens per word; dense code and structured data can easily double that. Mixed-language or non-Latin text can produce several tokens per character. Knowing the typical ratio for your content type helps estimate costs before you ever make an API call.
| Content Type | Tokens Per Word | Why |
|---|---|---|
| English prose | 1.2-1.4 | Common English words are often single tokens |
| Technical English (medical, legal) | 1.3-1.6 | Specialised terms get split into subwords |
| Python / JavaScript code | 1.5-2.5 | Operators, indentation, and variable names add tokens |
| JSON / XML / structured data | 2.0-3.0 | Braces, quotes, colons, and keys are individual tokens |
| URLs and file paths | 3.0-5.0 | Slashes, dots, and path segments each become tokens |
| Non-Latin scripts (Chinese, Japanese, Arabic) | 2.0-3.0 per character | Characters outside Latin training distribution need more tokens |
| Base64 / encoded blobs | Very high (1 token per 2-4 chars) | Random strings match no learned subwords |
Worked Example: Pricing a 2,000-Word Prompt
Suppose you want to send a 2,000-word legal document (roughly 2,600 input tokens at 1.3 tokens/word) to four different models and want to know what one request costs. Using the May 2026 published input rates:
- GPT-5.5 at $5.00/1M tokens = 2,600 x 0.000005 = $0.013 per request
- GPT-5.4 at $2.50/1M tokens = 2,600 x 0.0000025 = $0.0065 per request
- Claude Sonnet 4.6 at $3.00/1M tokens = 2,600 x 0.000003 = $0.0078 per request
- Gemini 3.1 Pro at $2.00/1M tokens (under 200K context) = 2,600 x 0.000002 = $0.0052 per request
Scale that to 10,000 requests per day and the daily input spend ranges from $52 (Gemini 3.1 Pro) to $130 (GPT-5.5) - before output tokens. Output tokens typically cost 3-6x the input rate, so the actual bill is usually 2-4x what input pricing alone suggests. For a full monthly projection, the AI pricing calculator handles input plus output plus volume across 25 current models.
Context Window Limits (as of May 2026)
| Model | Context Window | Approximate Words |
|---|---|---|
| GPT-5.5 | 400,000 tokens | ~300,000 words |
| GPT-5.4 | 272,000 tokens | ~200,000 words |
| GPT-4.1 | 1,000,000 tokens | ~750,000 words |
| Claude Opus 4.8 / 4.7 / Sonnet 4.6 | 1,000,000 tokens | ~750,000 words |
| Gemini 3.1 Pro / 2.5 Pro | 1,000,000 tokens | ~750,000 words |
| Llama 4 Scout | 10,000,000 tokens | ~7.5M words |
| Llama 3.3 70B / 3.1 8B | 128,000 tokens | ~96,000 words |
| Mistral Large 3 / Medium 3.5 | 128,000 tokens | ~96,000 words |
| Grok 4.3 (xAI) | 256,000 tokens | ~192,000 words |
| DeepSeek V4 Flash | 128,000 tokens | ~96,000 words |
The context window covers both input and output tokens combined. If you send 800,000 tokens of input to a 1M-context model, only 200,000 tokens remain for the response. Pricing also shifts on several providers once prompts cross a threshold: Claude 4.6 input pricing roughly doubles past 200K tokens in long-context mode, GPT-5.4 input rate rises from $2.50 to $5.00/1M above 272K, and Gemini 3 Pro input rises from $2.00 to $4.00/1M above 200K. Always check whether your prompt is sitting above a tier boundary before optimising elsewhere.
Why Tokenizer Efficiency Matters
The same English paragraph can tokenize to meaningfully different counts across providers. OpenAI's o200k_base, released with GPT-4o and carried into GPT-5.4, is roughly 15% more efficient than the older cl100k_base on English prose and dramatically more efficient on code and non-Latin languages. A 1,000-word Chinese article that used 3,500 tokens under cl100k_base might use closer to 1,800 under o200k_base. That directly affects cost and how much content fits inside the context window. If a provider advertises a bigger vocabulary but quotes a higher per-token price, the effective cost-per-word can still come out cheaper.
Anthropic has not publicly released the Claude tokenizer, but Claude 4.6 models are widely reported to be broadly comparable to cl100k_base on English text and more efficient on XML and structured prompts. Gemini 3 uses a 256K vocabulary SentencePiece variant that handles 140+ languages efficiently. Llama 4 uses a native 128K SentencePiece tokenizer trained on a multimodal corpus. For precise numbers before a production deployment, always run a sample through each provider's official counting endpoint - Anthropic's count_tokens API, OpenAI's tiktoken library, and Google's count_tokens method on the Gemini SDK all give authoritative counts.
Common Mistakes When Estimating Tokens
- Assuming 1 token = 1 word. Off by 30-50% on most content. Always count with a real tokenizer.
- Forgetting the system prompt. System prompts, few-shot examples, and tool definitions all count toward input tokens. A 300-token prompt with a 2,000-token system message is a 2,300-token request.
- Ignoring output tokens in cost calculations. Output is where the bulk of the bill lands on reasoning-heavy tasks. GPT-5.4 charges $15/1M output vs $2.50 input - a 6x premium.
- Treating non-English text like English. A Chinese or Japanese document of the same character count can tokenize to 2-3x more tokens than its English equivalent.
- Base64 or binary-like payloads. Pasting encoded data into a prompt can balloon token counts - a 10KB image encoded as base64 can exceed 15,000 tokens.
- Not accounting for streamed output caps.
max_tokenscaps output, not input. If you set it too low, the model truncates mid-sentence and you still pay for the partial response.
Practical Uses for Token Counting
| Use Case | How Token Counting Helps |
|---|---|
| Prompt engineering | See how much of the context window a prompt uses before sending |
| Cost estimation | Calculate per-request cost across different providers |
| Context limit checks | Verify input plus expected output fits the model's window |
| Document chunking for RAG | Split long documents into chunks that fit the embedding or generation model limit |
| Tokenizer comparison | See how the same text tokenizes differently across vocabularies - some are 15-20% more efficient |
| Batch sizing | Fit the largest possible batch per request without exceeding the window |
Prompt Caching and Batch Discounts
Raw per-token pricing is only part of the real cost. Every major provider now offers prompt caching: if the same prefix (system prompt, tool definitions, retrieved documents) repeats across requests, the cached portion is billed at roughly 10% of the standard input rate. OpenAI offers prompt caching automatically on prefixes over 1,024 tokens. Anthropic charges about 25% of the input rate for cache writes and 10% for cache reads on Claude 4.6 models. Google Gemini 3 charges around $0.31/1M tokens for cached input, a 6x discount on the standard rate. Batch APIs from both OpenAI and Anthropic offer a flat 50% discount on input and output pricing in exchange for 24-hour turnaround. For production workloads with repetitive prompts, the effective cost can land well below the headline per-token figures shown in pricing tables.
For word and character counts, the word counter provides detailed text statistics. For byte-level analysis, the character counter shows character breakdowns including whitespace and code points. Everything in this tool runs in your browser - no text is sent to any server at any point.
Sources
Frequently Asked Questions
What is a token in the context of LLMs?
A token is a chunk of text that a language model processes as a single unit. Tokens can be whole words, parts of words, or even individual characters. On average, one token is roughly 0.75 words in English, but this varies by language and content type.
Are the Claude and Llama token counts exact?
The counts for Claude and Llama are estimates. This tool uses OpenAI's cl100k_base tokenizer as a proxy, which is a close approximation. The actual token count may differ by a small percentage depending on the model's specific tokenizer.
Why do different models produce different token counts?
Each model family uses its own tokenizer with a different vocabulary. GPT-5.5, GPT-5.4, and the GPT-4.1/o-series use the o200k_base tokenizer with a 200,000-token vocabulary, while older GPT-4 and GPT-3.5 models used cl100k_base with 100,000 tokens. Larger vocabularies tend to produce fewer tokens for the same text. Claude Opus 4.7 also uses a new Anthropic tokenizer that can consume up to roughly 35% more tokens for the same text than the previous generation.
Does this tool send my text to any server?
No. All tokenization happens entirely in your browser using the gpt-tokenizer library. Your text never leaves your device.
How are the API costs calculated?
Costs are based on publicly listed input token pricing for each model. The tool multiplies your token count by the per-token rate. Output tokens, which models charge separately, are not included in this estimate.
Related Tools
Link to this tool
Copy this HTML to link to this tool from your website or blog.
<a href="https://toolboxkit.io/tools/ai-token-counter/" title="AI Token Counter - Free Online Tool">Try AI Token Counter on ToolboxKit.io</a>