Claude Tokenizer: The 30% Hiding in Your Token Bill

Claude Opus 4.6 and Claude Opus 5 cost exactly the same on paper: $5 per million input tokens, $25 per million output. Same numbers, same billing page. Swap one for the other in a WordPress chatbot and your monthly bill goes up by roughly a third.

Nothing was repriced. The models count tokens differently.

This is the least-discussed line item in AI pricing, and if you are budgeting a site chatbot off a comparison table, it is quietly breaking your numbers. Here is what the Claude tokenizer change actually is, which models it affects, and what it does to the cost of a real support conversation.

What Anthropic actually changed

Buried in Anthropic’s own pricing documentation is a note that most pricing round-ups never repeat:

Anthropic’s documentation states that Claude 4.7 and later models use a newer tokenizer, which it credits for improved performance across a range of tasks, and that Sonnet 4.6 and earlier stay on the previous one. The consequence is stated plainly:

This tokenizer produces approximately 30% more tokens for the same text. The exact increase depends on the content and workload shape.

Anthropic platform documentation, pricing

Read that twice, because two things are happening at once.

First, a tokenizer is the thing that chops your text into billable units. It is not a quality setting or a model parameter you can turn off. The same sentence, sent to two models with different tokenizers, produces two different token counts, and you are billed on the count, not the sentence.

Second, the split is by version number, not by family or tier. Everything from 4.7 upward uses the new tokenizer. Everything at Sonnet 4.6 and below uses the old one. That produces some genuinely counter-intuitive placements.

The one that catches people out is Haiku 4.5. It is a current, actively recommended model, and it sits on the old tokenizer, because 4.5 is below 4.7. Meanwhile Sonnet 5 — the obvious default for most site chatbots — is on the new one.

Why 30% and not some other number

Anthropic does not publish the tokenizer’s vocabulary, so the honest answer is that “approximately 30%” is their figure, not an independently derived one, and the documentation is explicit that the real number moves with your content. Text that is heavy on code, non-English languages, or unusual formatting will diverge from the average more than plain English prose will.

Treat 30% as a planning assumption, not a constant. Later in this post there is a way to measure it exactly for your own traffic, which is what you should actually do before committing to a model.

The current price list, for reference

These are the standard API rates as published on 8 September 2026. Prices are per million tokens.

One piece of genuinely good news while we are here: the $2/$10 rate on Sonnet 5 was originally introductory pricing due to expire on 31 August 2026, with an increase to $3/$15 scheduled for 1 September. That increase was cancelled and the lower rate made permanent. If you budgeted for the rise, you can unwind that.

What this does to a real conversation

Abstract percentages are easy to shrug at, so let us cost an actual WordPress support chat.

The model below is deliberately explicit, so you can substitute your own numbers rather than trusting mine:

• System prompt plus knowledge-base context: 2,000 tokens, resent on every turn
• Six turns per conversation
• User message: ~30 tokens
• Assistant reply: ~200 tokens
• Full history resent each turn, as almost every chatbot implementation does

That works out to 15,630 input tokens and 1,200 output tokens per conversation on the old tokenizer. On the new tokenizer, the same text becomes roughly 20,319 input and 1,560 output.

Per 1,000 conversations:

A site handling 1,000 conversations a month is a fairly ordinary small-business support load. At that volume the spread between the cheapest sensible option and the most expensive is over $250 a month, for work most visitors could not tell apart.

The two cases worth understanding

Case 1: the price cut that is a third of what it looks like

Sonnet 4.6 costs $3/$15. Sonnet 5 costs $2/$10. Line them up and the input price has dropped by a third — a headline 33.3% saving.

Now put the same conversation text through both. Sonnet 4.6 charges you $64.89 per 1,000 conversations. Sonnet 5, counting roughly 30% more tokens for that identical text, charges $56.24.

That is a 13.3% saving, not 33.3%. The cut is real, and Sonnet 5 is still the better buy on price alone, but it is worth less than half of what the sticker implies. If you sized a migration on the sticker figure, you over-forecast the saving by a factor of two and a half.

Case 2: the price rise nobody announced

Opus 4.6 and Opus 5 are both $5/$25. Identical. Nothing on any pricing page suggests a change.

Because Opus 5 is on the new tokenizer and Opus 4.6 is not, the same conversation costs $108.15 on Opus 4.6 and $140.60 on Opus 5. That is a 30% increase, arriving with no announcement, no changelog entry, and no line on your invoice explaining it — because from the billing system’s point of view nothing happened. You simply sent more tokens.

This is the case that bites hardest, because “upgrade to the newer model, same price” is exactly the kind of change a developer makes on a Friday without telling anyone.

The fair reading: more tokens is not the same as worse value

It would be easy to write this up as a stealth price rise. That framing is not quite honest, and it is worth saying why.

Anthropic’s own note attributes the new tokenizer to “improved performance on a wide range of tasks.” A tokenizer that splits text more finely gives the model more positions to reason over, which is a plausible mechanism for better output, not a billing trick. If Opus 5 resolves a support question in four turns where Opus 4.6 needed six, it is cheaper in practice despite costing 30% more per unit of text.

The honest claim is narrower, and it is still important:

• Cost per unit of text has risen for new-tokenizer models. That part is not in dispute.
• Cost per resolved conversation is an open question that depends entirely on your traffic, and nobody can answer it for you from a pricing table.
• Every published price comparison — including the ones ranking above this article — compares the first number and calls it the second.

So do not read the chart above as “avoid new models.” Read it as “the sticker price is no longer a like-for-like comparison, so stop treating it as one.”

Measure it on your own traffic

You do not have to accept 30% as an article of faith. Anthropic exposes a token counting endpoint that returns the exact input token count for a request without running inference, and without charging you for it.

Take a genuine conversation out of your chatbot logs — a real one, with your real system prompt and your real knowledge-base context, not a toy example — and count it against both an old-tokenizer and a new-tokenizer model:

curl https://api.anthropic.com/v1/messages/count_tokens 
  -H "x-api-key: $ANTHROPIC_API_KEY" 
  -H "anthropic-version: 2023-06-01" 
  -H "content-type: application/json" 
  -d '{
    "model": "claude-sonnet-4-5",
    "system": "YOUR REAL SYSTEM PROMPT HERE",
    "messages": [{"role": "user", "content": "A REAL USER QUESTION"}]
  }'

Run it again with a new-tokenizer model id and divide. The ratio you get is your number, and it is the only one that should go into your budget. For a site with heavy code samples or non-English content it can land well away from 30% in either direction.

What to actually do about it

1. Turn on prompt caching before you change anything else

Look again at the cost table. Sonnet 5 with prompt caching costs $34.14 per 1,000 conversations against $56.24 without — a 39% reduction, which is three times larger than the entire tokenizer effect.

The mechanism is simple. Your 2,000-token system prompt and knowledge-base block are identical on every turn. Cached, you pay a small write premium once and then read the block back at a tenth of the input price for the rest of the conversation. A cache read costs 10% of the standard input rate; a five-minute cache write costs 1.25x. That means caching pays for itself after a single re-read.

If you take one thing from this article, it is this: arguing about tokenizers while running an uncached system prompt is optimising the wrong number by a wide margin.

2. Do not default to the biggest model

Haiku 4.5 costs $21.63 per 1,000 conversations. Fable 5.1 costs $281.19 — thirteen times more. For answering “what are your shipping times” against a knowledge base, that gap buys you very little.

A sensible pattern for a site chatbot is a cheap model for the common path and an expensive one only where it earns its place. We have written before about where batch processing genuinely cuts costs and about what self-hosting actually costs once you count the hardware, both of which are worth reading before you assume the answer is a bigger model.

3. Re-check your assumptions when you change model

The Opus 4.6 to Opus 5 case is the important one to internalise. A model swap that looks free on the pricing page is not necessarily free on the invoice. Any time you move between a pre-4.7 and a post-4.7 model, recount your tokens rather than assuming the bill scales with the sticker.

This applies across vendors, not just Anthropic. Tokenization changes quietly at OpenAI and Google too, which is part of why headline prices for new flagship models so rarely predict what you end up paying.

4. Own your API key

None of this analysis is possible on a platform that bills you per conversation and hides the model. If you cannot see your token counts, you cannot audit any of the above — you just get an invoice. Plugins that use your own API key, MxChat included, mean the numbers in this article are ones you can check on your own dashboard rather than take on trust. Our setup documentation covers configuring models and caching.

Frequently asked questions

Does the new tokenizer make Claude more expensive overall?

Per unit of text, yes, for models at 4.7 and above — roughly 30% more tokens for the same input. Whether that makes it more expensive for your workload depends on whether the newer model needs fewer turns to do the job. Measure it; do not assume either way.

Can I switch back to the old tokenizer?

No. The tokenizer is a property of the model, not a request parameter. The only way to get the old tokenization is to use a model that shipped with it, such as Sonnet 4.6 or Haiku 4.5.

Which Claude model is cheapest for a WordPress chatbot?

On raw cost, Haiku 4.5 at $21.63 per 1,000 conversations. On cost-to-quality, Sonnet 5 with prompt caching at $34.14 is the one most site owners should start with — it is the current generation, it is under half the price of running Sonnet 5 uncached, and it comfortably undercuts the older Sonnet 4.6.

Does prompt caching work with the new tokenizer?

Yes. Caching operates on tokens after tokenization, so the discount applies exactly the same way. Cache reads cost 10% of the base input price on every model discussed here.

Where can I check current prices myself?

Anthropic publishes rates at claude.com/pricing and a fuller breakdown, including the tokenizer note quoted above, in its platform documentation. Prices in this article were verified there on 8 September 2026 — check before budgeting, because this year they have moved repeatedly.

The short version

The Claude tokenizer split at version 4.7 means published prices stopped being comparable across model generations. A 33% price cut is worth 13%. An unchanged price is a 30% rise. Neither shows up anywhere on a pricing page.

Count your own tokens, cache your system prompt, and pick the smallest model that does the job. That order matters — caching is worth roughly three times more than the entire tokenizer question, and it is the one thing here you can fix this afternoon.

If you are still choosing a chatbot for your site, our comparison of the best WordPress chatbot plugins covers which ones let you bring your own key and see these numbers at all.