Token counts are not word counts

Billing, context limits and truncation all operate on tokens, and tokens are produced by a learned segmentation of text, not by splitting on spaces. Ordinary English prose tends to land near a stable ratio of tokens to words, which is exactly what makes the shortcut tempting, and exactly why it fails silently when the input stops being ordinary English prose.

Where the ratio breaks

Segmentation cost rises for text the tokenizer was not shaped around: identifiers and code, chemical or legal notation, long digit strings, URLs, base64 blobs, emoji and most non-Latin scripts. A line of source code can consume several times the tokens its character count suggests, because punctuation, indentation and camel-cased names each fragment into multiple pieces. The same document translated into a script with poorer coverage can cost a multiple of the original without a word of content changing.

What to do instead

Count tokens with the tokenizer that belongs to the candidate you are testing, on your own sample of real inputs rather than on a paragraph of clean prose. Do it once, early, and record the distribution rather than the mean: the tail is what overruns a context limit, and the tail is exactly what a mean hides. Where two candidates use different tokenizers, note that a comparison of their per-token prices is not a comparison of their per-request prices.