Analysis

AI Token Cost by Language: 8 Tokenizers Measured

Non-English text averages a 1.54x token multiplier across 8 AI models. With reasoning tokens billed as output, English system prompts cut costs.

AI Token Cost by Language: 8 Tokenizers Measured
Table of Contents

Developers keep saying that working with AI in a language other than English costs more. The same content turns into a different number of tokens depending on the language, and API bills are counted in tokens, reasoning tokens included. What has been hard to find is a like-for-like measurement of how large that gap is across the models in use today.

So AIScroll measured it. On Oct. 6, 2026 (UTC), AIScroll sent 200 identical sentences in nine languages to eight models, including GPT 6.1, Claude Opus 5.5 and Gemini 3.8 Flash. On average, non-English text cost 1.54 times as many tokens as the same content in English. With English set to 1x, Chinese came in at 1.08x. Spanish, German and French ran 1.4x to 1.5x, Korean and Arabic about 1.6x, and Hindi 2.27x.

API prices are quoted per token. Even at the same unit price, a higher token multiplier means a larger bill. This applies to system prompts and other fixed instructions attached to every request. The gap also extends beyond input: answers and hidden reasoning are tokens too, and output tokens cost several times more than input.

Why Tokenizers Charge Languages Differently

A tokenizer splits text into the units a model processes. Strings that appear frequently in training data become single tokens; rare strings are broken into smaller pieces. Because English dominates most training data, an English word often fits into one or two tokens, while non-English text in other scripts requires more tokens to convey the same meaning.

Hindi, written in Devanagari, represents the extreme case. GLM 5.3 Flash gave Hindi a token multiplier of 4.29x. Conversely, models from Chinese labs handled Chinese with as few tokens as English, or fewer. The languages a lab trains its models on show up directly in its tokenizer.

Tokenizers also change between model generations. Anthropic's documentation notes that the tokenizer introduced with Claude Opus 4.7 produces roughly 30% more tokens for the same text than older models. In this test, Opus 5.5 counted the 200 English sentences at 9,023 tokens, while the other seven models landed between 5,469 and 5,791. A per-million-token price list alone does not reveal actual workload costs.

Token Multiplier Results: 8 Models, 9 Languages

Heatmap of input-token multipliers for eight AI models across nine languages
Input-token multiplier versus English by model and language; lower is better. AIScroll measurement, October 6, 2026 (UTC)

Averaged across the eight models, non-English text carried a token multiplier of 1.54x. Only two results fell below 1x, both for Chinese: Kimi K3 at 0.92x and DeepSeek V4.1 Flash at 0.98x. GLM 5.3 Flash matched English on Chinese at 1.00x. Every other combination cost more than English.

Input-token multiplier versus English: eight-model average with the lowest and highest model. AIScroll measurement
LanguageAverageLowest modelHighest model
Chinese1.08xKimi K3 0.92xGPT 6.1 1.28x
Spanish1.41xGemini 3.8 Flash 1.28xKimi K3 1.67x
Japanese1.42xGrok 4.7 1.22xGPT 6.1 1.68x
German1.46xMuse Spark 1.3 1.31xKimi K3 1.73x
French1.47xMuse Spark 1.3 1.37xKimi K3 1.73x
Russian1.50xMuse Spark 1.3 1.26xKimi K3 2.07x
Korean1.61xMuse Spark 1.3 1.27xKimi K3 1.99x
Arabic1.63xGrok 4.7 1.36xKimi K3 2.06x
Hindi2.27xGemini 3.8 Flash 1.30xGLM 5.3 Flash 4.29x

These figures cover input tokens only. Output passes through the same tokenizer, so similar multipliers likely apply, but output length is determined by the model and was not measured directly. Translations also vary slightly in length across languages. Even so, the cost spread for identical content remains clear.

Language Gaps: Chinese Is the Exception

European languages share the Latin alphabet with English yet still cost 1.4x to 1.5x more: Spanish hit 1.41x, German 1.46x and French 1.47x. Russian, written in Cyrillic, came in at 1.50x, and Arabic at 1.63x.

East Asian languages split widely. Chinese, at 1.08x, was the only language close to English, while Japanese reached 1.42x and Korean 1.61x. Kimi K3, the most efficient model for Chinese, proved among the most expensive for its neighbors, hitting 1.99x for Korean and 1.65x for Japanese. A tokenizer tuned for one language does not automatically carry over to the next.

By model, Gemini 3.8 Flash had the lowest average token multiplier for non-English text at 1.32x, followed by Muse Spark 1.3 (1.36x), Grok 4.7 (1.39x), GPT 6.1 (1.43x) and Opus 5.5 (1.51x). The Chinese models—DeepSeek V4.1 Flash (1.64x), GLM 5.3 Flash (1.82x) and Kimi K3 (1.84x)—were the most efficient on Chinese but had the highest averages elsewhere. The low-priced Muse Spark 1.3 also maintained low language multipliers.

Output Costs More, and Reasoning Tokens Bill as Output

Input multipliers alone can appear modest. The larger gap opens on the output side. Claude Opus 5.5's list price is $4 per million input tokens and $20 per million output tokens, five times the input rate.

Reasoning models think before they answer. OpenAI's documentation states that reasoning tokens are invisible in the API but are billed as output tokens. Claude also counts its thinking process as output. On Opus 5.5 and Fable 5.1, thinking cannot be disabled; the effort setting only controls its depth.

The arithmetic is straightforward. On Opus 5.5, 1,000 responses containing 2,000 English tokens of reasoning and answers cost $40. The same volume in Korean, applying Opus's 1.55x Korean token multiplier, comes to about $62. Arabic (1.75x) reaches roughly $70, and Hindi (2.09x) about $84. Chinese adds the least, totaling about $45.

Fable 5.1 and Opus 5.5 Keep Drifting Into English

With reasoning tokens billed at output rates, some users suspect recent models are optimized to do their reasoning in English. Anthropic has not confirmed this. However, users working with non-English text continually report side effects.

Complaints about reasoning language date back to around Opus 4.6. A Claude Code feature request noted that thinking remained in English regardless of the language specified in settings or system prompts; the issue was closed without a new setting. With Opus 5.5 and Fable 5.1, reports indicate this drift now affects user-facing text: progress lines before tool calls appear in English, and replies switch to English after the model processes English material and stay there.

In a bug report filed on Sept. 23, a user restated a Japanese-only reply rule three times, yet the model repeatedly reverted to English. The same thing shows up fairly often in everyday work, too. While it may make sense from a cost perspective, it remains a recurring annoyance for users working in their native languages.

Some argue models now handle non-English text well enough. Anthropic's multilingual support page shows Claude Sonnet 4.5 scoring 96.7% to 96.8% of its English baseline in Korean, Japanese and Hindi. While the quality gap has narrowed, the token multiplier gap measured here persists.

English System Prompts Pay Off Unless You Work in Chinese

Chinese was the only language in this test that cost roughly the same as English. All other non-English text used 1.2x to more than 4x the tokens, depending on the model. The most reliable way to save money is using English system prompts: write the fixed instructions sent with every request in English.

System prompts, skills, tool descriptions and project instruction files accompany every API call. On Opus 5.5, sending a 10,000-token English fixed prompt 1,000 times costs $40 in input. The same system prompts written in Korean cost about $62, and in German about $67. Prompt caching cuts the rate to $0.20 per million tokens, but the token multiplier still applies to cached tokens.

The reply language is a separate setting. Anthropic's documentation recommends specifying the response language in system prompts. Keep the instructions in English, add a line such as 'Always respond in French', and users still get answers in their own language while the fixed cost stays low.

If you can write prompts in English instead of your native language, conducting the entire conversation in English improves both cost and reasoning quality. English system prompts and conversations consume fewer tokens and assign tasks to the model in the language it understands best.

Related posts

Sources

Article by

Plankton

Has worked in the games industry since 2016, helping plan and build a number of titles. Now builds the Mixdog coding agent, makes games and a range of apps, and runs GamerScroll and AIScroll. Writes here about hands-on use of AI models and tools, original measurements, and what building things with AI has taught along the way.