What Is a Token?
Tokens as the sole unit for model limits and costs, why Vietnamese consumes more tokens than English, and optimizing System Prompt language.
Tokens: What the Model Actually "Sees" - Not Characters or Words
In Lesson 2, we saw models generate output in discrete fragments called tokens. Tokens are the sole unit measuring all technical limits and operating costs of AI models.
PM Insight: Avoid the trap of assuming "shorter text by feel = fewer tokens". For Vietnamese in particular, token counts diverge significantly from English.
1. Models Split Text into Tokens, Not Characters or Words
Models do not process raw words or characters. A Tokenizer slices input text into sub-units before feeding them into the neural network:
- Each token can be a word, subword, punctuation mark, or whitespace.
- Slicing rules are fixed during tokenizer training (which heavily favors English corpora).
Product Implication: Context limits, streaming latency, and API costs are calculated entirely in tokens, not word counts.
Same meaning, different token count
A tokenizer trained mostly on English keeps common English words whole - Vietnamese words often aren't so lucky.
2. Why Vietnamese Always Costs More Tokens Than English
- English:
"The cat is happy"(4 words) splits cleanly into 4 tokens because common English words exist whole in tokenizer vocabularies. - Vietnamese: The same sentence
"Con mèo đang vui"(4 words) gets chopped into 6 tokens - accented words like "mèo" and "đang" split in half because Vietnamese syllables appear less frequently in training corpora.
Production Reality: Vietnamese text typically consumes 1.5–2× more tokens than English for the exact same message.
3. Three Product Decisions Directly Dictated by Tokens
- Context Window Allocation: Vietnamese System Prompts consume more token budget, shrinking available room for conversation history and RAG context.
- API Operating Costs: Providers charge by total tokens (input + output). Serving Vietnamese users costs more per transaction for equivalent content.
- Response Latency: Higher token counts directly increase time-to-first-token and overall streaming duration.
4. Application: Should You Translate System Prompts to English?
EcoCart Case: The team considers translating an 800-word Vietnamese support System Prompt into English to reduce token overhead (~1,500 tokens down to ~950 tokens).
Translating cuts tokens - the trade-off stays
Toggle the language of EcoCart's 800-word support system prompt to compare estimated token cost.
Relative cost & latency
Illustrative estimate, not a real measurement - the ratio matches what the lesson above described.
PM Trade-off Evaluation:
- Pros: ~35% token cost savings and noticeably faster response streaming.
- Cons: Nuanced Vietnamese support policies and tone guidelines may suffer in translation; local teams face friction editing non-native prompts quickly.
- Recommendation: Translate only once prompt logic is stable and verified against Vietnamese regression test sets.