Multilingual sentiment is harder than it looks
Why sentiment analysis breaks across languages — translation loss, code-switching, sarcasm and low-resource languages — and how to evaluate it properly.
Why one score hides the problem
A single overall accuracy figure is dominated by the largest language in the data. A system can report strong results while performing poorly for the customers who write in smaller languages — often the ones with the least support elsewhere.
Always break evaluation down by language, channel and topic. The gaps that matter only appear at that level.
Where multilingual sentiment breaks
These failure modes show up across most projects, regardless of model choice.
- Translation loss — translating to English first can flatten intensity, idioms and politeness markers that carry the sentiment.
- Code-switching — messages that mix languages in one sentence, common in many markets, confuse models trained on single-language text.
- Sarcasm and irony — cues are culture-specific and often depend on context outside the message.
- Script and spelling variation — transliteration, dialect spellings and emoji usage differ between communities.
- Domain shift — 'unpredictable' is praise for a thriller and a complaint about a bank app.
Choosing an approach per language
Treat languages in tiers based on data volume and business importance, and choose the approach per tier.
| Approach | Works best when | Watch out for |
|---|---|---|
| Translate, then classify in English | Quick baseline for low-volume languages | Lost nuance; translation errors compound. |
| Multilingual pre-trained model, fine-tuned | Several languages with some labelled data | Weaker results for languages under-represented in pre-training. |
| Large language model with careful prompts | Little labelled data; aspect-level detail needed | Cost at volume; consistency must be measured. |
Building an evaluation set that tells the truth
Use native speakers, ideally from the markets you serve, to label a sample of real messages for each language. Write labelling guidelines with examples of sarcasm, mixed sentiment and neutral statements, and measure agreement between labellers — if humans disagree often, the label definitions need work before any model is judged.
Include code-switched and dialect examples deliberately; random samples tend to under-represent them.
Running it in production
Sentiment is a signal for prioritising attention, not a verdict. Systems that treat it that way, with people reviewing the edge cases, earn trust faster.
- Report accuracy per language and channel on a regular cadence.
- Route low-confidence or high-impact messages to human review.
- Track language mix over time — a new market can shift performance overnight.
- Refresh evaluation sets as slang, products and topics change.
Working on something similar?
Discuss your workflow, available data and operating constraints with Talee’s engineers to assess a suitable next step.
Talk to an engineer