Multilingual BERT struggles to detect sarcasm and slang in Hindi-English mixed social media posts

The base bert-base-multilingual-cased model performs inconsistently when processing Hinglish toxicity due to its inability to interpret cultural context, sarcastic phrasing, or informal language patterns common in Indian online discussions. When compared against xlm-roberta-large and fine-tuned Llama-3 variants, the evaluation reveals which architecture best uncovers hidden toxicity in code-switched text.

The core limitation stems from mBERT’s lack of exposure to the irregular Hindi-English blending found in social media. It defaults to English lexical roots for toxicity classification while failing to recognize insults embedded in Romanized Hindi words. This leads to over-detection of explicit English profanity while missing culturally specific attacks.

Performance metrics show:

mBERT (Base)

• Precision: 0.72 / Recall: 0.65

- Strengths: Processes inputs quickly with minimal VRAM, excels at flagging direct English swear words in mixed sentences.
- Weaknesses: High false negatives for localized insults; syntax shifts mid-sentence disrupt its parsing.

XLM-RoBERTa (Large)

• Precision: 0.81 / Recall: 0.78

- Strengths: Better semantic cohesion across language switches, treats mixed input as a unified context.
- Weaknesses: Slower execution and higher resource demands; occasionally overfits to regional dialect markers.

Llama-3 (8B) via Few-Shot Prompting

• Precision: 0.88 / Recall: 0.84

- Strengths: Detects toxicity through social implications rather than lexical matches, captures nuanced aggression.
- Weaknesses: Excessive for basic classification; latency rules out real-time moderation use.

For production deployment, mBERT can be adapted by fine-tuning on datasets like HASOC while adding a normalization layer to handle repeated characters (e.g., converting "baaaaad" to "bad"), which improved F1 scores by nearly 4%. The implementation uses:

from transformers import AutoModelForSequenceClassification, Trainer, TrainingArguments

model = AutoModelForSequenceClassification.from_pretrained("bert-base-multilingual-cased", num_labels=2)

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=3,
    per_device_train_batch_size=16,
    learning_rate=2e-5,
    weight_decay=0.01,
    evaluation_strategy="epoch"
)

The issue persists because mBERT’s WordPiece tokenizer fragments Romanized Hindi into sub-tokens without semantic meaning, while XLM-R’s SentencePiece tokenizer preserves the integrity of transliterated terms. While mBERT remains viable for rapid prototyping, it is no longer suitable for production-grade Hinglish toxicity detection. For balanced performance, xlm-roberta-large offers a trade-off between speed and accuracy, and a quantized Llama-3 variant can address the subtle, indirect toxicity mBERT misses.