The mBERT approach to Hinglish toxicity

Standard English toxicity models are a trap for anyone working with South Asian social media data. I learned this the hard way after a project I worked on collapsed because our "toxic" labels were actually just standard news reports—we had scraped the data without actually looking at it. If you blindly feed a vanilla BERT model code-mixed Hinglish like "yaar tum kitne pagal ho," it fails to grasp the nuance because the tokenizer isn't built for that script-switching chaos.

Instead of trying to build a massive, centralized transformer from scratch—which is how most companies waste their Series A funding—I’ve found that leaning on existing multilingual weights is much more efficient. I used bert-base-multilingual-cased for a small-scale experiment. Because it’s already pre-trained on 104 languages, it actually understands the Devanagari and Latin subword relationship better than a specialized English model.

You don't need a massive GPU cluster for this. I ran a training loop on a standard CPU in under 20 minutes using a disciplined, hand-checked dataset of 500 samples. It’s like fine-tuning a specialized tool rather than building a whole new factory. I kept the architecture lightweight, pulling the pooler_output and feeding it into a single linear layer for binary classification.

import torch.nn as nn
from transformers import BertModel
class BertClassifier(nn.Module):
    def __init__(self):
        super().__init__()
        self.bert = BertModel.from_pretrained('bert-base-multilingual-cased')
        self.classifier = nn.Linear(768, 1)
    def forward(self, input_ids, attention_mask):
        pooled = self.bert(input_ids=input_ids, attention_mask=attention_mask).pooler_output
        return torch.sigmoid(self.classifier(pooled)).squeeze()
model = BertClassifier()

Just remember: the model is only as good as your curation. If you don't spot-check your CSVs, you're just training a very expensive way to misinterpret data. Set your max_length to 128 since social media outbursts are short anyway; there's no point in burning memory on empty padding.

pip install torch transformers scikit-learn tqdm