twitter xlm roberta base sentiment
简介
这是一个由 CardiffNLP 团队基于 XLM-RoBERTa 预训练的轻量级多语言情感分析模型。它专门针对社交媒体文本进行了优化，能够识别文本中的正向、负向或中性情绪。对于国内开发者而言，该模型最大的价值在于其跨语言能力，无需为每种语言单独部署模型，即可快速处理包括中文在内的多种语言社交数据。由于其 Base 规模，部署成本低，推理速度快，非常适合集成到舆情监控、用户反馈分析等实际业务场景中，是替代复杂 NLP 管道的高效方案。
核心亮点
- 支持多语言识别，无需翻译即可分析中文情感
- 专为社交媒体短文本优化，对非规范表达鲁棒性强
- 模型规模适中，易于在个人服务器或边缘端部署
- Apache-2.0 协议，可直接用于商业项目开发
使用方法
安装依赖
# 安装 Hugging Face transformers
pip install transformers torchSDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained("cardiffnlp/twitter-xlm-roberta-base-sentiment")
tokenizer = AutoTokenizer.from_pretrained("cardiffnlp/twitter-xlm-roberta-base-sentiment")Hugging Face 下载
我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。
操作指引：在下载前，请先通过如下命令安装 huggingface_hub：
操作指引
pip install -U huggingface_hub命令行下载
下载完整模型库
下载完整模型库
huggingface-cli download cardiffnlp/twitter-xlm-roberta-base-sentiment下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）
下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）
huggingface-cli download cardiffnlp/twitter-xlm-roberta-base-sentiment config.json --local-dir ./dirSDK 下载
SDK 下载
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('cardiffnlp/twitter-xlm-roberta-base-sentiment')Git 下载
请确保 lfs 已经被正确安装
Git 下载
git lfs install
git clone https://huggingface.co/cardiffnlp/twitter-xlm-roberta-base-sentiment如果您希望跳过 lfs 大文件下载，可以使用如下命令
跳过 LFS
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/cardiffnlp/twitter-xlm-roberta-base-sentiment模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。
PyTorch / Transformers 使用
安装 Transformers
安装 Transformers
pip install -U transformers torch模型加载和推理
模型加载和推理
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained('cardiffnlp/twitter-xlm-roberta-base-sentiment')
tokenizer = AutoTokenizer.from_pretrained('cardiffnlp/twitter-xlm-roberta-base-sentiment')模型下载
我们推荐使用命令行或者 ModelScope SDK 来进行模型的下载。
操作指引：在下载前，请先通过如下命令安装 ModelScope：
操作指引
pip install modelscope命令行下载
下载完整模型库
下载完整模型库
modelscope download --model cardiffnlp/twitter-xlm-roberta-base-sentiment下载单个文件到指定本地文件夹（以下载 README.md 到当前路径下 dir 目录为例）
下载单个文件到指定本地文件夹（以下载 README.md 到当前路径下 dir 目录为例）
modelscope download --model cardiffnlp/twitter-xlm-roberta-base-sentiment README.md --local_dir ./dirSDK 下载
SDK 下载
# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('cardiffnlp/twitter-xlm-roberta-base-sentiment')Git 下载
请确保 lfs 已经被正确安装
Git 下载
git lfs install
git clone https://www.modelscope.cn/cardiffnlp/twitter-xlm-roberta-base-sentiment.git如果您希望跳过 lfs 大文件下载，可以使用如下命令
跳过 LFS
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/cardiffnlp/twitter-xlm-roberta-base-sentiment.gitModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。
Notebook 快速开发
下载并安装 ModelScope library
下载并安装 ModelScope library
pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html模型加载和推理
模型加载和推理
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks
p = pipeline('text-generation', 'cardiffnlp/twitter-xlm-roberta-base-sentiment')完整文档
来源: HuggingFace
---
language: multilingual
widget:
- text: "🤗"
- text: "T'estimo! ❤️"
- text: "I love you!"
- text: "I hate you 🤮"
- text: "Mahal kita!"
- text: "사랑해!"
- text: "난 너가 싫어"
- text: "😍😍😍"
---
twitter-XLM-roBERTa-base for Sentiment Analysis
This is a multilingual XLM-roBERTa-base model trained on ~198M tweets and finetuned for sentiment analysis. The sentiment fine-tuning was done on 8 languages (Ar, En, Fr, De, Hi, It, Sp, Pt) but it can be used for more languages (see paper for details).
- Git Repo: XLM-T official repository.
This model has been integrated into the TweetNLP library.
Example Pipeline
python
from transformers import pipeline
model_path = "cardiffnlp/twitter-xlm-roberta-base-sentiment"
sentiment_task = pipeline("sentiment-analysis", model=model_path, tokenizer=model_path)
sentiment_task("T'estimo!")code
[{'label': 'Positive', 'score': 0.6600581407546997}]Full classification example
python
from transformers import AutoModelForSequenceClassification
from transformers import TFAutoModelForSequenceClassification
from transformers import AutoTokenizer, AutoConfig
import numpy as np
from scipy.special import softmax
Preprocess text (username and link placeholders)
def preprocess(text):
new_text = []
for t in text.split(" "):
t = '@user' if t.startswith('@') and len(t) > 1 else t
t = 'http' if t.startswith('http') else t
new_text.append(t)
return " ".join(new_text)
MODEL = f"cardiffnlp/twitter-xlm-roberta-base-sentiment"
tokenizer = AutoTokenizer.from_pretrained(MODEL)
config = AutoConfig.from_pretrained(MODEL)
PT
model = AutoModelForSequenceClassification.from_pretrained(MODEL)
model.save_pretrained(MODEL)
text = "Good night 😊"
text = preprocess(text)
encoded_input = tokenizer(text, return_tensors='pt')
output = model(**encoded_input)
scores = output[0][0].detach().numpy()
scores = softmax(scores)
# TF
model = TFAutoModelForSequenceClassification.from_pretrained(MODEL)
model.save_pretrained(MODEL)
text = "Good night 😊"
encoded_input = tokenizer(text, return_tensors='tf')
output = model(encoded_input)
scores = output[0][0].numpy()
scores = softmax(scores)
Print labels and scores
ranking = np.argsort(scores)
ranking = ranking[::-1]
for i in range(scores.shape[0]):
l = config.id2label[ranking[i]]
s = scores[ranking[i]]
print(f"{i+1}) {l} {np.round(float(s), 4)}")Output:
code
1) Positive 0.7673
2) Neutral 0.2015
3) Negative 0.0313Reference
code
@inproceedings{barbieri-etal-2022-xlm,
title = "{XLM}-{T}: Multilingual Language Models in {T}witter for Sentiment Analysis and Beyond",
author = "Barbieri, Francesco and
Espinosa Anke, Luis and
Camacho-Collados, Jose",
booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation Conference",
month = jun,
year = "2022",
address = "Marseille, France",
publisher = "European Language Resources Association",
url = "https://aclanthology.org/2022.lrec-1.27",
pages = "258--266"
}