xlm roberta base ner hrl

提供商Davlan
分类token-classification
许可证afl-3.0
下载量383.9K
星标0

简介

这是一个基于 XLM-RoBERTa 预训练模型微调的命名实体识别（NER）模型。它最大的特点是支持多语言处理，能够从非结构化文本中自动识别出人名、地名、组织机构等关键实体。对于需要处理全球化数据或多语言混合文本的开发者来说，它比单一语言模型更具通用性。该模型属于 Token 分类任务，上手难度低，可直接集成到 Hugging Face 管道中，是构建多语言信息抽取系统或知识图谱的理想底层组件。

核心亮点

  • 基于 XLM-R 架构，具备强大的多语言泛化能力
  • 专注于 NER 任务，精准提取文本关键实体
  • 兼容 Hugging Face 生态，快速部署低门槛
  • 适用于多语言文档分析与自动化信息抽取

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("Davlan/xlm-roberta-base-ner-hrl")
tokenizer = AutoTokenizer.from_pretrained("Davlan/xlm-roberta-base-ner-hrl")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download Davlan/xlm-roberta-base-ner-hrl

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download Davlan/xlm-roberta-base-ner-hrl config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('Davlan/xlm-roberta-base-ner-hrl')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/Davlan/xlm-roberta-base-ner-hrl

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/Davlan/xlm-roberta-base-ner-hrl

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('Davlan/xlm-roberta-base-ner-hrl')
tokenizer = AutoTokenizer.from_pretrained('Davlan/xlm-roberta-base-ner-hrl')

完整文档

来源: HuggingFace

---
license: afl-3.0
---
Hugging Face's logo
---
language:

  • ar

  • de

  • en

  • es

  • fr

  • it

  • lv

  • nl

  • pt

  • zh

  • multilingual

---

xlm-roberta-base-ner-hrl


Model description


xlm-roberta-base-ner-hrl is a Named Entity Recognition model for 10 high resourced languages (Arabic, German, English, Spanish, French, Italian, Latvian, Dutch, Portuguese and Chinese) based on a fine-tuned XLM-RoBERTa base model. It has been trained to recognize three types of entities: location (LOC), organizations (ORG), and person (PER).
Specifically, this model is a *xlm-roberta-base* model that was fine-tuned on an aggregation of 10 high-resourced languages

Intended uses & limitations


#### How to use
You can use this model with Transformers *pipeline* for NER.
python
from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline
tokenizer = AutoTokenizer.from_pretrained("Davlan/xlm-roberta-base-ner-hrl")
model = AutoModelForTokenClassification.from_pretrained("Davlan/xlm-roberta-base-ner-hrl")
nlp = pipeline("ner", model=model, tokenizer=tokenizer)
example = "Nader Jokhadar had given Syria the lead with a well-struck header in the seventh minute."
ner_results = nlp(example)
print(ner_results)

#### Limitations and bias
This model is limited by its training dataset of entity-annotated news articles from a specific span of time. This may not generalize well for all use cases in different domains.

Training data


The training data for the 10 languages are from:

Language|Dataset
-|-
Arabic | ANERcorp
German | conll 2003
English | conll 2003
Spanish | conll 2002
French | Europeana Newspapers
Italian | Italian I-CAB
Latvian | Latvian NER
Dutch | conll 2002
Portuguese |Paramopama + Second Harem
Chinese | MSRA

The training dataset distinguishes between the beginning and continuation of an entity so that if there are back-to-back entities of the same type, the model can output where the second entity begins. As in the dataset, each token will be classified as one of the following classes:
Abbreviation|Description
-|-
O|Outside of a named entity
B-PER |Beginning of a person’s name right after another person’s name
I-PER |Person’s name
B-ORG |Beginning of an organisation right after another organisation
I-ORG |Organisation
B-LOC |Beginning of a location right after another location
I-LOC |Location

Training procedure


This model was trained on NVIDIA V100 GPU with recommended hyperparameters from HuggingFace code.

查看官方来源