bert base multilingual cased ner hrl

提供商Davlan
分类token-classification
许可证afl-3.0
下载量301.9K
星标0

简介

这是一个基于多语言 BERT 模型微调的命名实体识别（NER）工具。它在多语言语料上进行了训练，能够识别多种语言文本中的关键实体。对于中国开发者来说，它最大的价值在于其跨语言能力，无需为每种语言单独部署模型即可实现基础的实体提取。上手难度低，可直接集成到 Hugging Face 生态的 Pipeline 中，适合作为多语言信息抽取任务的快速原型开发，或在处理包含外语的混合文本时作为轻量级预处理方案。

核心亮点

  • 支持多语言实体识别，无需切换模型
  • 基于 BERT 架构，兼容 Hugging Face 生态
  • 适用于快速构建跨语言信息抽取原型
  • 轻量级部署，适合处理混合语言文本

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("Davlan/bert-base-multilingual-cased-ner-hrl")
tokenizer = AutoTokenizer.from_pretrained("Davlan/bert-base-multilingual-cased-ner-hrl")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download Davlan/bert-base-multilingual-cased-ner-hrl

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download Davlan/bert-base-multilingual-cased-ner-hrl config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('Davlan/bert-base-multilingual-cased-ner-hrl')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/Davlan/bert-base-multilingual-cased-ner-hrl

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/Davlan/bert-base-multilingual-cased-ner-hrl

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('Davlan/bert-base-multilingual-cased-ner-hrl')
tokenizer = AutoTokenizer.from_pretrained('Davlan/bert-base-multilingual-cased-ner-hrl')

完整文档

来源: HuggingFace

---
license: afl-3.0
---
Hugging Face's logo
---
language:

  • ar

  • de

  • en

  • es

  • fr

  • it

  • lv

  • nl

  • pt

  • zh

  • multilingual

---

bert-base-multilingual-cased-ner-hrl


Model description


bert-base-multilingual-cased-ner-hrl is a Named Entity Recognition model for 10 high resourced languages (Arabic, German, English, Spanish, French, Italian, Latvian, Dutch, Portuguese and Chinese) based on a fine-tuned mBERT base model. It has been trained to recognize three types of entities: location (LOC), organizations (ORG), and person (PER).
Specifically, this model is a *bert-base-multilingual-cased* model that was fine-tuned on an aggregation of 10 high-resourced languages

Intended uses & limitations


#### How to use
You can use this model with Transformers *pipeline* for NER.
python
from transformers import AutoTokenizer, AutoModelForTokenClassification
from transformers import pipeline
tokenizer = AutoTokenizer.from_pretrained("Davlan/bert-base-multilingual-cased-ner-hrl")
model = AutoModelForTokenClassification.from_pretrained("Davlan/bert-base-multilingual-cased-ner-hrl")
nlp = pipeline("ner", model=model, tokenizer=tokenizer)
example = "Nader Jokhadar had given Syria the lead with a well-struck header in the seventh minute."
ner_results = nlp(example)
print(ner_results)

#### Limitations and bias
This model is limited by its training dataset of entity-annotated news articles from a specific span of time. This may not generalize well for all use cases in different domains.

Training data


The training data for the 10 languages are from:

Language|Dataset
-|-
Arabic | ANERcorp
German | conll 2003
English | conll 2003
Spanish | conll 2002
French | Europeana Newspapers
Italian | Italian I-CAB
Latvian | Latvian NER
Dutch | conll 2002
Portuguese |Paramopama + Second Harem
Chinese | MSRA

The training dataset distinguishes between the beginning and continuation of an entity so that if there are back-to-back entities of the same type, the model can output where the second entity begins. As in the dataset, each token will be classified as one of the following classes:
Abbreviation|Description
-|-
O|Outside of a named entity
B-PER |Beginning of a person’s name right after another person’s name
I-PER |Person’s name
B-ORG |Beginning of an organisation right after another organisation
I-ORG |Organisation
B-LOC |Beginning of a location right after another location
I-LOC |Location

Training procedure


This model was trained on NVIDIA V100 GPU with recommended hyperparameters from HuggingFace code.

查看官方来源