fullstop punctuation multilang large

提供商oliverguhr
分类token-classification
许可证mit
下载量289.4K
星标0

简介

这是一个专门用于多语言句末标点预测的 Token 分类模型。在处理非结构化文本或多语言抓取数据时，经常会遇到缺失标点导致语义断句困难的问题，该模型能高效识别文本中的句末位置并补全标点。对于需要进行文本清洗、自然语言处理（NLP）预处理的开发者来说，它是一个轻量级的实用工具，可以无缝集成到数据 pipeline 中，降低后续分句或翻译任务的复杂度，上手难度极低。

核心亮点

  • 支持多语言识别，解决跨境文本断句痛点
  • 精准定位句末标点，提升文本结构化质量
  • 轻量级 Token 分类架构，推理速度快
  • MIT 协议开源，企业级部署无压力

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("oliverguhr/fullstop-punctuation-multilang-large")
tokenizer = AutoTokenizer.from_pretrained("oliverguhr/fullstop-punctuation-multilang-large")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download oliverguhr/fullstop-punctuation-multilang-large

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download oliverguhr/fullstop-punctuation-multilang-large config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('oliverguhr/fullstop-punctuation-multilang-large')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/oliverguhr/fullstop-punctuation-multilang-large

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/oliverguhr/fullstop-punctuation-multilang-large

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('oliverguhr/fullstop-punctuation-multilang-large')
tokenizer = AutoTokenizer.from_pretrained('oliverguhr/fullstop-punctuation-multilang-large')

完整文档

来源: HuggingFace

---
language:

  • en

  • de

  • fr

  • it

  • multilingual

tags:
  • punctuation prediction

  • punctuation

datasets: wmt/europarl
license: mit
widget:
  • text: "Ho sentito che ti sei laureata il che mi fa molto piacere"

example_title: "Italian"
  • text: "Tous les matins vers quatre heures mon père ouvrait la porte de ma chambre"

example_title: "French"
  • text: "Ist das eine Frage Frau Müller"

example_title: "German"
  • text: "Yet she blushed as if with guilt when Cynthia reading her thoughts said to her one day Molly you're very glad to get rid of us are not you"

example_title: "English"
metrics:
  • f1

---

This model predicts the punctuation of English, Italian, French and German texts. We developed it to restore the punctuation of transcribed spoken language.

This multilanguage model was trained on the Europarl Dataset provided by the SEPP-NLG Shared Task. *Please note that this dataset consists of political speeches. Therefore the model might perform differently on texts from other domains.*

The model restores the following punctuation markers: "." "," "?" "-" ":"

Sample Code


We provide a simple python package that allows you to process text of any length.

Install

To get started install the package from pypi:

bash
pip install deepmultilingualpunctuation

Restore Punctuation

python
from deepmultilingualpunctuation import PunctuationModel

model = PunctuationModel()
text = "My name is Clara and I live in Berkeley California Ist das eine Frage Frau Müller"
result = model.restore_punctuation(text)
print(result)

output
> My name is Clara and I live in Berkeley, California. Ist das eine Frage, Frau Müller?

Predict Labels

python
from deepmultilingualpunctuation import PunctuationModel

model = PunctuationModel()
text = "My name is Clara and I live in Berkeley California Ist das eine Frage Frau Müller"
clean_text = model.preprocess(text)
labled_words = model.predict(clean_text)
print(labled_words)

output

> [['My', '0', 0.9999887], ['name', '0', 0.99998665], ['is', '0', 0.9998579], ['Clara', '0', 0.6752215], ['and', '0', 0.99990904], ['I', '0', 0.9999877], ['live', '0', 0.9999839], ['in', '0', 0.9999515], ['Berkeley', ',', 0.99800044], ['California', '.', 0.99534047], ['Ist', '0', 0.99998784], ['das', '0', 0.99999154], ['eine', '0', 0.9999918], ['Frage', ',', 0.99622655], ['Frau', '0', 0.9999889], ['Müller', '?', 0.99863917]]

Results

The performance differs for the single punctuation markers as hyphens and colons, in many cases, are optional and can be substituted by either a comma or a full stop. The model achieves the following F1 scores for the different languages:

| Label | EN | DE | FR | IT |
| ------------- | ----- | ----- | ----- | ----- |
| 0 | 0.991 | 0.997 | 0.992 | 0.989 |
| . | 0.948 | 0.961 | 0.945 | 0.942 |
| ? | 0.890 | 0.893 | 0.871 | 0.832 |
| , | 0.819 | 0.945 | 0.831 | 0.798 |
| : | 0.575 | 0.652 | 0.620 | 0.588 |
| - | 0.425 | 0.435 | 0.431 | 0.421 |
| macro average | 0.775 | 0.814 | 0.782 | 0.762 |

Languages

Models

| Languages | Model |
| ------------------------------------------ | ------------------------------------------------------------ |
| English, Italian, French and German | oliverguhr/fullstop-punctuation-multilang-large |
| English, Italian, French, German and Dutch | oliverguhr/fullstop-punctuation-multilingual-sonar-base |
| Dutch | oliverguhr/fullstop-dutch-sonar-punctuation-prediction |

Community Models

| Languages | Model |
| ------------------------------------------ | ------------------------------------------------------------ |
|English, German, French, Spanish, Bulgarian, Italian, Polish, Dutch, Czech, Portugese, Slovak, Slovenian| kredor/punctuate-all |
| Catalan | softcatala/fullstop-catalan-punctuation-prediction |
| Welsh | techiaith/fullstop-welsh-punctuation-prediction |

You can use different models by setting the model parameter:

python
model = PunctuationModel(model = "oliverguhr/fullstop-dutch-punctuation-prediction")

Where do I find the code and can I train my own model?

Yes you can! For complete code of the reareach project take a look at this repository.

There is also an guide on how to fine tune this model for you data / language.

References

code
@article{guhr-EtAl:2021:fullstop,
  title={FullStop: Multilingual Deep Models for Punctuation Prediction},
  author    = {Guhr, Oliver  and  Schumann, Anne-Kathrin  and  Bahrmann, Frank  and  Böhme, Hans Joachim},
  booktitle      = {Proceedings of the Swiss Text Analytics Conference 2021},
  month          = {June},
  year           = {2021},
  address        = {Winterthur, Switzerland},
  publisher      = {CEUR Workshop Proceedings},  
  url       = {http://ceur-ws.org/Vol-2957/sepp_paper4.pdf}
}
查看官方来源