bigbird pegasus large arxiv

提供商google
分类summarization
许可证apache-2.0
下载量414
星标0

简介

BigBird-Pegasus-Large-Arxiv 是由 Google 开发的一款专注于长文本摘要的预训练模型。它解决了传统 Transformer 处理长文档时的计算瓶颈，能够高效处理数千个 Token 的输入。该模型在 arXiv 科学论文数据集上进行了深度优化，非常适合需要从冗长学术论文、技术报告或法律文档中提取核心结论的场景。对于开发者而言，它提供了比通用模型更专业的摘要能力，上手难度中等，可通过 Hugging Face 快速集成，是构建专业文献分析工具的理想选择。

核心亮点

  • 专为长文本设计，突破传统模型输入长度限制
  • 在 arXiv 学术论文摘要任务上表现极其出色
  • 采用 Apache-2.0 协议，对商业应用非常友好
  • 适合构建科研助手、论文精读等专业分析工具

使用方法

安装依赖
# 安装 Hugging Face transformers
pip install transformers torch
SDK 使用
# 使用 transformers 加载模型
from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("google/bigbird-pegasus-large-arxiv")
tokenizer = AutoTokenizer.from_pretrained("google/bigbird-pegasus-large-arxiv")

Hugging Face 下载

我们推荐使用命令行或者 Hugging Face Hub SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 huggingface_hub：

操作指引 
pip install -U huggingface_hub

命令行下载

下载完整模型库

下载完整模型库 
huggingface-cli download google/bigbird-pegasus-large-arxiv

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例）

下载单个文件到指定本地文件夹（以下载 config.json 到当前路径下 ./dir 目录为例） 
huggingface-cli download google/bigbird-pegasus-large-arxiv config.json --local-dir ./dir

更多命令行下载选项，可参见官方文档

SDK 下载

SDK 下载 
# 模型下载
from huggingface_hub import snapshot_download
model_dir = snapshot_download('google/bigbird-pegasus-large-arxiv')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://huggingface.co/google/bigbird-pegasus-large-arxiv

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://huggingface.co/google/bigbird-pegasus-large-arxiv

模型文件托管在 Hugging Face Hub，使用 HF CLI / SDK / Git 直接下载，不经过本站。

PyTorch / Transformers 使用

安装 Transformers

安装 Transformers 
pip install -U transformers torch

模型加载和推理

模型加载和推理 
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained('google/bigbird-pegasus-large-arxiv')
tokenizer = AutoTokenizer.from_pretrained('google/bigbird-pegasus-large-arxiv')

模型下载

我们推荐使用命令行或者 ModelScope SDK 来进行模型的下载。

操作指引：在下载前，请先通过如下命令安装 ModelScope：

操作指引 
pip install modelscope

命令行下载

下载完整模型库

下载完整模型库 
modelscope download --model google/bigbird-pegasus-large-arxiv

下载单个文件到指定本地文件夹（以下载 README.md 到当前路径下 dir 目录为例）

下载单个文件到指定本地文件夹（以下载 README.md 到当前路径下 dir 目录为例） 
modelscope download --model google/bigbird-pegasus-large-arxiv README.md --local_dir ./dir

更多更丰富的命令行下载选项，可参见具体文档

SDK 下载

SDK 下载 
# 模型下载
from modelscope import snapshot_download
model_dir = snapshot_download('google/bigbird-pegasus-large-arxiv')

Git 下载

请确保 lfs 已经被正确安装

Git 下载 
git lfs install
git clone https://www.modelscope.cn/google/bigbird-pegasus-large-arxiv.git

如果您希望跳过 lfs 大文件下载，可以使用如下命令

跳过 LFS 
GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/google/bigbird-pegasus-large-arxiv.git

ModelScope 模型页直接下载模型文件；无需将模型文件放在本站服务器。

Notebook 快速开发

下载并安装 ModelScope library

下载并安装 ModelScope library 
pip install "modelscope[audio,cv,nlp,multi-modal,science]" -f https://modelscope.oss-cn-beijing.aliyuncs.com/releases/repo.html

模型加载和推理

模型加载和推理 
from modelscope.pipelines import pipeline
from modelscope.utils.constant import Tasks

p = pipeline('text-generation', 'google/bigbird-pegasus-large-arxiv')

完整文档

来源: HuggingFace

---
language: en
license: apache-2.0
datasets:

  • scientific_papers

tags:
  • summarization

model-index:
  • name: google/bigbird-pegasus-large-arxiv

results:
- task:
type: summarization
name: Summarization
dataset:
name: scientific_papers
type: scientific_papers
config: pubmed
split: test
metrics:
- name: ROUGE-1
type: rouge
value: 36.0276
verified: true
- name: ROUGE-2
type: rouge
value: 13.4166
verified: true
- name: ROUGE-L
type: rouge
value: 21.9612
verified: true
- name: ROUGE-LSUM
type: rouge
value: 29.648
verified: true
- name: loss
type: loss
value: 2.774355173110962
verified: true
- name: meteor
type: meteor
value: 0.2824
verified: true
- name: gen_len
type: gen_len
value: 209.2537
verified: true
- task:
type: summarization
name: Summarization
dataset:
name: cnn_dailymail
type: cnn_dailymail
config: 3.0.0
split: test
metrics:
- name: ROUGE-1
type: rouge
value: 9.0885
verified: true
- name: ROUGE-2
type: rouge
value: 1.0325
verified: true
- name: ROUGE-L
type: rouge
value: 7.3182
verified: true
- name: ROUGE-LSUM
type: rouge
value: 8.1455
verified: true
- name: loss
type: loss
value: .nan
verified: true
- name: gen_len
type: gen_len
value: 210.4762
verified: true
- task:
type: summarization
name: Summarization
dataset:
name: xsum
type: xsum
config: default
split: test
metrics:
- name: ROUGE-1
type: rouge
value: 4.9787
verified: true
- name: ROUGE-2
type: rouge
value: 0.3527
verified: true
- name: ROUGE-L
type: rouge
value: 4.3679
verified: true
- name: ROUGE-LSUM
type: rouge
value: 4.1723
verified: true
- name: loss
type: loss
value: .nan
verified: true
- name: gen_len
type: gen_len
value: 230.4886
verified: true
- task:
type: summarization
name: Summarization
dataset:
name: scientific_papers
type: scientific_papers
config: arxiv
split: test
metrics:
- name: ROUGE-1
type: rouge
value: 43.4702
verified: true
- name: ROUGE-2
type: rouge
value: 17.4297
verified: true
- name: ROUGE-L
type: rouge
value: 26.2587
verified: true
- name: ROUGE-LSUM
type: rouge
value: 35.5587
verified: true
- name: loss
type: loss
value: 2.1113228797912598
verified: true
- name: gen_len
type: gen_len
value: 183.3702
verified: true
- task:
type: summarization
name: Summarization
dataset:
name: samsum
type: samsum
config: samsum
split: test
metrics:
- name: ROUGE-1
type: rouge
value: 3.621
verified: true
- name: ROUGE-2
type: rouge
value: 0.1699
verified: true
- name: ROUGE-L
type: rouge
value: 3.2016
verified: true
- name: ROUGE-LSUM
type: rouge
value: 3.3269
verified: true
- name: loss
type: loss
value: 7.664482116699219
verified: true
- name: gen_len
type: gen_len
value: 233.8107
verified: true
---

BigBirdPegasus model (large)

BigBird, is a sparse-attention based transformer which extends Transformer based models, such as BERT to much longer sequences. Moreover, BigBird comes along with a theoretical understanding of the capabilities of a complete transformer that the sparse model can handle.

BigBird was introduced in this paper and first released in this repository.

Disclaimer: The team releasing BigBird did not write a model card for this model so this model card has been written by the Hugging Face team.

Model description

BigBird relies on block sparse attention instead of normal attention (i.e. BERT's attention) and can handle sequences up to a length of 4096 at a much lower compute cost compared to BERT. It has achieved SOTA on various tasks involving very long sequences such as long documents summarization, question-answering with long contexts.

How to use

Here is how to use this model to get the features of a given text in PyTorch:

python
from transformers import BigBirdPegasusForConditionalGeneration, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("google/bigbird-pegasus-large-arxiv")


by default encoder-attention is block_sparse with num_random_blocks=3, block_size=64

model = BigBirdPegasusForConditionalGeneration.from_pretrained("google/bigbird-pegasus-large-arxiv")

decoder attention type can't be changed & will be "original_full"


you can change attention_type (encoder only) to full attention like this:

model = BigBirdPegasusForConditionalGeneration.from_pretrained("google/bigbird-pegasus-large-arxiv", attention_type="original_full")

you can change block_size & num_random_blocks like this:

model = BigBirdPegasusForConditionalGeneration.from_pretrained("google/bigbird-pegasus-large-arxiv", block_size=16, num_random_blocks=2)

text = "Replace me by any text you'd like."
inputs = tokenizer(text, return_tensors='pt')
prediction = model.generate(inputs)
prediction = tokenizer.batch_decode(prediction)

Training Procedure

This checkpoint is obtained after fine-tuning BigBirdPegasusForConditionalGeneration for summarization on arxiv dataset** from scientific_papers.

BibTeX entry and citation info

tex
@misc{zaheer2021big,
      title={Big Bird: Transformers for Longer Sequences}, 
      author={Manzil Zaheer and Guru Guruganesh and Avinava Dubey and Joshua Ainslie and Chris Alberti and Santiago Ontanon and Philip Pham and Anirudh Ravula and Qifan Wang and Li Yang and Amr Ahmed},
      year={2021},
      eprint={2007.14062},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}
查看官方来源